foreman-debug to collect whole newest (log)files instead of tailing all (log)files
Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. This causes the problem that collected logs have "gaps" in timeline, what means we lose track of events happened that time.
Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.
So we don't collect some log records in close history while we collect some logs in very past history.
What about to rather collect all newest files, until some per-directory limit is reached?
Something like that is done in sosreport  where it works well.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what logfiles are collected
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)
Collected logifles completely cover past few days without a gap; older logfiles not collected.
#4 Updated by Pavel Moravec over 2 years ago
- change -m to limit per size (5000kB by default - subject to change the value)
add_filesto traverse through files in argument from newest to oldest and capture whole files, until the size limit is reached
add_files_typesmethod: this extends foreman-debug "API" to help collecting multiple types of logs in the same directory. E.g.
/var/log/foremancontains multiple types of logs (
cron.log*etc. where the "collect 5MB newest files" limit must be applied to them separately. One could call
add_filesto each such type individually (and maintain such a list whenever new log type is added), or call:
add_files_types "[.]log" /var/log/foreman/*.log*
that will find all unique prefixes of log filenames (i.e.
cron) and will call
add_filesto them separatelly.
This change might break collecting complete (config) files if they are over the size limit (i.e. if
add_files is called to a set of (config) files consuming more than 5MB disk space). I don't expect this to happen, but worth reviewing this.
In the pending PR, I replaced
add_files_types whenever I understood all log types need to be collected. I am not sure if the same should be done for
add_files /var/log/splice/* in
katello-debug.sh since I dont know what logfiles can appear there.
The default limit 5MB (5000kB, checked by
du -k) per
add_files call is just first proposal. In my case, it caused bigger archive was generated than before, but lowering the value too much might cause only logs from insufficient past history are collected - the value is a subject of discussion and testing, surely.
foreman-debugitself) - https://github.com/theforeman/foreman/pull/3881
foreman-debug-proxy) - https://github.com/theforeman/smart-proxy/pull/461
katello-debug.sh) - https://github.com/Katello/katello-packaging/pull/297