Bug #16680
closedforeman-debug to collect whole newest (log)files instead of tailing all (log)files
Description
Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. This causes the problem that collected logs have "gaps" in timeline, what means we lose track of events happened that time.
Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.
So we don't collect some log records in close history while we collect some logs in very past history.
What about to rather collect all newest files, until some per-directory limit is reached?
Something like that is done in sosreport [1] where it works well.
[1] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508
Version-Release number of selected component (if applicable):
foreman-debug-1.11.0.49-1.el7sat.noarch
How reproducible:
100%
Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what logfiles are collected
Actual results:
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)
Expected results:
Collected logifles completely cover past few days without a gap; older logfiles not collected.
Additional info:
Updated by The Foreman Bot about 8 years ago
- Status changed from New to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/3881 added
Updated by The Foreman Bot about 8 years ago
- Pull request https://github.com/theforeman/smart-proxy/pull/461 added
Updated by The Foreman Bot about 8 years ago
- Pull request https://github.com/Katello/katello-packaging/pull/297 added
Updated by Pavel Moravec about 8 years ago
Proposed solution:
- change -m to limit per size (5000kB by default - subject to change the value)
- change
add_files
to traverse through files in argument from newest to oldest and capture whole files, until the size limit is reached - added
add_files_types
method: this extends foreman-debug "API" to help collecting multiple types of logs in the same directory. E.g./var/log/foreman
contains multiple types of logs (production.log*
,cron.log*
etc. where the "collect 5MB newest files" limit must be applied to them separately. One could calladd_files
to each such type individually (and maintain such a list whenever new log type is added), or call:add_files_types "[.]log" /var/log/foreman/*.log*
that will find all unique prefixes of log filenames (i.e.production
orcron
) and will calladd_files
to them separatelly.
This change might break collecting complete (config) files if they are over the size limit (i.e. if add_files
is called to a set of (config) files consuming more than 5MB disk space). I don't expect this to happen, but worth reviewing this.
In the pending PR, I replaced add_files
by add_files_types
whenever I understood all log types need to be collected. I am not sure if the same should be done for add_files /var/log/splice/*
in katello-debug.sh
since I dont know what logfiles can appear there.
The default limit 5MB (5000kB, checked by du -k
) per add_files
call is just first proposal. In my case, it caused bigger archive was generated than before, but lowering the value too much might cause only logs from insufficient past history are collected - the value is a subject of discussion and testing, surely.
theforeman/foreman
(forforeman-debug
itself) - https://github.com/theforeman/foreman/pull/3881theforeman/smart-proxy
(forforeman-debug-proxy
) - https://github.com/theforeman/smart-proxy/pull/461katello/katello-packaging
(forkatello-debug.sh
) - https://github.com/Katello/katello-packaging/pull/297
Updated by Justin Sherrill about 8 years ago
- Translation missing: en.field_release set to 162
Updated by Eric Helms about 8 years ago
- Translation missing: en.field_release changed from 162 to 188
Updated by Pavel Moravec about 8 years ago
- Copied to Bug #17044: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Updated by Pavel Moravec about 8 years ago
- Copied to Bug #17069: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Updated by Pavel Moravec about 8 years ago
- Copied to Bug #17114: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Updated by Justin Sherrill almost 8 years ago
- Status changed from Ready For Testing to Closed