Project

General

Profile

Bug #16680

foreman-debug to collect whole newest (log)files instead of tailing all (log)files

Added by Pavel Moravec over 2 years ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
foreman-debug
Target version:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. This causes the problem that collected logs have "gaps" in timeline, what means we lose track of events happened that time.

Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.

So we don't collect some log records in close history while we collect some logs in very past history.

What about to rather collect all newest files, until some per-directory limit is reached?

Something like that is done in sosreport [1] where it works well.

[1] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508‚Äč

Version-Release number of selected component (if applicable):
foreman-debug-1.11.0.49-1.el7sat.noarch

How reproducible:
100%

Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what logfiles are collected

Actual results:
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)

Expected results:
Collected logifles completely cover past few days without a gap; older logfiles not collected.

Additional info:


Related issues

Copied to Foreman - Bug #17044: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosed2016-09-24
Copied to Smart Proxy - Bug #17069: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosed2016-09-24
Copied to Packaging - Bug #17114: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosed

History

#1 Updated by The Foreman Bot over 2 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3881 added

#2 Updated by The Foreman Bot over 2 years ago

  • Pull request https://github.com/theforeman/smart-proxy/pull/461 added

#3 Updated by The Foreman Bot over 2 years ago

  • Pull request https://github.com/Katello/katello-packaging/pull/297 added

#4 Updated by Pavel Moravec over 2 years ago

Proposed solution:

  • change -m to limit per size (5000kB by default - subject to change the value)
  • change add_files to traverse through files in argument from newest to oldest and capture whole files, until the size limit is reached
  • added add_files_types method: this extends foreman-debug "API" to help collecting multiple types of logs in the same directory. E.g. /var/log/foreman contains multiple types of logs ( production.log*, cron.log* etc. where the "collect 5MB newest files" limit must be applied to them separately. One could call add_files to each such type individually (and maintain such a list whenever new log type is added), or call:
    add_files_types "[.]log" /var/log/foreman/*.log*
    

    that will find all unique prefixes of log filenames (i.e. production or cron) and will call add_files to them separatelly.

This change might break collecting complete (config) files if they are over the size limit (i.e. if add_files is called to a set of (config) files consuming more than 5MB disk space). I don't expect this to happen, but worth reviewing this.

In the pending PR, I replaced add_files by add_files_types whenever I understood all log types need to be collected. I am not sure if the same should be done for add_files /var/log/splice/* in katello-debug.sh since I dont know what logfiles can appear there.

The default limit 5MB (5000kB, checked by du -k) per add_files call is just first proposal. In my case, it caused bigger archive was generated than before, but lowering the value too much might cause only logs from insufficient past history are collected - the value is a subject of discussion and testing, surely.

Note the change requires 3 PRs in:

#5 Updated by Justin Sherrill over 2 years ago

  • Legacy Backlogs Release (now unused) set to 162

#6 Updated by Eric Helms over 2 years ago

  • Legacy Backlogs Release (now unused) changed from 162 to 188

#7 Updated by Pavel Moravec over 2 years ago

  • Copied to Bug #17044: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added

#8 Updated by Pavel Moravec over 2 years ago

  • Copied to Bug #17069: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added

#9 Updated by Pavel Moravec over 2 years ago

  • Copied to Bug #17114: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added

#10 Updated by Justin Sherrill over 2 years ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF