Project

General

Profile

Actions

Bug #16680

closed

foreman-debug to collect whole newest (log)files instead of tailing all (log)files

Added by Pavel Moravec over 7 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:

Description

Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. This causes the problem that collected logs have "gaps" in timeline, what means we lose track of events happened that time.

Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.

So we don't collect some log records in close history while we collect some logs in very past history.

What about to rather collect all newest files, until some per-directory limit is reached?

Something like that is done in sosreport [1] where it works well.

[1] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508

Version-Release number of selected component (if applicable):
foreman-debug-1.11.0.49-1.el7sat.noarch

How reproducible:
100%

Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what logfiles are collected

Actual results:
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)

Expected results:
Collected logifles completely cover past few days without a gap; older logfiles not collected.

Additional info:


Related issues 3 (0 open3 closed)

Copied to Foreman - Bug #17044: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosedPavel Moravec09/24/2016Actions
Copied to Smart Proxy - Bug #17069: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosedPavel Moravec09/24/2016Actions
Copied to Packaging - Bug #17114: foreman-debug to collect whole newest (log)files instead of tailing all (log)filesClosedPavel MoravecActions
Actions #1

Updated by The Foreman Bot over 7 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3881 added
Actions #2

Updated by The Foreman Bot over 7 years ago

  • Pull request https://github.com/theforeman/smart-proxy/pull/461 added
Actions #3

Updated by The Foreman Bot over 7 years ago

  • Pull request https://github.com/Katello/katello-packaging/pull/297 added
Actions #4

Updated by Pavel Moravec over 7 years ago

Proposed solution:

  • change -m to limit per size (5000kB by default - subject to change the value)
  • change add_files to traverse through files in argument from newest to oldest and capture whole files, until the size limit is reached
  • added add_files_types method: this extends foreman-debug "API" to help collecting multiple types of logs in the same directory. E.g. /var/log/foreman contains multiple types of logs ( production.log*, cron.log* etc. where the "collect 5MB newest files" limit must be applied to them separately. One could call add_files to each such type individually (and maintain such a list whenever new log type is added), or call:
    add_files_types "[.]log" /var/log/foreman/*.log*
    

    that will find all unique prefixes of log filenames (i.e. production or cron) and will call add_files to them separatelly.

This change might break collecting complete (config) files if they are over the size limit (i.e. if add_files is called to a set of (config) files consuming more than 5MB disk space). I don't expect this to happen, but worth reviewing this.

In the pending PR, I replaced add_files by add_files_types whenever I understood all log types need to be collected. I am not sure if the same should be done for add_files /var/log/splice/* in katello-debug.sh since I dont know what logfiles can appear there.

The default limit 5MB (5000kB, checked by du -k) per add_files call is just first proposal. In my case, it caused bigger archive was generated than before, but lowering the value too much might cause only logs from insufficient past history are collected - the value is a subject of discussion and testing, surely.

Note the change requires 3 PRs in:
Actions #5

Updated by Justin Sherrill over 7 years ago

  • translation missing: en.field_release set to 162
Actions #6

Updated by Eric Helms over 7 years ago

  • translation missing: en.field_release changed from 162 to 188
Actions #7

Updated by Pavel Moravec over 7 years ago

  • Copied to Bug #17044: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Actions #8

Updated by Pavel Moravec over 7 years ago

  • Copied to Bug #17069: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Actions #9

Updated by Pavel Moravec over 7 years ago

  • Copied to Bug #17114: foreman-debug to collect whole newest (log)files instead of tailing all (log)files added
Actions #10

Updated by Justin Sherrill about 7 years ago

  • Status changed from Ready For Testing to Closed
Actions

Also available in: Atom PDF