Project

General

Profile

Actions

Bug #26017

closed

foreman-maintain backup online fails on backup-config-files under Satellite load

Added by Martin Bacovsky about 5 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
-

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1673908

Description of problem:
When Satellite is under a load, foreman-maintain backup online fails in collecting config files. It is due to the fact that "config" contains also files/dirs changing over time, like:

/var/lib/candlepin/hornetq/journal/ (on 6.3 and older)
/var/lib/candlepin/activemq-artemis/journal/ (on 6.4 and newer)
/var/lib/qpidd/.qpidd/qls/jrnl2/

The first pair is due to candlepin sending events to qpidd and temporarily storing them in hornetQ/ActiveMQ broker internally before the send.

The later changes its content when either below action triggers a new journal file is used or returned back to empty file pool:
- katello_event_queue gets a message from candlepin, or LOCE fetches a message from it
- many resource queues get updated when pulp tasks get progressed
- pulp.agent.* queues get updated when an katello-agent task is created or applied

Neither of those possibly corrupted data is essential for building in-house reproducer, but they can be beneficial to know for troubleshooting.

Ideally, foreman-maintain shall get over such issue just with warning (by default), not failing.

Version-Release number of selected component (if applicable):
rubygem-foreman_maintain-0.2.11-1

How reproducible:
100% within some time

Steps to Reproduce:
1. Generate heavier load of candlepin events (re-register systems frequently, attach/detach subscriptions etc.)
2. Or generate more pulp tasks (frequent repo sync that will be no-op at the end, CV publishing new version without a change,..)
3. foreman-maintain backup online -y /tmp/satellite-backup
(call it more times)

Actual results:
backup fails with errors like:

tar: /var/lib/candlepin/hornetq/journal/hornetq-data-497608.hq: file changed as we read it
tar: /var/lib/qpidd/.qpidd/qls/jrnl2/katello_event_queue/c2bc8b9e-8155-4b69-87f2-8f6a61df06b3.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/jrnl2/katello_event_queue/c2c98b9c-1027-4bb0-8487-18e05f615218.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/jrnl2/pulp.agent.04c39f77-df2c-4e08-9d61-d0825dbd14d8: Warning: Cannot open: No such file or directory
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/9e1b017b-ddad-41aa-a070-cacc908c4c7e.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/bdc9192a-02bf-4234-92bb-eaafb707c2b8.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/c0db25dd-2217-498d-b001-b9364a191e6a.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/c3791d4a-c0cf-4a10-8020-afc476d34d98.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/c3c08240-b57e-4603-a1bf-dff69179a8a2.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/c43ef88f-a800-456f-b754-5d6650d49e94.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/dd16a6c9-5692-4800-bddf-9230ba4900d7.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/f9c23c90-f6e1-4392-b17d-81eeff45cc24.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/ba6a2877-df47-4dee-bae1-e021912c2c75.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/bc237c99-61d0-485b-a225-93a7817fd50d.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/c0695e17-f134-49e3-bc1a-1394580d2d4e.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/c2bc8b9e-8155-4b69-87f2-8f6a61df06b3.jrnl: File removed before we read it
tar: /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/c2c98b9c-1027-4bb0-8487-18e05f615218.jrnl: File removed before we read it

or similar

Expected results:
Default behaviour should be posting such info and a WARNING, but continuing in backup. And let user to decide if that warnings matter.

Additional info:
implementing bz 1673797 is kind of workaround here, in some situations

Actions #1

Updated by The Foreman Bot almost 5 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman_maintain/pull/253 added
Actions #2

Updated by Pavel Moravec almost 5 years ago

  • Status changed from Ready For Testing to Closed
Actions #3

Updated by The Foreman Bot about 4 years ago

  • Pull request https://github.com/theforeman/foreman_maintain/pull/319 added
Actions #4

Updated by The Foreman Bot about 4 years ago

  • Pull request https://github.com/theforeman/foreman_maintain/pull/324 added
Actions #5

Updated by The Foreman Bot about 4 years ago

  • Pull request https://github.com/theforeman/foreman_maintain/pull/325 added
Actions #6

Updated by Amit Upadhye about 4 years ago

  • Assignee changed from Anurag Patel to Amit Upadhye
Actions

Also available in: Atom PDF