Project

General

Profile

Bug #9995

Capsules syncs hang indefinitely (perhaps when syncing multiple capsules); cyclical errors on capsules

Added by Eric Helms almost 8 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Foreman Proxy Content
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1205893
Description of problem:
Tried to sync a bunch of capsules at the same time. the tasks have been in a 'running' state in dynflow for hours but not moving beyond 50%. Two capsules are throwing the "resetting dropped connection" message (a different bz) and two are throwing other errors.

Version-Release number of selected component (if applicable):

How reproducible:

I am unsure. Probably more likely if you are trying to sync a bunch of capsules with a bunch of content at the same time.

Steps to Reproduce:
1. Create/register 4+ capsules
2. Sync a large swath of content across different CVs and environments
3. View logs in systems

Actual results:

Lots of errors, cyclically repeated, like:

Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:00:58 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:02:46 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:04:33 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

And sync never continues

Expected results:

Sync works

Additional info:

Associated revisions

Revision 61977d1e (diff)
Added by Eric Helms almost 8 years ago

Fixes #9995: Only send content IDs that user passes during incremental update.

Revision 742782f7
Added by Eric D Helms almost 8 years ago

Merge pull request #290 from ehelms/fixes-9955

Fixes #9995: Only send content IDs that user passes during incremental u...

History

#1 Updated by Eric Helms almost 8 years ago

  • Legacy Backlogs Release (now unused) set to 23
  • Triaged changed from No to Yes

#2 Updated by Eric Helms almost 8 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

#3 Updated by Eric Helms almost 8 years ago

  • Status changed from Closed to New

#4 Updated by Eric Helms almost 8 years ago

Erroneous revision due to fudging a number in the commit message.

#5 Updated by Eric Helms almost 8 years ago

  • Status changed from New to Resolved

Fixed by updating qpid-dispatch to qpid-dispatch-0.4-2.20150402.el7 (http://koji.katello.org/koji/buildinfo?buildID=21399)

Also available in: Atom PDF