Project

General

Profile

Actions

Bug #9995

closed

Capsules syncs hang indefinitely (perhaps when syncing multiple capsules); cyclical errors on capsules

Added by Eric Helms almost 9 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Foreman Proxy Content
Target version:
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1205893
Description of problem:
Tried to sync a bunch of capsules at the same time. the tasks have been in a 'running' state in dynflow for hours but not moving beyond 50%. Two capsules are throwing the "resetting dropped connection" message (a different bz) and two are throwing other errors.

Version-Release number of selected component (if applicable):

How reproducible:

I am unsure. Probably more likely if you are trying to sync a bunch of capsules with a bunch of content at the same time.

Steps to Reproduce:
1. Create/register 4+ capsules
2. Sync a large swath of content across different CVs and environments
3. View logs in systems

Actual results:

Lots of errors, cyclically repeated, like:

Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:00:58 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:00:58 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:02:46 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:02:46 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647|SSL: ca: /etc/rhsm/ca/katello-server-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:473 - connecting to rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647...
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] root:513 - Disconnected
Mar 25 17:04:33 sparks goferd: [ERROR][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647, failed: Connection amqps://rhsm-qe-3.rhq.lab.eng.bos.redhat.com:5647 disconnected
Mar 25 17:04:33 sparks goferd: [INFO][pulp.agent.830fbfeb-e4fb-4d9f-b300-8dc5bd5d4141] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

And sync never continues

Expected results:

Sync works

Additional info:

Actions #1

Updated by Eric Helms almost 9 years ago

  • translation missing: en.field_release set to 23
  • Triaged changed from No to Yes
Actions #2

Updated by Eric Helms almost 9 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100
Actions #3

Updated by Eric Helms almost 9 years ago

  • Status changed from Closed to New
Actions #4

Updated by Eric Helms almost 9 years ago

Erroneous revision due to fudging a number in the commit message.

Actions #5

Updated by Eric Helms almost 9 years ago

  • Status changed from New to Resolved

Fixed by updating qpid-dispatch to qpid-dispatch-0.4-2.20150402.el7 (http://koji.katello.org/koji/buildinfo?buildID=21399)

Actions

Also available in: Atom PDF