Bug #16543

if qpid is slow/unresponsive, candlepin event listener will freeze in event loop, causing dynflow executor to stop responding

Added by Chris Duryee over 5 years ago. Updated almost 4 years ago.

Target version:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:


If you suspend the qpid process (simulating high qpid load) the candlepin event listener process will hang in its event loop. This causes the dynflow executor to not proceed.

For example:

  • on tasks page, view candlepin task process, it should say something like:
 "last_message"=>"b190e333-7821-302a-8131-9693e66e2144 - import.created",
  • now, freeze the qpidd process: kill -19 `pidof qpidd`. Note that the candlepin event listener still thinks its connected.
  • do a "foreman-rake console" and run [:foreman_tasks]). Note that the executor failed to respond.

Once qpidd is unsuspended via kill -18, things will run normally again.

note: this is the call that hangs: The timeout appears to not be honored. I suspect that if qpidd was unresponsive for long enough, the kernel would eventually realize to sever the TCP connection. I have not tested freezing it for over 20 min, but it could take a couple of hours for this to happen. I suspect that the listener would not create a new connection at this point, and if qpid came back, events would start piling up on the katello_event_queue. may provide additional clues about this.


#1 Updated by Chris Duryee over 5 years ago

  • Description updated (diff)

#2 Updated by Eric Helms over 5 years ago

  • Legacy Backlogs Release (now unused) set to 188

#3 Updated by Justin Sherrill over 5 years ago

  • Legacy Backlogs Release (now unused) changed from 188 to 114

Also available in: Atom PDF