Project

General

Profile

Bug #9068

foreman-tasks qpid connection seems broken

Added by Justin Sherrill about 5 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Candlepin
Target version:
Difficulty:
medium
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Started noticing this in nightly:

in /var/log/messages i see:

qpidd: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)
qpidd15489: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)

in /var/log/foreman/production.log, i see:

step ca15a9bc-3dac-4190-9bc4-add6f79402d2:2 dropping event Dynflow::Executors::Abstract::Event[execution_pl
an_id: ca15a9bc-3dac-4190-9bc4-add6f79402d2, step_id: 2, event: Actions::Candlepin::ListenOnCandlepinEvents
::NotConnected[message: Failed to connect (reconnect disabled)], result: #<Dynflow::Future:0x007fa3447f26c0

]

... Core terminated.

and in the task within dynflow for "Listen on candlepin events" i see:

{"messages"=>0,
"last_message"=>nil,
"connection"=>"Failed to connect (reconnect disabled)"}


Related issues

Is duplicate of Katello - Bug #9491: Listen on candlepin events paused after upgrade from 2.0 to 2.1Duplicate2015-02-20

History

#1 Updated by Justin Sherrill about 5 years ago

note this was observed on el7

#2 Updated by Justin Sherrill about 5 years ago

  • Assignee set to dustin tsang

#3 Updated by dustin tsang about 5 years ago

may be related to https://issues.apache.org/jira/browse/QPID-5788
possibly a new qpid-cpp package came into nightly..and fix may have been lost in that new package.
need to investigate if this is the case.

#4 Updated by Eric Helms about 5 years ago

  • Target version changed from 63 to 66

#5 Updated by Eric Helms about 5 years ago

  • Target version deleted (66)

#6 Updated by Mattias Giese about 5 years ago

Same problem here. A new version qpid-cpp-server seems unlikely, as there is only one in EPEL7: http://ftp-stud.hs-esslingen.de/pub/epel/7/x86_64/repoview/qpid-cpp-server.html

foreman-debug-8ELwR.tar.xz was uploaded to rsync://theforeman.org/debug-incoming

HTH

#7 Updated by Daniel Lobato Garcia about 5 years ago

Still an issue on el7 after a nightly install, it doesn't allow me to edit organizations, delete them, etc... because the related tasks never finish.

#8 Updated by Jorick Astrego about 5 years ago

Also same sort of issue on Centos 6.6

I see this in /var/log/candlepin/error.log

Caused by: org.apache.qpid.transport.TransportException: Error connecting to broker
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:86) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.Connection.connect(Connection.java:232) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:619) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:398) ~[qpid-client-0.22.jar:na]
... 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_75]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_75]
at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:82) ~[qpid-common-0.22.jar:na]
... 51 common frames omitted
2015-02-19 12:54:26,477 [=, org=] ERROR org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection exception: conn:63d8fdbe
org.apache.qpid.transport.ConnectionException: connection aborted
at org.apache.qpid.transport.Connection.closed(Connection.java:563) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222) [qpid-common-0.22.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

qpid-qmf-0.26-2.el6.x86_64
qpid-cpp-server-store-0.26-11.el6.x86_64
qpid-proton-c-0.8-1.el6.x86_64
qpid-cpp-client-0.26-11.el6.x86_64
python-qpid-0.26-2.el6.noarch
qpid-tools-0.26-11.el6.x86_64
qpid-cpp-client-devel-0.26-11.el6.x86_64
qpid-cpp-server-0.26-11.el6.x86_64
python-qpid-common-0.26-2.el6.noarch
python-qpid-qmf-0.26-2.el6.x86_64
katello-test.netbulae.test-qpid-client-cert-1.0-1.noarch
ruby193-rubygem-qpid_messaging-0.26.1-4.el6.x86_64
katello-test.netbulae.test-qpid-broker-1.0-1.noarch
python-gofer-qpid-1.3.0-1.el6.noarch

#9 Updated by Lukas Zapletal almost 5 years ago

Not sure if this is related, but there was a bug in I think RHEL 7 on qpidd not starting. Check SELinux logs, this was a bug for the core policy.

#10 Updated by Bryce Camp almost 5 years ago

SELinux is disabled on my EL7 Katello server. I have the same problem.

#11 Updated by Jorick Astrego almost 5 years ago

I also have SELinux disabled on CentOS 6.6. It's a requirement of Katello.

The error is consistent on every reboot of the VM

#12 Updated by Bryce Camp almost 5 years ago

Idea on how to get around this? This is creating long running and pending events that are preventing other simple actions like promoting content view version to lifecycle environments.

#13 Updated by Justin Sherrill almost 5 years ago

  • Legacy Backlogs Release (now unused) set to 34

#14 Updated by David Somers-Harris almost 5 years ago

I noticed that the target version is blank for this ticket.
Is that on purpose or is it supposed to be Sprint 44?

#15 Updated by Eric Helms almost 5 years ago

  • Target version set to 67

#16 Updated by Eric Helms almost 5 years ago

  • Target version changed from 67 to 68

#17 Updated by Justin Sherrill almost 5 years ago

  • Is duplicate of Bug #9491: Listen on candlepin events paused after upgrade from 2.0 to 2.1 added

#18 Updated by Justin Sherrill almost 5 years ago

  • Status changed from New to Duplicate

#19 Updated by Justin Sherrill almost 5 years ago

Current plan for this:

  • Katello 2.2 we have fully resolved this issue by upgrading qpid to 0.30. This requires an upgrade to Katello 2.2
  • For Katello 2.1: Because the qpid upgrade is such a big upgrade (over a dozen packages), and such a major upgrade we are not going to backport to Katello 2.1. However as part of 2.1.2 we are going to back port a small change that will allow you to disable this feature by simply:

open /etc/foreman/plugins/katello.yaml

and comment out this section:

  qpid:
    url: amqp:ssl:katello-centos7.example.com:5671
    subscriptions_queue_address: katello_event_queue

so it looks like:

#  qpid:
#    url: amqp:ssl:katello-centos7.example.com:5671
#    subscriptions_queue_address: katello_event_queue

#20 Updated by Justin Sherrill almost 5 years ago

  • Status changed from Duplicate to Resolved
  • Legacy Backlogs Release (now unused) changed from 34 to 23

Also available in: Atom PDF