Bug #9068
closedforeman-tasks qpid connection seems broken
Description
Started noticing this in nightly:
in /var/log/messages i see:
qpidd: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)
qpidd15489: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)
in /var/log/foreman/production.log, i see:
step ca15a9bc-3dac-4190-9bc4-add6f79402d2:2 dropping event Dynflow::Executors::Abstract::Event[execution_pl
an_id: ca15a9bc-3dac-4190-9bc4-add6f79402d2, step_id: 2, event: Actions::Candlepin::ListenOnCandlepinEvents
::NotConnected[message: Failed to connect (reconnect disabled)], result: #<Dynflow::Future:0x007fa3447f26c0
]
... Core terminated.
and in the task within dynflow for "Listen on candlepin events" i see:
{"messages"=>0,
"last_message"=>nil,
"connection"=>"Failed to connect (reconnect disabled)"}
Updated by dustin tsang almost 10 years ago
may be related to https://issues.apache.org/jira/browse/QPID-5788
possibly a new qpid-cpp package came into nightly..and fix may have been lost in that new package.
need to investigate if this is the case.
Updated by Mattias Giese almost 10 years ago
Same problem here. A new version qpid-cpp-server seems unlikely, as there is only one in EPEL7: http://ftp-stud.hs-esslingen.de/pub/epel/7/x86_64/repoview/qpid-cpp-server.html
foreman-debug-8ELwR.tar.xz was uploaded to rsync://theforeman.org/debug-incoming
HTH
Updated by Daniel Lobato Garcia almost 10 years ago
Still an issue on el7 after a nightly install, it doesn't allow me to edit organizations, delete them, etc... because the related tasks never finish.
Updated by Jorick Astrego almost 10 years ago
Also same sort of issue on Centos 6.6
I see this in /var/log/candlepin/error.log
Caused by: org.apache.qpid.transport.TransportException: Error connecting to broker
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:86) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.Connection.connect(Connection.java:232) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:619) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:398) ~[qpid-client-0.22.jar:na]
... 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_75]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_75]
at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:82) ~[qpid-common-0.22.jar:na]
... 51 common frames omitted
2015-02-19 12:54:26,477 [=, org=] ERROR org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection exception: conn:63d8fdbe
org.apache.qpid.transport.ConnectionException: connection aborted
at org.apache.qpid.transport.Connection.closed(Connection.java:563) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222) [qpid-common-0.22.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
qpid-qmf-0.26-2.el6.x86_64
qpid-cpp-server-store-0.26-11.el6.x86_64
qpid-proton-c-0.8-1.el6.x86_64
qpid-cpp-client-0.26-11.el6.x86_64
python-qpid-0.26-2.el6.noarch
qpid-tools-0.26-11.el6.x86_64
qpid-cpp-client-devel-0.26-11.el6.x86_64
qpid-cpp-server-0.26-11.el6.x86_64
python-qpid-common-0.26-2.el6.noarch
python-qpid-qmf-0.26-2.el6.x86_64
katello-test.netbulae.test-qpid-client-cert-1.0-1.noarch
ruby193-rubygem-qpid_messaging-0.26.1-4.el6.x86_64
katello-test.netbulae.test-qpid-broker-1.0-1.noarch
python-gofer-qpid-1.3.0-1.el6.noarch
Updated by Lukas Zapletal almost 10 years ago
Not sure if this is related, but there was a bug in I think RHEL 7 on qpidd not starting. Check SELinux logs, this was a bug for the core policy.
Updated by Bryce Camp almost 10 years ago
SELinux is disabled on my EL7 Katello server. I have the same problem.
Updated by Jorick Astrego almost 10 years ago
I also have SELinux disabled on CentOS 6.6. It's a requirement of Katello.
The error is consistent on every reboot of the VM
Updated by Bryce Camp almost 10 years ago
Idea on how to get around this? This is creating long running and pending events that are preventing other simple actions like promoting content view version to lifecycle environments.
Updated by Justin Sherrill almost 10 years ago
- Translation missing: en.field_release set to 34
Updated by David Somers-Harris almost 10 years ago
I noticed that the target version is blank for this ticket.
Is that on purpose or is it supposed to be Sprint 44?
Updated by Justin Sherrill almost 10 years ago
- Is duplicate of Bug #9491: Listen on candlepin events paused after upgrade from 2.0 to 2.1 added
Updated by Justin Sherrill almost 10 years ago
- Status changed from New to Duplicate
Duplicate of http://projects.theforeman.org/issues/9068
Updated by Justin Sherrill almost 10 years ago
Current plan for this:
- Katello 2.2 we have fully resolved this issue by upgrading qpid to 0.30. This requires an upgrade to Katello 2.2
- For Katello 2.1: Because the qpid upgrade is such a big upgrade (over a dozen packages), and such a major upgrade we are not going to backport to Katello 2.1. However as part of 2.1.2 we are going to back port a small change that will allow you to disable this feature by simply:
open /etc/foreman/plugins/katello.yaml
and comment out this section:
qpid: url: amqp:ssl:katello-centos7.example.com:5671 subscriptions_queue_address: katello_event_queue
so it looks like:
# qpid: # url: amqp:ssl:katello-centos7.example.com:5671 # subscriptions_queue_address: katello_event_queue
Updated by Justin Sherrill almost 10 years ago
- Status changed from Duplicate to Resolved
- Translation missing: en.field_release changed from 34 to 23