Project

General

Profile

Actions

Bug #9068

closed

foreman-tasks qpid connection seems broken

Added by Justin Sherrill over 9 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Difficulty:
medium
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Started noticing this in nightly:

in /var/log/messages i see:

qpidd: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)
qpidd15489: 2015-01-21 14:37:36 [System] error Error reading socket: Success(0)

in /var/log/foreman/production.log, i see:

step ca15a9bc-3dac-4190-9bc4-add6f79402d2:2 dropping event Dynflow::Executors::Abstract::Event[execution_pl
an_id: ca15a9bc-3dac-4190-9bc4-add6f79402d2, step_id: 2, event: Actions::Candlepin::ListenOnCandlepinEvents
::NotConnected[message: Failed to connect (reconnect disabled)], result: #<Dynflow::Future:0x007fa3447f26c0

]

... Core terminated.

and in the task within dynflow for "Listen on candlepin events" i see:

{"messages"=>0,
"last_message"=>nil,
"connection"=>"Failed to connect (reconnect disabled)"}


Related issues 1 (0 open1 closed)

Is duplicate of Katello - Bug #9491: Listen on candlepin events paused after upgrade from 2.0 to 2.1Duplicatedustin tsang02/20/2015Actions
Actions #1

Updated by Justin Sherrill over 9 years ago

note this was observed on el7

Actions #2

Updated by Justin Sherrill over 9 years ago

  • Assignee set to dustin tsang
Actions #3

Updated by dustin tsang over 9 years ago

may be related to https://issues.apache.org/jira/browse/QPID-5788
possibly a new qpid-cpp package came into nightly..and fix may have been lost in that new package.
need to investigate if this is the case.

Actions #4

Updated by Eric Helms about 9 years ago

  • Target version changed from 63 to 66
Actions #5

Updated by Eric Helms about 9 years ago

  • Target version deleted (66)
Actions #6

Updated by Mattias Giese about 9 years ago

Same problem here. A new version qpid-cpp-server seems unlikely, as there is only one in EPEL7: http://ftp-stud.hs-esslingen.de/pub/epel/7/x86_64/repoview/qpid-cpp-server.html

foreman-debug-8ELwR.tar.xz was uploaded to rsync://theforeman.org/debug-incoming

HTH

Actions #7

Updated by Daniel Lobato Garcia about 9 years ago

Still an issue on el7 after a nightly install, it doesn't allow me to edit organizations, delete them, etc... because the related tasks never finish.

Actions #8

Updated by Jorick Astrego about 9 years ago

Also same sort of issue on Centos 6.6

I see this in /var/log/candlepin/error.log

Caused by: org.apache.qpid.transport.TransportException: Error connecting to broker
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:86) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.Connection.connect(Connection.java:232) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:221) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:619) ~[qpid-client-0.22.jar:na]
at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:398) ~[qpid-client-0.22.jar:na]
... 47 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_75]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_75]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_75]
at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_75]
at org.apache.qpid.transport.network.io.IoNetworkTransport.connect(IoNetworkTransport.java:82) ~[qpid-common-0.22.jar:na]
... 51 common frames omitted
2015-02-19 12:54:26,477 [=, org=] ERROR org.apache.qpid.client.AMQConnectionDelegate_0_10 - connection exception: conn:63d8fdbe
org.apache.qpid.transport.ConnectionException: connection aborted
at org.apache.qpid.transport.Connection.closed(Connection.java:563) ~[qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:112) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:204) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.security.ssl.SSLReceiver.closed(SSLReceiver.java:66) [qpid-common-0.22.jar:na]
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:222) [qpid-common-0.22.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

qpid-qmf-0.26-2.el6.x86_64
qpid-cpp-server-store-0.26-11.el6.x86_64
qpid-proton-c-0.8-1.el6.x86_64
qpid-cpp-client-0.26-11.el6.x86_64
python-qpid-0.26-2.el6.noarch
qpid-tools-0.26-11.el6.x86_64
qpid-cpp-client-devel-0.26-11.el6.x86_64
qpid-cpp-server-0.26-11.el6.x86_64
python-qpid-common-0.26-2.el6.noarch
python-qpid-qmf-0.26-2.el6.x86_64
katello-test.netbulae.test-qpid-client-cert-1.0-1.noarch
ruby193-rubygem-qpid_messaging-0.26.1-4.el6.x86_64
katello-test.netbulae.test-qpid-broker-1.0-1.noarch
python-gofer-qpid-1.3.0-1.el6.noarch

Actions #9

Updated by Lukas Zapletal about 9 years ago

Not sure if this is related, but there was a bug in I think RHEL 7 on qpidd not starting. Check SELinux logs, this was a bug for the core policy.

Actions #10

Updated by Bryce Camp about 9 years ago

SELinux is disabled on my EL7 Katello server. I have the same problem.

Actions #11

Updated by Jorick Astrego about 9 years ago

I also have SELinux disabled on CentOS 6.6. It's a requirement of Katello.

The error is consistent on every reboot of the VM

Actions #12

Updated by Bryce Camp about 9 years ago

Idea on how to get around this? This is creating long running and pending events that are preventing other simple actions like promoting content view version to lifecycle environments.

Actions #13

Updated by Justin Sherrill about 9 years ago

  • translation missing: en.field_release set to 34
Actions #14

Updated by David Somers-Harris about 9 years ago

I noticed that the target version is blank for this ticket.
Is that on purpose or is it supposed to be Sprint 44?

Actions #15

Updated by Eric Helms about 9 years ago

  • Target version set to 67
Actions #16

Updated by Eric Helms about 9 years ago

  • Target version changed from 67 to 68
Actions #17

Updated by Justin Sherrill about 9 years ago

  • Is duplicate of Bug #9491: Listen on candlepin events paused after upgrade from 2.0 to 2.1 added
Actions #18

Updated by Justin Sherrill about 9 years ago

  • Status changed from New to Duplicate
Actions #19

Updated by Justin Sherrill about 9 years ago

Current plan for this:

  • Katello 2.2 we have fully resolved this issue by upgrading qpid to 0.30. This requires an upgrade to Katello 2.2
  • For Katello 2.1: Because the qpid upgrade is such a big upgrade (over a dozen packages), and such a major upgrade we are not going to backport to Katello 2.1. However as part of 2.1.2 we are going to back port a small change that will allow you to disable this feature by simply:

open /etc/foreman/plugins/katello.yaml

and comment out this section:

  qpid:
    url: amqp:ssl:katello-centos7.example.com:5671
    subscriptions_queue_address: katello_event_queue

so it looks like:

#  qpid:
#    url: amqp:ssl:katello-centos7.example.com:5671
#    subscriptions_queue_address: katello_event_queue
Actions #20

Updated by Justin Sherrill about 9 years ago

  • Status changed from Duplicate to Resolved
  • translation missing: en.field_release changed from 34 to 23
Actions

Also available in: Atom PDF