Project

General

Profile

Bug #12427

Unable to apply updates/erratas

Added by Sameer Syed over 3 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Urgent
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

I've a katello setup with a bunch of Capsules. The clients are not registered directly to Katello server, they are registered to the Capsules by installing the cert from the Capsules and running the subscription-manager. The Capsules were build by running the following:

capsule-installer --parent-fqdn "katello.example.com"\
--register-in-foreman "true"\
--foreman-oauth-key "xxxxxxxxxxxxxxxxxxxxxxx"\
--foreman-oauth-secret "xxxxxxxxxxxxxxxxxxxxxxx"\
--pulp-oauth-secret "xxxxxxxxxxxxxxxxxxxxxxx"\
--certs-tar "capsule.exmaple.com.tar"\
--puppet "true"\
--puppetca "true"\
--pulp "true"\
--qpid-router "true"\
--reverse-proxy "true"

I see the registered clients in the foreman console; when I try to apply the errats/patches, it fails with the error "Host did not respond within 30 seconds. Is katello-agent installed and goferd running on the Host?". I've verified goferd is running on the client (and have restarted it). I also see the following info when I restart goferd on the client:

Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] root:481 - connecting to capsule.example.com:5647...
Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] root:521 - Disconnected
Nov 9 19:27:06 katello.client.example.com goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://capsule.example.com:5647, failed: Connection amqps://capsule.example.com:5647 disconnected
Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Please advise on any resolutions.

Thanks.

client.log client.log 40.8 KB Katello client logs Sameer Syed, 03/23/2016 07:05 PM

Related issues

Related to Katello - Bug #11971: Generate Capsule Metadata and Sync fails - Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the HostResolved2015-09-25

History

#1 Updated by Shlomi Zadok over 3 years ago

  • Project changed from OpenSCAP to Foreman

#2 Updated by Dominic Cleal over 3 years ago

  • Project changed from Foreman to Katello
  • Triaged set to No

#3 Updated by Sameer Syed over 3 years ago

Shalomi/Dominic

Any advise on this issue?

Sameer Syed wrote:

I've a katello setup with a bunch of Capsules. The clients are not registered directly to Katello server, they are registered to the Capsules by installing the cert from the Capsules and running the subscription-manager. The Capsules were build by running the following:

capsule-installer --parent-fqdn "katello.example.com"\
--register-in-foreman "true"\
--foreman-oauth-key "xxxxxxxxxxxxxxxxxxxxxxx"\
--foreman-oauth-secret "xxxxxxxxxxxxxxxxxxxxxxx"\
--pulp-oauth-secret "xxxxxxxxxxxxxxxxxxxxxxx"\
--certs-tar "capsule.exmaple.com.tar"\
--puppet "true"\
--puppetca "true"\
--pulp "true"\
--qpid-router "true"\
--reverse-proxy "true"

I see the registered clients in the foreman console; when I try to apply the errats/patches, it fails with the error "Host did not respond within 30 seconds. Is katello-agent installed and goferd running on the Host?". I've verified goferd is running on the client (and have restarted it). I also see the following info when I restart goferd on the client:

Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] root:481 - connecting to capsule.example.com:5647...
Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] root:521 - Disconnected
Nov 9 19:27:06 katello.client.example.com goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://capsule.example.com:5647, failed: Connection amqps://capsule.example.com:5647 disconnected
Nov 9 19:27:06 katello.client.example.com goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 106 seconds

Please advise on any resolutions.

Thanks.

#4 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) set to 70
  • Triaged changed from No to Yes

#5 Updated by Justin Sherrill over 3 years ago

Apologies for the delay.

Is there a firewall at all blocking port 5647 on the capsule server? A full foreman-debug from the katello server may help as well.

#6 Updated by Sameer Syed over 3 years ago

Justin Sherrill wrote:

Apologies for the delay.

Is there a firewall at all blocking port 5647 on the capsule server? A full foreman-debug from the katello server may help as well.

Hi Justin,

Yes there is a firewall, but the ports are open.

#7 Updated by Justin Sherrill about 3 years ago

  • Legacy Backlogs Release (now unused) changed from 70 to 113

#8 Updated by Eric Helms about 3 years ago

  • Related to Bug #11971: Generate Capsule Metadata and Sync fails - Host did not respond within 20 seconds. Is katello-agent installed and goferd running on the Host added

#9 Updated by Justin Sherrill about 3 years ago

  • Category set to 91
  • Status changed from New to Resolved
  • Assignee set to Justin Sherrill

As part of katello 2.4.1 we are updating a new qpid stack with a bunch of bug fixes around stability. I think this will fix a ton of the qpid related issues we are seeing including this one. Could you try out this release when it is out and give it a try?

The updated packages include:

libqpid-dispatch x86_64 0.4-11.el7
python-qpid noarch 0.30-9.el7
qpid-cpp-client x86_64 0.30-8.el7
qpid-cpp-client-devel x86_64 0.30-8.el7
qpid-cpp-server x86_64 0.30-8.el7
qpid-cpp-server-linearstore x86_64 0.30-8.el7
qpid-dispatch-router x86_64 0.4-11.el7
qpid-proton-c x86_64 0.9-13.el7

I will go ahead and close. Please re-open if you continue to have the issue after 2.4.1 is released and you have tried these updated packages.

Thanks!

#10 Updated by Justin Sherrill about 3 years ago

  • Status changed from Resolved to Closed

#11 Updated by Wagner Sartori about 3 years ago

I'm having the same problem for some time now. I upgraded to 2.4.1 yesterday to try a fix, my qpid package versions now, are:
qpid-tools-0.30-4.el7.noarch
python-qpid-proton-0.9-13.el7.x86_64
qpid-dispatch-router-0.4-11.el7.x86_64
qpid-cpp-server-0.30-8.el7.x86_64
qpid-proton-c-0.9-13.el7.x86_64
python-qpid-qmf-0.30-5.el7.x86_64
libqpid-dispatch-0.4-11.el7.x86_64
python-qpid-0.30-9.el7.noarch
qpid-cpp-client-devel-0.30-8.el7.x86_64
python-gofer-qpid-2.6.2-1.el7.noarch
tfm-rubygem-qpid_messaging-0.30.0-3.el7.x86_64
qpid-cpp-server-linearstore-0.30-8.el7.x86_64
qpid-cpp-client-0.30-8.el7.x86_64
qpid-qmf-0.30-5.el7.x86_64

On my capsules I'm receiving this:

Mar 15 07:09:56 server01 goferd: [INFO][MainThread] katelloplugin:344 - reporting: {'enabled_repos': {'repos': [{'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/CentOS_7/x86_64'], 'repositoryid': 'mydomain_com_CentOS_7_x86_64'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/CentOS_7/x86_64_extras'], 'repositoryid': 'mydomain_com_CentOS_7_x86_64_extras'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/CentOS_7/x86_64_updates'], 'repositoryid': 'mydomain_com_CentOS_7_x86_64_updates'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/EPEL_7/x86_64'], 'repositoryid': 'mydomain_com_EPEL_7_x86_64'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/InfluxDB_7/x86_64_stable'], 'repositoryid': 'mydomain_com_InfluxDB_7_x86_64_stable'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/Katello_Client_el7/x86_64'], 'repositoryid': 'mydomain_com_Katello_Client_el7_x86_64'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/PuppetLabs_el7/dependencies_x86_64'], 'repositoryid': 'mydomain_com_PuppetLabs_el7_dependencies_x86_64'}, {'baseurl': ['https://foreman01.mydomain.com/pulp/repos/mydomain_com/Library/Production_7/custom/PuppetLabs_el7/products_x86_64'], 'repositoryid': 'mydomain_com_PuppetLabs_el7_products_x86_64'}]}}
Mar 15 07:09:58 server01 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://foreman01.mydomain.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 15 07:09:58 server01 goferd: [INFO][worker-0] root:505 - connecting to foreman01.mydomain.com:5647...
Mar 15 07:09:58 server01 goferd: [INFO][MainThread] gofer.agent.main:79 - agent started.
Mar 15 07:09:58 server01 goferd: [INFO][worker-0] root:547 - Disconnected
Mar 15 07:09:58 server01 goferd: [ERROR][worker-0] gofer.messaging.adapter.proton.connection:106 - connect: proton+amqps://foreman01.mydomain.com:5647, failed: Connection amqps://foreman01.mydomain.com:5647 disconnected
Mar 15 07:09:58 server01 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:108 - retry in 10 seconds
Mar 15 07:10:08 server01 goferd: [INFO][worker-0] gofer.messaging.adapter.proton.connection:100 - connecting: URL: amqps://foreman01.mydomain.com:5647|SSL: ca: /etc/rhsm/ca/katello-default-ca.pem|key: None|certificate: /etc/pki/consumer/bundle.pem|host-validation: None
Mar 15 07:10:08 server01 goferd: [INFO][worker-0] root:505 - connecting to foreman01.mydomain.com:5647...
Mar 15 07:10:08 server01 goferd: [INFO][worker-0] root:547 - Disconnected

This continues all over, always trying to reconnect.

SSL is working fine:

openssl s_client -connect foreman01.mydomain.com:5647 -cert /etc/pki/consumer/bundle.pem -key /etc/pki/consumer/bundle.pem -state -debug
...
    Start Time: 1458037569
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

#12 Updated by Wagner Sartori about 3 years ago

this was caused by IPv6 connection. foreman01.mydomain.com has both A and AAAA records. it prefers IPv6... soon as I removed the AAAA record, it started working. I'll try to debug qpid-dispatcher now, why it's not listening on IPv6.

#13 Updated by Wagner Sartori about 3 years ago

problem fixed:
/etc/qpid-dispatch/qdrouterd.conf shoud replace addr: 0.0.0.0 to addr: :: on listeners. this will make it ipv6 ready.

#14 Updated by Justin Sherrill about 3 years ago

Hi Wagner,

We are tracking the issue here: http://projects.theforeman.org/issues/12386

The current version of qpid-dispatch we are using only allows for either ipv4 OR ipv6 but not both. We filed a bug with them and it has been resolved here: https://issues.apache.org/jira/browse/DISPATCH-190

I need to see if that fix is available in epel yet as we will likely be moving to those qpid packages soon.

#15 Updated by Wagner Sartori about 3 years ago

Thanks for the heads up Justin.

#16 Updated by Sameer Syed about 3 years ago

Hi Justin,

I've upgraded to katello 2.4.1 version on the Katello Server and the Capsules, but I still see the error 'Host did not respond within 20 seconds. Sync has been cancelled. Is katello-agent installed and goferd running on the Host?' when publishing new content or when I sync using hammer. Do I have to change/add any configurations?

#17 Updated by Justin Sherrill about 3 years ago

Hi Sameer,

Did you update any of the packages on the client? New versions of python-qpid-proton and qpid-proton-c were released for the clients as well.

Try updating those (if you haven't) and restart goferd. If the problem persists, the output generated from /var/log/messages would be useful.

Thanks

#18 Updated by Sameer Syed about 3 years ago

Hi Justin,

I've the latest client packages:

rpm -qa | egrep 'katello|qpid'
katello-agent-2.4.0-3.el6.noarch
qpid-proton-c-0.9-13.el6.x86_64
python-qpid-proton-0.9-13.el6.x86_64

Attaching the logs as well.

Also available in: Atom PDF