Plugins » Katello

I enabled sql logging, and it appeared that the postgres queries that were holding up everything were related to processing OpenSCAP. I turned off OpenSCAP and ran vacuum analyze, and things seem to be much better. I am still getting the qpidd/pulp.agent errors.

I believe it was not:

[root@foreman-01 katello-backup-20180428152442]# ls la
total 249345816
drwxrwx--. 2 root postgres 4096 Apr 28 20:29 .
drwxrwx---. 6 root postgres 12288 Apr 30 07:57 ..
rw-r--r-. 1 root root 352057984 Apr 28 15:42 config_files.tar.gz
rw-r--r-. 1 root root 25013972 Apr 28 15:42 .config.snar
rw-r--r-. 1 root root 41008 Apr 28 15:24 metadata.yml
rw-r--r-. 1 root root 14166466469 Apr 28 18:29 mongo_data.tar.gz
rw-r--r-. 1 root root 126 Apr 28 18:29 .mongo.snar
rw-r--r-. 1 root root 37676653124 Apr 28 18:08 pgsql_data.tar.gz
rw-r--r-. 1 root root 44445 Apr 28 18:08 .postgres.snar
rw-r--r-. 1 root root 202906142720 Apr 28 17:10 pulp_data.tar
rw-r--r-. 1 root root 203443453 Apr 28 17:10 .pulp.snar

Yes - I extracted config_files.tar.gz, and the only directories it had under /var/lib were "candlepin" and "puppet" - nothing for qpidd.

The on-going issue I have is that qdrouterd does not seem to be functional; even the foreman server itself can't connect. I have this repeatedly in the log:
2018-05-21 20:57:29.586236 +0000 SERVER (info) Connection from <IP>:47878 (to 0.0.0.0:5646) failed: amqp:connection:framing-error SSL Failure: Unknown error

Discussed this off-thread, where I suggested taring /var/lib/qpidd from the old machine and untaring on the new machine. I'm not sure why qpidd was missed in the backup, it should be included from the code logic

As John stated, I manually copied over the .qpidd data from the old server to the new. That is now in place, but I am still having issues, specifically from what I can tell related to qdrouterd related functions.

As I see qdrouterd SSL errors, I looked at the config:
ssl-profile {
name: server
cert-db: /etc/pki/katello/certs/katello-default-ca.crt
cert-file: /etc/pki/katello/qpid_router_server.crt
key-file: /etc/pki/katello/qpid_router_server.key
}

If I look at /etc/pki/katello-certs-tools/certs/katello-default-ca.crt, I see the old hostname in there (and it has a date from 2017, before the server rename/migration).

The errors in qdrouterd.log that I see are:
2018-05-31 15:04:46.277338 +0000 SERVER (info) Connection from <redacted IP>:58952 (to 0.0.0.0:5646) failed: amqp:connection:framing-error SSL Failure: Unknown error
and
2018-05-31 15:04:56.347872 +0000 SERVER (info) Connection from <redacted IP>:41064 (to 0.0.0.0:5646) failed: amqp:connection:framing-error No valid protocol header found

Those seem coupled with:
2018-05-31 15:04:46.277869 +0000 ROUTER_LS (info) Link to Neighbor Router Lost - link_tag=8

Josh,

Were you able to resolve this issue?

-John

Unfortunately no. Qdrouterd is still full of errors, and all hosts report "unknown" status, even with katello-agent running. I believe the certs are mismatched with the hostname. How can I regenerate those?

You could try using katello-change-hostname to change to a new hostname (and then potentially change it back). That should force the regeneration of all the certs.

Do you have multiple smart proxies running running qdrouterd? There is currently a bug in proton that causes all qdrouterd's to start erroring if any one smart proxy has an ssl issue: https://issues.apache.org/jira/browse/PROTON-1587

What made you think there is some cert hostname mismatch? Have you found any evidence of that?

The reason I suspect it is that the ca issuer/subject/X509v3 Authority Key Identifier all contain the original hostname.
/etc/pki/katello/qpid_router_server.crt's Issuer is the old hostname; the subject is the new name, and the X509v3 Authority Key Identifier is the old name.

Did you observe that behavior after running katello-change-hostname? You may want to give it a try!