Project

General

Profile

Bug #11285

Katello 2.3 Repo Sync issue

Added by Duncan Innes almost 7 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
High
Category:
-
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Clean install of base OS - CentOS 7.1
Clean install of Katello 2.3 RC

katello-installer run without any options to keep everything vanilla.

After creating a product (CentOS) and 3 repos (CentOS, CentOS Updates, CentOS Extras), then syncing these repos, I end up in a state where 2 out of the 3 repos hang during sync and I no longer have the 'event' Exchange when listing the Exchanges via qpid.

candlepin.tgz candlepin.tgz 12.7 KB /var/log/candleping tarball Duncan Innes, 08/04/2015 10:13 AM
katello-installer.log katello-installer.log 1.67 MB /var/log/katello-installer/katello-installer.log Duncan Innes, 08/04/2015 10:14 AM

Related issues

Related to Katello - Bug #10574: pulp-katello plugin depends on qpidClosed2015-05-21

Associated revisions

Revision fcf0c6b2 (diff)
Added by Stephen Benjamin almost 7 years ago

fixes #11285 - require qpid-cpp-server-linearstore

Revision 488db727
Added by Stephen Benjamin almost 7 years ago

Merge pull request #31 from stbenjam/11285

fixes #11285 - require qpid-cpp-server-linearstore

Revision 8b177a39 (diff)
Added by Stephen Benjamin over 6 years ago

fixes #11285 - require qpid-cpp-server-linearstore

Revision 8181cabe
Added by Stephen Benjamin over 6 years ago

Merge pull request #26 from stbenjam/11285

fixes #11285 - require qpid-cpp-server-linearstore

History

#2 Updated by Duncan Innes almost 7 years ago

[root@foreman log]# rpm -qa | grep katello
katello-installer-2.3.1-3.el7.noarch
pulp-katello-0.4-2.el7.noarch
katello-server-ca-1.0-1.noarch
katello-debug-2.3.0-4.el7.noarch
katello-repos-2.3.0-7.el7.noarch
katello-service-2.3.0-4.el7.noarch
katello-common-2.3.0-4.el7.noarch
katello-certs-tools-2.3.0-4.el7.noarch
katello-2.3.0-4.el7.noarch
katello-installer-base-2.3.1-3.el7.noarch
ruby193-rubygem-katello-2.3.0-3.el7.noarch
katello-default-ca-1.0-1.noarch
rubygem-hammer_cli_katello-0.0.15-1.el7.noarch
katello-selinux-2.2.1-1.el7.noarch
[root@foreman log]#

#3 Updated by Duncan Innes almost 7 years ago

The system is a libvirt guest running on a laptop installed with RHEL 7.1 and fully updated.

The guest system is kickstarted with a basic install of CentOS 7.1.1503 which is served from httpd running on the host OS. The kickstart has the following installation lines in the %post section:

echo "Modify NTP settings"

sed -i -e 's/0\..*\.pool\.ntp\.org/192.168.122.1/g' /etc/chrony.conf
sed -i -e '/org iburst/d' /etc/chrony.conf

echo "yum update"

yum -y update

echo "Install Katello repos"

#yum -y localinstall http://fedorapeople.org/groups/katello/releases/yum/2.2/katello/RHEL/7Server/x86_64/katello-repos-latest.rpm
#yum -y localinstall http://yum.theforeman.org/releases/1.8/el7/x86_64/foreman-release.rpm
#yum -y localinstall http://mirror.pnl.gov/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
#yum -y localinstall http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
#yum -y localinstall https://www.softwarecollections.org/en/scls/rhscl/v8314/epel-7-x86_64/download/rhscl-v8314-epel-7-x86_64.noarch.rpm
#yum -y localinstall https://www.softwarecollections.org/en/scls/rhscl/ruby193/epel-7-x86_64/download/rhscl-ruby193-epel-7-x86_64.noarch.rpm

yum -y localinstall http://fedorapeople.org/groups/katello/releases/yum/2.3/katello/RHEL/7Server/x86_64/katello-repos-latest.rpm
yum -y localinstall http://yum.theforeman.org/releases/1.9/el7/x86_64/foreman-release.rpm
yum -y localinstall http://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
yum -y localinstall http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum -y install foreman-release-scl

echo "Install Katello packages"

yum -y install katello

%end

Once the install is complete, the guest is booted, I log ssh in as root and run

[root@foreman ~]# katello-installer

without any options. Installation logs are uploaded, but I'll also upload the kickstart logs as there are some warnings about the tomcat user not being available when gutterball is installed.

Once Katello is installed, I log in and change the admin password to something I can remember.

I have then created a product called CentOS and 3 repos within that: CentOS 7, CentOS 7 Updates, and CentOS 7 Extras - all using the www.mirrorservice.org/sites/mirror.centos.org/7.1.1503(insert_relevant_yum_directory_here) as their source.

Initial sync of the 3 repos was mainly successful - each synched the majority of packages down, but left a handful behind. The second sync completed the repositories, leaving no packages unaccounted for. The third sync hung on 2 of the repos and I can no longer see the Product page in the Katello WebUI. Not sure of the hammer command to list products & repos as yet.

It appears that the 'event' Exchange is now missing on my system:

[root@foreman ~]# qpid-config -b amqps://localhost:5671 --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes ==================================================
direct --replicate=none
direct C.dq --durable
direct amq.direct --durable --replicate=none
fanout amq.fanout --durable --replicate=none
headers amq.match --durable --replicate=none
topic amq.topic --durable --replicate=none
direct celery --durable
fanout celery.pidbox
topic celeryev --durable
direct qmf.default.direct --replicate=none
topic qmf.default.topic --replicate=none
topic qpid.management --replicate=none
direct resource_manager --durable
[root@foreman ~]#

#4 Updated by Stephen Benjamin almost 7 years ago

  • Priority changed from Normal to High

#5 Updated by Stephen Benjamin almost 7 years ago

  • Category set to 143
  • Triaged changed from No to Yes

#6 Updated by Stephen Benjamin almost 7 years ago

  • Category deleted (143)

#7 Updated by Alex Wood almost 7 years ago

I've looked at this a bit with Duncan and I'm not sure why the event exchange is not being created. The installer log shows:

[DEBUG 2015-08-02 19:30:48 main]  Exec[create candlepin qpid exchange](provider=posix): Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://foreman.rhel7.local:5671' add exchange topic event --durable'
[DEBUG 2015-08-02 19:30:48 main]  Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://foreman.rhel7.local:5671' add exchange topic event --durable'
[ WARN 2015-08-02 19:30:48 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/returns: executed successfully

It's a bit strange that a command that "executed successfully" would be logged at WARN level, but aside from that general observation, I don't have a specific suggestion. I do see Puppet verify that the exchange exists later:

[DEBUG 2015-08-02 19:30:48 main]  Exec[create candlepin qpid exchange](provider=posix): Executing check 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://foreman.rhel7.local:5671' exchanges event'
[DEBUG 2015-08-02 19:30:48 main]  Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://foreman.rhel7.local:5671' exchanges event'
[DEBUG 2015-08-02 19:30:48 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/unless: Type      Exchange Name  Attributes
[DEBUG 2015-08-02 19:30:48 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/unless: =============================================
[DEBUG 2015-08-02 19:30:48 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/unless: topic     event          --durable
[ WARN 2015-08-02 19:30:48 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]: Triggered 'refresh' from 4 events

Is the qpid-cpp-server-store package getting installed?

#8 Updated by Peter Verbist almost 7 years ago

No, the qpid-cpp-server-store package is missing.
[vagrant@katello ~]$ rpm -qa | grep qpid-cpp
qpid-cpp-client-0.30-7.proton.0.9.el7.x86_64
qpid-cpp-client-devel-0.30-7.proton.0.9.el7.x86_64
qpid-cpp-server-linearstore-0.30-7.proton.0.9.el7.x86_64
qpid-cpp-server-0.30-7.proton.0.9.el7.x86_64

When I try to install the package I receive:
Package qpid-cpp-server-store-0.32-3.el7.x86_64 is obsoleted by qpid-cpp-server-linearstore-0.30-7.proton.0.9.el7.x86_64 which is already installed

#9 Updated by Alex Wood almost 7 years ago

I will wager that qpid-cpp-server-store/linearstore missing on Duncan's machine is the problem. According to a fellow I spoke with, "If you create an exchange and restart qpidd, the exchange will be gone (if there's no store installed)."

#10 Updated by Duncan Innes almost 7 years ago

Peter is right. qpid-cpp-server-store is missing.

When I try to install it, I'm told the same thing (that it's been obsoleted by linearstore), but I don't have that package installed on my server.

#11 Updated by Stephen Benjamin almost 7 years ago

  • Assignee set to Stephen Benjamin

qpid-cpp-server-linearstore was mistanekly removed as a dep of pulp-katello.

https://github.com/Katello/pulp-katello-plugins/commit/befa66c09bce0df73131dc2c25cbe45eee6fc1cc

Will fix in the next RC. Thanks guys!

#12 Updated by Stephen Benjamin almost 7 years ago

  • Related to Bug #10574: pulp-katello plugin depends on qpid added

#13 Updated by Peter Verbist almost 7 years ago

Installed from scratch and ran qpid-config command:
qpid-config -b amqps://localhost:5671 --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes ==================================================
direct --replicate=none
direct C.dq --durable
direct amq.direct --durable --replicate=none
fanout amq.fanout --durable --replicate=none
headers amq.match --durable --replicate=none
topic amq.topic --durable --replicate=none
direct celery --durable
fanout celery.pidbox
topic celeryev --durable
topic event --durable
direct qmf.default.direct --replicate=none
topic qmf.default.topic --replicate=none
topic qpid.management --replicate=none
direct resource_manager --durable

Seems to be quiet normal. Stop/start katello using the katello-service and issue the same command:

qpid-config -b amqps://localhost:5671 --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key exchanges
Type Exchange Name Attributes ==================================================
direct --replicate=none
direct C.dq --durable
direct amq.direct --durable --replicate=none
fanout amq.fanout --durable --replicate=none
headers amq.match --durable --replicate=none
topic amq.topic --durable --replicate=none
direct celery --durable
fanout celery.pidbox
topic celeryev --durable
direct qmf.default.direct --replicate=none
topic qmf.default.topic --replicate=none
topic qpid.management --replicate=none
direct resource_manager --durable

--> the exchange name called event is missing.

#14 Updated by Stephen Benjamin almost 7 years ago

Peter, install qpid-cpp-server-linearstore and run that command again and it'll fix it.

We'll correct this in 2.3 RC2.

#15 Updated by The Foreman Bot almost 7 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/5399 added
  • Pull request deleted ()

#16 Updated by Peter Verbist almost 7 years ago

Thank you Stephen, this works fine!

#17 Updated by Eric Helms almost 7 years ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF