Bug #18129
closedqpidd connection error during upgrade-puppet
Description
Hi,
during foreman-installer --upgrade-puppet I get an error because qpidd is not yet up when installer hits Exec[create candlepin qpid exchange].
foreman-installer --upgrade-puppet [...] Puppet 3 to 4 upgrade initialization complete, continuing with installation qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://server.tld:5671' add exchange topic event --durable returned 1 instead of one of [0] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:230:in `apply' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:246:in `eval_resource' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block (2 levels) in evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:386:in `block in thinmark' /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:385:in `thinmark' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block in evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:154:in `evaluate' /usr/share/gems/gems/kafo-0.9.8/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:31:in `evaluate_with_trigger' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:222:in `block in apply' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:155:in `with_destination' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:142:in `as_logging_destination' /usr/share/gems/gems/kafo-0.9.8/modules/kafo_configure/lib/kafo/puppet/report_wrapper.rb:34:in `method_missing' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:221:in `apply' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:171:in `block in apply_catalog' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `block in benchmark' /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:170:in `apply_catalog' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:343:in `run_internal' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:350:in `apply_catalog' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:274:in `block in main' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute' /opt/puppetlabs/puppet/bin/puppet:5:in `<main>' /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/returns: change from notrun to 0 failed: qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://server.tld:5671' add exchange topic event --durable returned 1 instead of one of [0] Installing Done [100%] [.........................................................................................................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/katello.log Upgrade Step: restart_services.. [...]
I looked at the code and qpidd doesn't seem to be fully up when the unless condition is executed. I can work around the issue by inserting a sleep 60 before running "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' exchanges ${certs::candlepin_qpid_exchange}", but I guess that should be properly fixed in systemd unit file of qpidd or in the puppet service definition of qpidd.
in /usr/share/katello-installer-base/modules/certs/manifests/candlepin.pp: exec { 'create candlepin qpid exchange': command => "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' add exchange topic ${certs::candlepin_qpid_exchange} --durable", unless => "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' exchanges ${certs::candlepin_qpid_exchange}", require => Service['qpidd'], } ~>
Greetings
Klaas
Files
Updated by Justin Sherrill over 8 years ago
- Category changed from Installer to 89
Updated by Justin Sherrill over 8 years ago
- Status changed from New to Need more information
Do you have the installer logs from this upgrade attempt?
Updated by Klaas D over 8 years ago
Sorry, I don't think I kept them - I have another machine to upgrade tomorrow, I'll look if it throws the same error.
Updated by Klaas D over 8 years ago
- File katello.log katello.log added
I'm attaching the full log of another server having the same issue
Updated by Justin Sherrill about 8 years ago
- Status changed from Need more information to Assigned
- Assignee set to Eric Helms
- Difficulty set to medium
Thanks klass that helped a lot, looks like its possible that https://github.com/Katello/puppet-service_wait/blob/877ff3ba4b8151bb49aac040d13b940950a504f5/bin/service-wait#L91
is never actually sleeping for qpidd. Needs a bit more investigation on our part.
Updated by Eric Helms about 8 years ago
- Status changed from Assigned to Needs design
Updated by John Mitsch about 8 years ago
- Translation missing: en.field_release set to 228
Updated by Eric Helms almost 8 years ago
- Translation missing: en.field_release changed from 228 to 258
Updated by Justin Sherrill almost 8 years ago
- Translation missing: en.field_release changed from 258 to 267
Updated by Justin Sherrill almost 8 years ago
- Translation missing: en.field_release changed from 267 to 281
Updated by Justin Sherrill almost 8 years ago
- Translation missing: en.field_release changed from 281 to 286
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 286 to 295
Updated by Jonathon Turel over 5 years ago
- Status changed from Needs design to Rejected
- Target version deleted (
Katello 3.4.6) - Triaged set to Yes
Since we are long past Katello 3.4 I am closing this issue. Please open a new issue if you're still experiencing this problem.