Project

General

Profile

Actions

Bug #18129

closed

qpidd connection error during upgrade-puppet

Added by Klaas D over 7 years ago. Updated over 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Difficulty:
medium
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Hi,
during foreman-installer --upgrade-puppet I get an error because qpidd is not yet up when installer hits Exec[create candlepin qpid exchange].

foreman-installer --upgrade-puppet
[...]
Puppet 3 to 4 upgrade initialization complete, continuing with installation
 qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://server.tld:5671' add exchange topic event --durable returned 1 instead of one of [0]                                                                                                                                               
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:230:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:246:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:386:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:385:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:163:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:154:in `evaluate'
/usr/share/gems/gems/kafo-0.9.8/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:31:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:222:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:155:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:142:in `as_logging_destination'
/usr/share/gems/gems/kafo-0.9.8/modules/kafo_configure/lib/kafo/puppet/report_wrapper.rb:34:in `method_missing'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:221:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:171:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:170:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:343:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:350:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:274:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/returns: change from notrun to 0 failed: qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://server.tld:5671' add exchange topic event --durable returned 1 instead of one of [0]
Installing             Done                                               [100%] [.........................................................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services..
[...]

I looked at the code and qpidd doesn't seem to be fully up when the unless condition is executed. I can work around the issue by inserting a sleep 60 before running "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' exchanges ${certs::candlepin_qpid_exchange}", but I guess that should be properly fixed in systemd unit file of qpidd or in the puppet service definition of qpidd.

in /usr/share/katello-installer-base/modules/certs/manifests/candlepin.pp:
    exec { 'create candlepin qpid exchange':
      command => "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' add exchange topic ${certs::candlepin_qpid_exchange} --durable",
      unless  => "qpid-config --ssl-certificate ${client_cert} --ssl-key ${client_key} -b 'amqps://${::fqdn}:5671' exchanges ${certs::candlepin_qpid_exchange}",
      require => Service['qpidd'],
    } ~>

Greetings
Klaas


Files

katello.log katello.log 2.17 MB Klaas D, 02/03/2017 08:21 AM
Actions #1

Updated by Justin Sherrill over 7 years ago

  • Category changed from Installer to 89
Actions #2

Updated by Justin Sherrill over 7 years ago

  • Status changed from New to Need more information

Do you have the installer logs from this upgrade attempt?

Actions #3

Updated by Klaas D over 7 years ago

Sorry, I don't think I kept them - I have another machine to upgrade tomorrow, I'll look if it throws the same error.

Actions #4

Updated by Klaas D over 7 years ago

I'm attaching the full log of another server having the same issue

Actions #5

Updated by Justin Sherrill over 7 years ago

  • Status changed from Need more information to Assigned
  • Assignee set to Eric Helms
  • Difficulty set to medium

Thanks klass that helped a lot, looks like its possible that https://github.com/Katello/puppet-service_wait/blob/877ff3ba4b8151bb49aac040d13b940950a504f5/bin/service-wait#L91

is never actually sleeping for qpidd. Needs a bit more investigation on our part.

Actions #6

Updated by Eric Helms about 7 years ago

  • Status changed from Assigned to Needs design
Actions #7

Updated by John Mitsch about 7 years ago

  • translation missing: en.field_release set to 228
Actions #8

Updated by Eric Helms almost 7 years ago

  • translation missing: en.field_release changed from 228 to 258
Actions #9

Updated by Justin Sherrill almost 7 years ago

  • translation missing: en.field_release changed from 258 to 267
Actions #10

Updated by Justin Sherrill almost 7 years ago

  • translation missing: en.field_release changed from 267 to 281
Actions #11

Updated by Justin Sherrill almost 7 years ago

  • translation missing: en.field_release changed from 281 to 286
Actions #12

Updated by Eric Helms over 6 years ago

  • translation missing: en.field_release changed from 286 to 295
Actions #13

Updated by Jonathon Turel over 4 years ago

  • Status changed from Needs design to Rejected
  • Target version deleted (Katello 3.4.6)
  • Triaged set to Yes

Since we are long past Katello 3.4 I am closing this issue. Please open a new issue if you're still experiencing this problem.

Actions

Also available in: Atom PDF