Project

General

Profile

Bug #8370

Race condition for Candlepin during installation

Added by Og Maciel over 4 years ago. Updated about 1 year ago.

Status:
New
Priority:
High
Assignee:
-
Category:
Installer
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

QE has seen an intermittent installer issue, mostly on RHEL 6.x installs where the installer shows:

[ERROR 2014-11-10 15:10:13 main] Repeating errors encountered during run:
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not evaluate: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Failed to call refresh: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: 500 Internal Server Error

Upon looking at the installer log file we then see this:

[DEBUG 2014-11-10 15:03:56 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:56 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
[DEBUG 2014-11-10 15:03:56 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[DEBUG 2014-11-10 15:03:56 main]  Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[ WARN 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: executed successfully
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: The container Class[Certs::Candlepin] will propagate my refresh event
[DEBUG 2014-11-10 15:03:57 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Alias <katello-default-ca> does not exist

This then seems to lead to the following error:

[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: PGError: ERROR:  duplicate key value violates unique constraint "index_settings_on_name" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: : INSERT INTO "settings" ("category", "created_at", "default", "description", "name", "settings_type", "updated_at", "value") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Tasks: TOP => db:abort_if_pending_migrations => environment
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: (See full trace by running task with --trace)
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:104:in `fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:160:in `sync'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:583:in `refresh'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `send'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `process_callback'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:20:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:93:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `each'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:19:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:221:in `eval_resource'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `call'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:169:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:112:in `as_logging_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:168:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:118:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:117:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:224:in `run_internal'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:132:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:131:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:297:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:231:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:193:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:154:in `run_command'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:507:in `plugin_hook'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:488:in `exit_on_fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in `execute'
[ INFO 2014-11-10 15:05:13 main] /usr/bin/puppet:8
[ INFO 2014-11-10 15:05:14 main]  Foreman::Rake[apipie:cache]: Scheduling refresh of Exec[foreman-rake-apipie:cache]

I'm not 100% sure that the candlepin issue is responsible for all other errors I mentioned here, but decided to mention them anyway.

History

#1 Updated by Alex Wood over 4 years ago

I believe the actual issue surfaces here:

/Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks

The "unless" directive on the 'import client certificate into Candlepin keystore' task needs to check for the existence of the candlepin.jks file before attempting to run keytool on it. Adding "[ -f /etc/candlepin/certs/amqp/candlepin.jks ] &&" to the front of the "unless" value should fix the issue.

#2 Updated by Eric Helms over 4 years ago

  • Target version set to 62
  • Legacy Backlogs Release (now unused) set to 14
  • Triaged changed from No to Yes

#3 Updated by Eric Helms over 4 years ago

  • Target version changed from 62 to 63

#4 Updated by Eric Helms over 4 years ago

  • Target version changed from 63 to 66

#5 Updated by Eric Helms over 4 years ago

  • Legacy Backlogs Release (now unused) changed from 14 to 33

#6 Updated by Eric Helms over 4 years ago

  • Legacy Backlogs Release (now unused) changed from 33 to 23

#7 Updated by Eric Helms over 4 years ago

  • Legacy Backlogs Release (now unused) changed from 23 to 31

#8 Updated by Eric Helms over 4 years ago

  • Target version deleted (66)

#9 Updated by Eric Helms over 4 years ago

  • Legacy Backlogs Release (now unused) deleted (31)

#10 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) set to 114

Also available in: Atom PDF