Bug #8370
closedRace condition for Candlepin during installation
Description
QE has seen an intermittent installer issue, mostly on RHEL 6.x installs where the installer shows:
[ERROR 2014-11-10 15:10:13 main] Repeating errors encountered during run: [ERROR 2014-11-10 15:10:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ERROR 2014-11-10 15:10:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ERROR 2014-11-10 15:10:13 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not evaluate: 500 Internal Server Error [ERROR 2014-11-10 15:10:13 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Failed to call refresh: 500 Internal Server Error [ERROR 2014-11-10 15:10:13 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: 500 Internal Server Error
Upon looking at the installer log file we then see this:
[DEBUG 2014-11-10 15:03:56 main] Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca' [DEBUG 2014-11-10 15:03:56 main] /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks [DEBUG 2014-11-10 15:03:56 main] Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12' [DEBUG 2014-11-10 15:03:56 main] Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12' [ WARN 2014-11-10 15:03:57 main] /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: executed successfully [DEBUG 2014-11-10 15:03:57 main] /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: The container Class[Certs::Candlepin] will propagate my refresh event [DEBUG 2014-11-10 15:03:57 main] Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca' [DEBUG 2014-11-10 15:03:57 main] Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca' [DEBUG 2014-11-10 15:03:57 main] /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Alias <katello-default-ca> does not exist
This then seems to lead to the following error:
[ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: PGError: ERROR: duplicate key value violates unique constraint "index_settings_on_name" [ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: : INSERT INTO "settings" ("category", "created_at", "default", "description", "name", "settings_type", "updated_at", "value") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" [ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: [ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Tasks: TOP => db:abort_if_pending_migrations => environment [ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: (See full trace by running task with --trace) [ WARN 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls. [ERROR 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ERROR 2014-11-10 15:05:13 main] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:104:in `fail' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:160:in `sync' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:583:in `refresh' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `send' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `process_callback' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:20:in `process_events' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:93:in `queued_events' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `each' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `queued_events' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:19:in `process_events' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:221:in `eval_resource' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `call' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138:in `evaluate' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:169:in `apply' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:112:in `as_logging_destination' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:168:in `apply' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:118:in `apply_catalog' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:117:in `apply_catalog' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:224:in `run_internal' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:132:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:131:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:297:in `apply_catalog' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:231:in `main' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:193:in `main' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:154:in `run_command' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:507:in `plugin_hook' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:488:in `exit_on_fail' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in `run' [ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in `execute' [ INFO 2014-11-10 15:05:13 main] /usr/bin/puppet:8 [ INFO 2014-11-10 15:05:14 main] Foreman::Rake[apipie:cache]: Scheduling refresh of Exec[foreman-rake-apipie:cache]
I'm not 100% sure that the candlepin issue is responsible for all other errors I mentioned here, but decided to mention them anyway.
Updated by Alex Wood almost 10 years ago
I believe the actual issue surfaces here:
/Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
The "unless" directive on the 'import client certificate into Candlepin keystore' task needs to check for the existence of the candlepin.jks file before attempting to run keytool on it. Adding "[ -f /etc/candlepin/certs/amqp/candlepin.jks ] &&" to the front of the "unless" value should fix the issue.
Updated by Eric Helms over 9 years ago
- Target version set to 62
- Translation missing: en.field_release set to 14
- Triaged changed from No to Yes
Updated by Eric Helms over 9 years ago
- Translation missing: en.field_release changed from 14 to 33
Updated by Eric Helms over 9 years ago
- Translation missing: en.field_release changed from 33 to 23
Updated by Eric Helms over 9 years ago
- Translation missing: en.field_release changed from 23 to 31
Updated by Eric Helms over 9 years ago
- Translation missing: en.field_release deleted (
31)
Updated by Eric Helms over 8 years ago
- Translation missing: en.field_release set to 114
Updated by John Mitsch almost 5 years ago
- Status changed from New to Rejected
- Target version deleted (
Katello Backlog)
Thanks for reporting this issue. This issue was created over 4 years ago and hasn't seen an update in 1 year. We are closing this in an effort to keep a realistic backlog. Please open up a new issue that includes a link to this issue if you feel this still needs to be addressed. We can then triage the new issue and reassess.