Project

General

Profile

Actions
Copy link

Bug #8370

closed

Race condition for Candlepin during installation

Added by Og Maciel about 10 years ago. Updated over 5 years ago.

Status:
Rejected
Priority:
High
Assignee:
-
Category:
Installer
Target version:
-
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

QE has seen an intermittent installer issue, mostly on RHEL 6.x installs where the installer shows:

[ERROR 2014-11-10 15:10:13 main] Repeating errors encountered during run:
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not evaluate: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Failed to call refresh: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: 500 Internal Server Error

Upon looking at the installer log file we then see this:

[DEBUG 2014-11-10 15:03:56 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:56 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
[DEBUG 2014-11-10 15:03:56 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[DEBUG 2014-11-10 15:03:56 main]  Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[ WARN 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: executed successfully
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: The container Class[Certs::Candlepin] will propagate my refresh event
[DEBUG 2014-11-10 15:03:57 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Alias <katello-default-ca> does not exist

This then seems to lead to the following error:

[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: PGError: ERROR:  duplicate key value violates unique constraint "index_settings_on_name" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: : INSERT INTO "settings" ("category", "created_at", "default", "description", "name", "settings_type", "updated_at", "value") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Tasks: TOP => db:abort_if_pending_migrations => environment
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: (See full trace by running task with --trace)
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:104:in `fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:160:in `sync'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:583:in `refresh'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `send'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `process_callback'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:20:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:93:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `each'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:19:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:221:in `eval_resource'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `call'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:169:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:112:in `as_logging_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:168:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:118:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:117:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:224:in `run_internal'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:132:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:131:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:297:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:231:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:193:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:154:in `run_command'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:507:in `plugin_hook'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:488:in `exit_on_fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in `execute'
[ INFO 2014-11-10 15:05:13 main] /usr/bin/puppet:8
[ INFO 2014-11-10 15:05:14 main]  Foreman::Rake[apipie:cache]: Scheduling refresh of Exec[foreman-rake-apipie:cache]

I'm not 100% sure that the candlepin issue is responsible for all other errors I mentioned here, but decided to mention them anyway.

Actions
Copy link
 #1

Updated by Alex Wood about 10 years ago

I believe the actual issue surfaces here:

/Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks

The "unless" directive on the 'import client certificate into Candlepin keystore' task needs to check for the existence of the candlepin.jks file before attempting to run keytool on it. Adding "[ -f /etc/candlepin/certs/amqp/candlepin.jks ] &&" to the front of the "unless" value should fix the issue.

Actions
Copy link
 #2

Updated by Eric Helms about 10 years ago

  • Target version set to 62
  • Translation missing: en.field_release set to 14
  • Triaged changed from No to Yes
Actions
Copy link
 #3

Updated by Eric Helms almost 10 years ago

  • Target version changed from 62 to 63
Actions
Copy link
 #4

Updated by Eric Helms almost 10 years ago

  • Target version changed from 63 to 66
Actions
Copy link
 #5

Updated by Eric Helms almost 10 years ago

  • Translation missing: en.field_release changed from 14 to 33
Actions
Copy link
 #6

Updated by Eric Helms almost 10 years ago

  • Translation missing: en.field_release changed from 33 to 23
Actions
Copy link
 #7

Updated by Eric Helms almost 10 years ago

  • Translation missing: en.field_release changed from 23 to 31
Actions
Copy link
 #8

Updated by Eric Helms almost 10 years ago

  • Target version deleted (66)
Actions
Copy link
 #9

Updated by Eric Helms almost 10 years ago

  • Translation missing: en.field_release deleted (31)
Actions
Copy link
 #10

Updated by Eric Helms about 9 years ago

  • Translation missing: en.field_release set to 114
Actions
Copy link
 #11

Updated by John Mitsch over 5 years ago

  • Status changed from New to Rejected
  • Target version deleted (Katello Backlog)

Thanks for reporting this issue. This issue was created over 4 years ago and hasn't seen an update in 1 year. We are closing this in an effort to keep a realistic backlog. Please open up a new issue that includes a link to this issue if you feel this still needs to be addressed. We can then triage the new issue and reassess.

Actions
Copy link

Also available in: Atom PDF