Plugins » Katello

QE has seen an intermittent installer issue, mostly on RHEL 6.x installs where the installer shows:

[ERROR 2014-11-10 15:10:13 main] Repeating errors encountered during run:
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not evaluate: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Failed to call refresh: 500 Internal Server Error
[ERROR 2014-11-10 15:10:13 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: 500 Internal Server Error

Upon looking at the installer log file we then see this:

[DEBUG 2014-11-10 15:03:56 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:56 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Keystore file does not exist: /etc/candlepin/certs/amqp/candlepin.jks
[DEBUG 2014-11-10 15:03:56 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[DEBUG 2014-11-10 15:03:56 main]  Executing 'openssl pkcs12 -export -name amqp-client -in /etc/pki/katello/certs/java-client.crt -inkey /etc/pki/katello/private/java-client.key -out /tmp/keystore.p12 -passout file:/etc/pki/katello/keystore_password-file && keytool -importkeystore -destkeystore /etc/candlepin/certs/amqp/candlepin.jks -srckeystore /tmp/keystore.p12 -srcstoretype pkcs12 -alias amqp-client -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -srcstorepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -noprompt && rm /tmp/keystore.p12'
[ WARN 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/returns: executed successfully
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]: The container Class[Certs::Candlepin] will propagate my refresh event
[DEBUG 2014-11-10 15:03:57 main]  Exec[import client certificate into Candlepin keystore](provider=posix): Executing check 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  Executing 'keytool -list -keystore /etc/candlepin/certs/amqp/candlepin.jks -storepass ke5ciHzwx58ddE2kB3UeykvusJbK4yzH -alias katello-default-ca'
[DEBUG 2014-11-10 15:03:57 main]  /Stage[main]/Certs::Candlepin/Exec[import client certificate into Candlepin keystore]/unless: keytool error: java.lang.Exception: Alias <katello-default-ca> does not exist

This then seems to lead to the following error:

[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: PGError: ERROR:  duplicate key value violates unique constraint "index_settings_on_name" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: : INSERT INTO "settings" ("category", "created_at", "default", "description", "name", "settings_type", "updated_at", "value") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: 
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Tasks: TOP => db:abort_if_pending_migrations => environment
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: (See full trace by running task with --trace)
[ WARN 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-10 15:05:13 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:104:in `fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:160:in `sync'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/type/exec.rb:583:in `refresh'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `send'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:101:in `process_callback'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:20:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:93:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `each'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:92:in `queued_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb:19:in `process_events'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:221:in `eval_resource'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `call'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:147:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:138:in `evaluate'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:169:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:112:in `as_logging_destination'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:168:in `apply'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:118:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:117:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:224:in `run_internal'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:132:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:131:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:297:in `apply_catalog'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:231:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/context.rb:64:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet.rb:244:in `override'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:193:in `main'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:154:in `run_command'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:507:in `plugin_hook'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:488:in `exit_on_fail'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:381:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:146:in `run'
[ INFO 2014-11-10 15:05:13 main] /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:92:in `execute'
[ INFO 2014-11-10 15:05:13 main] /usr/bin/puppet:8
[ INFO 2014-11-10 15:05:14 main]  Foreman::Rake[apipie:cache]: Scheduling refresh of Exec[foreman-rake-apipie:cache]

I'm not 100% sure that the candlepin issue is responsible for all other errors I mentioned here, but decided to mention them anyway.