Plugins » Katello

We have 32 content views and 4 environments, which has generated 223 Puppet Environments. With this number of Puppet Environments promotions are hanging during the Dynflow finalization step "Content Update" and often failing with 406 errors. If we resume the task (2-3 times), it will eventually finish.

The following Dynflow step is where we encounter the error:

466: Actions::Katello::Foreman::ContentUpdate (error) [ 19.80s / 19.80s ]

Error:

ProxyAPI::ProxyException

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42 ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://xxxxxxx.xxxxxxx.biz:9090/puppet

---
- /usr/share/foreman/lib/proxy_api/puppet.rb:27:in `rescue in classes'
- /usr/share/foreman/lib/proxy_api/puppet.rb:21:in `classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:155:in `actual_classes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:78:in `new_classes_for'
- /usr/share/foreman/app/services/puppet_class_importer.rb:25:in `block in changes'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `each'
- /usr/share/foreman/app/services/puppet_class_importer.rb:24:in `changes'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/services/katello/puppet_class_importer_extensions.rb:22:in
  `update_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/foreman.rb:37:in
  `update_puppet_environment'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/katello/foreman/content_update.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:465:in
  `block (2 levels) in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:58:in
  `block (2 levels) in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:85:in
  `set_pulp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:44:in
  `as_pulp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:57:in
  `block in as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/katello/util/thread_session.rb:92:in
  `set_cp_config'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_cp_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:56:in
  `as_remote_user'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/remote_action.rb:32:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:30:in
  `with_progress_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action/progress.rb:22:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:35:in
  `with_locale'
- /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.0.0/app/lib/actions/middleware/keep_locale.rb:28:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:464:in
  `block in execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `block in with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:365:in
  `with_error_handling'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:463:in
  `execute_finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/action.rb:230:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
  `block (2 levels) in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract.rb:152:in
  `with_meta_calculation'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
  `block in execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
  `open_action'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:72:in
  `run_step'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:57:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `block in run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `all?'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:64:in
  `run_in_sequence'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:53:in
  `dispatch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:28:in
  `block (2 levels) in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:26:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:16:in
  `pass'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware.rb:41:in
  `finalize_phase'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/stack.rb:22:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/middleware/world.rb:30:in
  `execute'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:27:in
  `block in finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/transaction_adapters/active_record.rb:5:in
  `transaction'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/sequential_manager.rb:24:in
  `finalize'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:23:in
  `block in on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in
  `block in assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `tap'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in
  `assigns'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in
  `match_value'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in
  `block in match'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `each'
- /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in
  `match'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/executors/parallel/worker.rb:17:in
  `on_message'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:82:in
  `on_envelope'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:72:in
  `receive'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block (2 levels) in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `loop'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `block in run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `catch'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:99:in
  `run'
- /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.3/lib/dynflow/micro_actor.rb:13:in
  `block in initialize'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `call'
- /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in
  `block in create_with_logging_context'

From foreman-proxy/proxy.log:

10.60.141.42 - - [13/Jan/2015 12:51:42] "GET /environments/KT_xxxxxxx_prod_centos_6_server_x86_64_cv__16/classes HTTP/1.1" 200 2 2.4716
I, [2015-01-13T12:51:43.053304 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:43.146193 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.108623 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.141426 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.143053 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.250840 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.252718 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.414303 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
E, [2015-01-13T12:51:45.414820 #63199] ERROR -- : Failed to show puppet classes: Error converting value for param 'modulepath': Error converting value for param 'basemodulepath': Could not find value for $confdir
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_dev_puppet_el5_x86_64_cv_42/classes HTTP/1.1" 406 162 2.3751
I, [2015-01-13T12:51:45.433601 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.522164 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
10.60.141.42 - - [13/Jan/2015 12:51:45] "GET /environments/KT_xxxxxxx_qa_rhel_5_8_server_x86_64_cv_69/classes HTTP/1.1" 200 2 2.7583
I, [2015-01-13T12:51:45.603873 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:45.702921 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.561855 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.662264 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.731477 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.748059 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2015-01-13T12:51:47.830031 #63199]  INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf