Plugins » foreman-tasks

I started getting these today and I have no idea how to resolve them. I've tried cleaning up some failed tasks (duplicate key issues that resolve themselves if I tell the task to resume) and I've tried to clean up the dependent tasks as indicated in the logs. After a few cleanups, with no additional errors, I started getting this for all tasks I've tried to start :

cannot accept execution_plan_id:1da9f49c-0d88-491e-a653-fdf7c047efe0 core is terminating

Dynflow::Error
cannot accept execution_plan_id:1da9f49c-0d88-491e-a653-fdf7c047efe0 core is terminating
app/controllers/concerns/application_shared.rb:13:in `set_timezone'
app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'

foreman/production.log :

2015-06-24 17:12:00 [I] Processing by Api::V2::ReportsController#create as JSON
2015-06-24 17:12:00 [I] Parameters: {"report"=>"[FILTERED]", "apiv"=>"v2"}
2015-06-24 17:12:00 [I] processing report for test.example.com
2015-06-24 17:12:01 [I] Imported report for test.example.com in 0.16 seconds
2015-06-24 17:12:01 [I] Completed 201 Created in 173ms (Views: 1.4ms | ActiveRecord: 0.0ms)
2015-06-24 17:12:24 [I] Processing by DashboardController#index as HTML
2015-06-24 17:12:24 [I] Redirected to https://katello.example.com/
2015-06-24 17:12:24 [I] Filter chain halted as :require_ssl rendered or redirected
2015-06-24 17:12:24 [I] Completed 302 Found in 1ms (ActiveRecord: 0.0ms)
2015-06-24 17:12:38 [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2015-06-24 17:12:38 [I] Parameters: {"searches"=>[{"type"=>"resource", "resource_type"=>"Katello::Product", "resource_id"=>"4", "search_id"=>"1"}], "task"=>{}}
2015-06-24 17:12:40 [I] Completed 200 OK in 1686ms (Views: 276.9ms | ActiveRecord: 11.3ms)
2015-06-24 17:12:42 [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2015-06-24 17:12:42 [I] Parameters: {"searches"=>[{"type"=>"resource", "resource_type"=>"Katello::Product", "resource_id"=>"4", "search_id"=>"1"}], "task"=>{}}
2015-06-24 17:12:44 [I] Completed 200 OK in 1806ms (Views: 270.7ms | ActiveRecord: 11.0ms)
2015-06-24 17:12:45 [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2015-06-24 17:12:45 [I] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"89874fe4-6031-4493-854c-a3f1f119e258", "search_id"=>"2"}], "task"=>{}}
2015-06-24 17:12:45 [I] Completed 200 OK in 99ms (Views: 15.3ms | ActiveRecord: 1.6ms)
2015-06-24 17:12:51 [I] Processing by ForemanTasks::TasksController#show as HTML
2015-06-24 17:12:51 [I] Parameters: {"id"=>"89874fe4-6031-4493-854c-a3f1f119e258"}
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/_details.html.erb (582.4ms)
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/_running_steps.html.erb (0.1ms)
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/_errors.html.erb (28.3ms)
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/_raw.html.erb (39.0ms)
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/_locks.html.erb (8.7ms)
2015-06-24 17:12:51 [I] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.13/app/views/foreman_tasks/tasks/show.html.erb within layouts/application (810.7ms)
2015-06-24 17:12:51 [I] Rendered home/_user_dropdown.html.erb (1.0ms)
2015-06-24 17:12:51 [I] Read fragment views/tabs_and_title_records-4 (0.1ms)
2015-06-24 17:12:51 [I] Rendered home/_topbar.html.erb (1.7ms)
2015-06-24 17:12:51 [I] Rendered layouts/base.html.erb (2.8ms)
2015-06-24 17:12:51 [I] Completed 200 OK in 825ms (Views: 805.2ms | ActiveRecord: 10.9ms)
2015-06-24 17:13:00 [I] Processing by ForemanTasks::TasksController#resume as HTML
2015-06-24 17:13:00 [I] Parameters: {"authenticity_token"=>"x3Mu1BG+rja+sxAI+stj/40vKkTIgEYkz7IxUFZGLDA=", "id"=>"89874fe4-6031-4493-854c-a3f1f119e258"}
2015-06-24 17:13:00 [E] cannot accept execution_plan_id:1da9f49c-0d88-491e-a653-fdf7c047efe0 core is terminating (Dynflow::Error)
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:54:in `track_execution_plan'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:23:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:21:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:80:in `block in on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/future.rb:75:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/future.rb:75:in `evaluate_to'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:80:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2015-06-24 17:13:00 [E] cannot accept execution_plan_id:1da9f49c-0d88-491e-a653-fdf7c047efe0 core is terminating (Dynflow::Error)
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:54:in `track_execution_plan'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:23:in `block in on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/executors/parallel/core.rb:21:in `on_message'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:80:in `block in on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/future.rb:75:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/future.rb:75:in `evaluate_to'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:80:in `on_envelope'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:99:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.8/lib/dynflow/micro_actor.rb:13:in `block in initialize'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2015-06-24 17:13:00 [W] Operation FAILED: cannot accept execution_plan_id:1da9f49c-0d88-491e-a653-fdf7c047efe0 core is terminating
2015-06-24 17:13:00 [I] Rendered common/500.html.erb within layouts/application (2.8ms)
2015-06-24 17:13:00 [I] Rendered layouts/base.html.erb (0.9ms)
2015-06-24 17:13:00 [I] Completed 500 Internal Server Error in 631ms (Views: 5.0ms | ActiveRecord: 1.6ms)
2015-06-24 17:13:24 [I] Processing by DashboardController#index as HTML
2015-06-24 17:13:24 [I] Redirected to https://katello.example.com/
2015-06-24 17:13:24 [I] Filter chain halted as :require_ssl rendered or redirected
2015-06-24 17:13:24 [I] Completed 302 Found in 1ms (ActiveRecord: 0.0ms)
2015-06-24 17:14:24 [I] Processing by DashboardController#index as HTML
2015-06-24 17:14:24 [I] Redirected to https://katello.example.com/
2015-06-24 17:14:24 [I] Filter chain halted as :require_ssl rendered or redirected
2015-06-24 17:14:24 [I] Completed 302 Found in 1ms (ActiveRecord: 0.0ms)