Bug #18171: Tasks (Play Ansible roles and Remote Execution) hang infinitly on Foreman 13.3/Tasks 0.8.2 - foreman-tasks - Foreman

Actions

Copy link

Bug #18171

closed

Tasks (Play Ansible roles and Remote Execution) hang infinitly on Foreman 13.3/Tasks 0.8.2

Added by Dirk Götz about 7 years ago. Updated almost 7 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Target version:

Difficulty:

Triaged:

Bugzilla link:

Pull request:

Fixed in Releases:

Found in Releases:

Foreman - 1.13.3

Red Hat JIRA:

Description

I started to test the Ansible-Plugin with Foreman upgraded to 13.3. Integration via callback and dynamic inventory works fine, also import and assignment of roles, but when I hit the "Play Ansible roles" button the tasks is not executed (stays infinitly on planned/pending). So I first thought it is a bug there, but trying Remote Execution gave me the same problem. So it seems like upgrade to 13.1 or 13.3 broke the tasks plugin for me. System is CentOS 7 (fully updated), Foreman Tasks is 0.8.2.

I do not see an error in the production.log or any entry in another logfile.

2017-01-20 09:30:37 57a33314 [app] [I] Started GET "/ansible/hosts/1/play_roles" for 192.168.142.1 at 2017-01-20 09:30:37 +0100
2017-01-20 09:30:37 57a33314 [app] [I] Processing by HostsController#play_roles as HTML
2017-01-20 09:30:37 57a33314 [app] [I]   Parameters: {"id"=>"1"}
2017-01-20 09:30:38 57a33314 [app] [I] Redirected to https://foreman.localdomain/foreman_tasks/tasks/9aa6be25-3aeb-455f-88fd-ace0eb6f8e21
2017-01-20 09:30:38 57a33314 [app] [I] Completed 302 Found in 617ms (ActiveRecord: 59.0ms)
2017-01-20 09:30:38 57a33314 [app] [I] Started GET "/foreman_tasks/tasks/9aa6be25-3aeb-455f-88fd-ace0eb6f8e21" for 192.168.142.1 at 2017-01-20 09:30:38 +0100
2017-01-20 09:30:38 57a33314 [app] [I] Processing by ForemanTasks::TasksController#show as HTML
2017-01-20 09:30:38 57a33314 [app] [I]   Parameters: {"id"=>"9aa6be25-3aeb-455f-88fd-ace0eb6f8e21"}
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_details.html.erb (46.5ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_running_steps.html.erb (1.6ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_errors.html.erb (1.1ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_raw.html.erb (3.6ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_locks.html.erb (1.9ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/show.html.erb within layouts/application (68.9ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered layouts/_application_content.html.erb (0.9ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_user_dropdown.html.erb (17.9ms)
2017-01-20 09:30:38 57a33314 [app] [I] Read fragment views/tabs_and_title_records-4 (0.1ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_org_switcher.html.erb (0.6ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (6.2ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (1.4ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (5.7ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (4.3ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (3.0ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_submenu.html.erb (3.8ms)
2017-01-20 09:30:38 57a33314 [app] [I] Write fragment views/tabs_and_title_records-4 (0.6ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_topbar.html.erb (180.3ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered layouts/base.html.erb (184.1ms)
2017-01-20 09:30:38 57a33314 [app] [I] Completed 200 OK in 270ms (Views: 262.1ms | ActiveRecord: 2.5ms)
2017-01-20 09:30:38 57a33314 [app] [I] Started GET "/ansible/hosts/1/play_roles" for 192.168.142.1 at 2017-01-20 09:30:38 +0100
2017-01-20 09:30:38 57a33314 [app] [I] Processing by HostsController#play_roles as HTML
2017-01-20 09:30:38 57a33314 [app] [I]   Parameters: {"id"=>"1"}
2017-01-20 09:30:38 57a33314 [app] [I] Redirected to https://foreman.localdomain/foreman_tasks/tasks/bb9f5987-3b7c-4aa6-bcde-ec2b747db727
2017-01-20 09:30:38 57a33314 [app] [I] Completed 302 Found in 249ms (ActiveRecord: 31.6ms)
2017-01-20 09:30:38 57a33314 [app] [I] Started GET "/foreman_tasks/tasks/bb9f5987-3b7c-4aa6-bcde-ec2b747db727" for 192.168.142.1 at 2017-01-20 09:30:38 +0100
2017-01-20 09:30:38 57a33314 [app] [I] Processing by ForemanTasks::TasksController#show as HTML
2017-01-20 09:30:38 57a33314 [app] [I]   Parameters: {"id"=>"bb9f5987-3b7c-4aa6-bcde-ec2b747db727"}
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_details.html.erb (27.4ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_running_steps.html.erb (1.3ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_errors.html.erb (0.9ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_raw.html.erb (3.3ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_locks.html.erb (1.4ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/show.html.erb within layouts/application (43.5ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered layouts/_application_content.html.erb (0.7ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_user_dropdown.html.erb (2.3ms)
2017-01-20 09:30:38 57a33314 [app] [I] Read fragment views/tabs_and_title_records-4 (0.1ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered home/_topbar.html.erb (8.9ms)
2017-01-20 09:30:38 57a33314 [app] [I]   Rendered layouts/base.html.erb (11.2ms)
2017-01-20 09:30:38 57a33314 [app] [I] Completed 200 OK in 65ms (Views: 57.2ms | ActiveRecord: 2.1ms)
2017-01-20 09:30:44 57a33314 [app] [I] Started GET "/foreman_tasks/tasks/bb9f5987-3b7c-4aa6-bcde-ec2b747db727" for 192.168.142.1 at 2017-01-20 09:30:44 +0100
2017-01-20 09:30:44 57a33314 [app] [I] Processing by ForemanTasks::TasksController#show as HTML
2017-01-20 09:30:44 57a33314 [app] [I]   Parameters: {"id"=>"bb9f5987-3b7c-4aa6-bcde-ec2b747db727"}
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_details.html.erb (27.5ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_running_steps.html.erb (0.2ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_errors.html.erb (0.2ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_raw.html.erb (2.4ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_locks.html.erb (1.2ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/show.html.erb within layouts/application (34.0ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered layouts/_application_content.html.erb (0.2ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered home/_user_dropdown.html.erb (1.6ms)
2017-01-20 09:30:44 57a33314 [app] [I] Read fragment views/tabs_and_title_records-4 (0.1ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered home/_topbar.html.erb (6.6ms)
2017-01-20 09:30:44 57a33314 [app] [I]   Rendered layouts/base.html.erb (7.9ms)
2017-01-20 09:30:44 57a33314 [app] [I] Completed 200 OK in 52ms (Views: 42.2ms | ActiveRecord: 2.7ms)
2017-01-20 09:30:49 57a33314 [app] [I] Started GET "/foreman_tasks/tasks/bb9f5987-3b7c-4aa6-bcde-ec2b747db727" for 192.168.142.1 at 2017-01-20 09:30:49 +0100
2017-01-20 09:30:49 57a33314 [app] [I] Processing by ForemanTasks::TasksController#show as HTML
2017-01-20 09:30:49 57a33314 [app] [I]   Parameters: {"id"=>"bb9f5987-3b7c-4aa6-bcde-ec2b747db727"}
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_details.html.erb (31.9ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_running_steps.html.erb (0.3ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_errors.html.erb (0.2ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_raw.html.erb (2.5ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/_locks.html.erb (1.3ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.8.2/app/views/foreman_tasks/tasks/show.html.erb within layouts/application (38.3ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered layouts/_application_content.html.erb (0.3ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered home/_user_dropdown.html.erb (2.0ms)
2017-01-20 09:30:49 57a33314 [app] [I] Read fragment views/tabs_and_title_records-4 (0.2ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered home/_topbar.html.erb (7.7ms)
2017-01-20 09:30:49 57a33314 [app] [I]   Rendered layouts/base.html.erb (9.2ms)
2017-01-20 09:30:49 57a33314 [app] [I] Completed 200 OK in 55ms (Views: 47.6ms | ActiveRecord: 2.1ms)

Actions

Copy link

Updated by Anonymous almost 7 years ago

Dirk, what's the status here with recent versions?

Actions

Copy link

Updated by Dirk Götz almost 7 years ago

With 14.1 (and 14.3, 15.0) behaviour changed from hanging infinitly to "Failed to initialize: Dynflow::ExecutionPlan::Steps::Error - The only applicable proxy foreman.localdomain is down". The proxy service is up, but now I also get a stacktrace:

2017-05-22 10:11:30 fe4eebb7 [app] [I] Started GET "/job_invocations/14" for 192.168.142.1 at 2017-05-22 10:11:30 +0200
2017-05-22 10:11:30 fe4eebb7 [app] [I] Processing by JobInvocationsController#show as HTML
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Parameters: {"id"=>"14"}
2017-05-22 10:11:30 fe4eebb7 [app] [I] Current user: dirk (administrator)
2017-05-22 10:11:30  [app] [I] Current user: dirk (administrator)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/_tab_overview.html.erb (44.8ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/_host_name_td.html.erb (1.3ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/_host_status_td.html.erb (1.1ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/_host_actions_td.html.erb (3.6ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/_tab_hosts.html.erb (19.6ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/job_invocations/show.html.erb within layouts/application (113.4ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered layouts/_application_content.html.erb (0.8ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered home/_user_dropdown.html.erb (3.4ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I] Read fragment views/tabs_and_title_records-4 (0.2ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered home/_topbar.html.erb (17.5ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I]   Rendered layouts/base.html.erb (21.0ms)
2017-05-22 10:11:30 fe4eebb7 [app] [I] Completed 200 OK in 171ms (Views: 138.4ms | ActiveRecord: 10.5ms)
2017-05-22 10:11:30 64d97542 [app] [I] Started GET "/assets/job_invocations-f079e92744b55af1d247e064a15a9711f56c06165fea3cd2a7724ef2bf5b011a.css" for 192.168.142.1 at 2017-05-22 10:11:30 +0200
2017-05-22 10:11:30 116210e2 [app] [I] Started GET "/assets/template_invocation-a9bdc63ae08856e191808c2379d1c6a691f6a78c5e3a193e56053f1cca2a7ddd.js" for 192.168.142.1 at 2017-05-22 10:11:30 +0200
2017-05-22 10:11:30  [app] [W] Could not fetch task counts from foreman.localdomain, skipped.
 | RestClient::InternalServerError: 500 Internal Server Error
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:495:in `process_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/resource.rb:51:in `get'
 | /usr/share/foreman/lib/proxy_api/resource.rb:67:in `get'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/proxy_api/foreman_dynflow/dynflow_proxy.rb:35:in `tasks_count'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:48:in `block in get_counts'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:45:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:45:in `each_with_object'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:45:in `get_counts'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:35:in `select_by_jobs_count'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:25:in `block in determine_proxy'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:23:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/services/foreman_tasks/proxy_selector.rb:23:in `determine_proxy'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_host_job.rb:28:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:467:in `block (3 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:34:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:466:in `block (2 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:370:in `concurrence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:465:in `block in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:464:in `execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:244:in `block (2 levels) in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:330:in `with_planning_scope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:243:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:42:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:242:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:177:in `block in plan_with_caller'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:175:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:175:in `plan_with_caller'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:76:in `block in trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:142:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:142:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:76:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:35:in `block in create_sub_plans'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:30:in `create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:37:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:60:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:33:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:26:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:61:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:42:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-05-22 10:11:30  [foreman-tasks/action] [E] The only applicable proxy foreman.localdomain is down (RuntimeError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_host_job.rb:32:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:467:in `block (3 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:34:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:466:in `block (2 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:370:in `concurrence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:465:in `block in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:464:in `execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:244:in `block (2 levels) in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:340:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:330:in `with_planning_scope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:243:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:42:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan.rb:242:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:177:in `block in plan_with_caller'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:175:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:175:in `plan_with_caller'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:76:in `block in trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:142:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/world.rb:142:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:76:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:35:in `block in create_sub_plans'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:30:in `create_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:37:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:60:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:33:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:26:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:61:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:42:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-05-22 10:11:30  [foreman-tasks/action] [E] A sub task failed (RuntimeError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:212:in `check_for_errors!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:115:in `wait_for_sub_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:39:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:60:in `spawn_plans'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:33:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:26:in `initiate'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:61:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:42:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-05-22 10:11:30  [foreman-tasks/dynflow] [W] step 4fa0030c-81dc-4e2a-9ec2-ad395d03e0c0:2 dropping event Dynflow::Director::Event[execution_plan_id: 4fa0030c-81dc-4e2a-9ec2-ad395d03e0c0, step_id: 2, event: Dynflow::Action::WithBulkSubPlans::PlanNextBatch, result: <#Concurrent::Edge::CompletableFuture:0xe3292f0 pending>]

Proxy reports following features:

[

    "abrt",
    "dhcp",
    "discovery",
    "dns",
    "dynflow",
    "monitoring",
    "omaha",
    "openscap",
    "puppet",
    "puppetca",
    "ssh",
    "tftp" 

]

Problem does not occur on freshly installed systems, but also colleagues were hit by this bug when upgrading environments.

Actions

Copy link

Updated by Adam Ruzicka almost 7 years ago

Is the smart_proxy_dynflow_core service running?

Actions

Copy link

Updated by Dirk Götz almost 7 years ago

Indeed the service was not enabled/running, I think this is because of the answer file from older versions did not know about it and nothing is enabling it by default or as dependency.

But it is also not running fine with the service enabled, running 'id' results in a failed state and gives me the following output:

Failed to initialize: Dynflow::ExecutionPlan::Steps::Error - ERF42-0769 [Foreman::Exception]: The smart proxy task 079a663f-efd7-41da-9ec9-b2f844e6da5f failed.
uid=0(root) gid=0(root) Gruppen=0(root) Kontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Stacktrace:

2017-05-22 11:40:16 acf5b6a4 [app] [I] Started GET "/template_invocations/32?since=1495445961.160904&line_counter=1&_=1495445985675" for 192.168.142.1 at 2017-05-22 11:40:16 +0200
2017-05-22 11:40:16 acf5b6a4 [app] [I] Processing by TemplateInvocationsController#show as JS
2017-05-22 11:40:16 acf5b6a4 [app] [I]   Parameters: {"since"=>"1495445961.160904", "line_counter"=>"1", "_"=>"1495445985675", "id"=>"32"}
2017-05-22 11:40:16 acf5b6a4 [app] [I] Current user: dirk (administrator)
2017-05-22 11:40:17 acf5b6a4 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_output_line_set.html.erb (0.0ms)
2017-05-22 11:40:17 acf5b6a4 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_refresh.js.erb (0.4ms)
2017-05-22 11:40:17 acf5b6a4 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/show.js.erb (1.7ms)
2017-05-22 11:40:17 acf5b6a4 [app] [I] Completed 200 OK in 98ms (Views: 3.4ms | ActiveRecord: 1.4ms)
2017-05-22 11:40:18 0f79f138 [app] [I] Started GET "/template_invocations/32?since=1495445961.160904&line_counter=1&_=1495445985676" for 192.168.142.1 at 2017-05-22 11:40:18 +0200
2017-05-22 11:40:18 0f79f138 [app] [I] Processing by TemplateInvocationsController#show as JS
2017-05-22 11:40:18 0f79f138 [app] [I]   Parameters: {"since"=>"1495445961.160904", "line_counter"=>"1", "_"=>"1495445985676", "id"=>"32"}
2017-05-22 11:40:18 0f79f138 [app] [I] Current user: dirk (administrator)
2017-05-22 11:40:18 0f79f138 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_output_line_set.html.erb (0.0ms)
2017-05-22 11:40:18 0f79f138 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_refresh.js.erb (0.3ms)
2017-05-22 11:40:18 0f79f138 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/show.js.erb (1.2ms)
2017-05-22 11:40:18 0f79f138 [app] [I] Completed 200 OK in 94ms (Views: 2.5ms | ActiveRecord: 1.3ms)
2017-05-22 11:40:19 b377d032 [app] [I] Started GET "/template_invocations/32?since=1495445961.160904&line_counter=1&_=1495445985677" for 192.168.142.1 at 2017-05-22 11:40:19 +0200
2017-05-22 11:40:19 b377d032 [app] [I] Processing by TemplateInvocationsController#show as JS
2017-05-22 11:40:19 b377d032 [app] [I]   Parameters: {"since"=>"1495445961.160904", "line_counter"=>"1", "_"=>"1495445985677", "id"=>"32"}
2017-05-22 11:40:19 b377d032 [app] [I] Current user: dirk (administrator)
2017-05-22 11:40:19 b377d032 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_output_line_set.html.erb (0.0ms)
2017-05-22 11:40:19 b377d032 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/_refresh.js.erb (0.3ms)
2017-05-22 11:40:19 b377d032 [app] [I]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/views/template_invocations/show.js.erb (1.3ms)
2017-05-22 11:40:19 b377d032 [app] [I] Completed 200 OK in 93ms (Views: 2.6ms | ActiveRecord: 1.1ms)
2017-05-22 11:40:19  [foreman-tasks/action] [E] ERF42-0769 [Foreman::Exception]: The smart proxy task 079a663f-efd7-41da-9ec9-b2f844e6da5f failed. (Foreman::Exception)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/proxy_action.rb:59:in `check_task_status'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/proxy_action.rb:37:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/proxy_action.rb:159:in `with_connection_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/proxy_action.rb:21:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:55:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-05-22 11:40:19  [app] [I] Current user: dirk (administrator)
2017-05-22 11:40:19  [foreman-tasks/action] [E] A sub task failed (RuntimeError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:212:in `check_for_errors!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:129:in `try_to_finish'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:21:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_sub_plans.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/with_bulk_sub_plans.rb:19:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.0/app/lib/actions/remote_execution/run_hosts_job.rb:61:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/app/lib/actions/middleware/keep_current_user.rb:28:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:55:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-05-22 11:40:19  [app] [I] Current user: dirk (administrator)

Any further ideas?

Actions

Copy link

Updated by Adam Ruzicka almost 7 years ago

Please take a log at smart_proxy_dynflow_core's log (in /var/log/foreman-proxy by default). You could also inspect the task running in the smart_proxy_dynflow_core by accessing the dynflow console running there (at /dynflow/console). It seems that the smart_proxy_dynflow_core is unable to report back to foreman and so it fails, even though the execution itself completed successfully. From what you wrote I would guess the core is not configured properly and is running with default settings. At the bare minimum you need to set foreman_url in /etc/smart_proxy_dynflow_core/settings.yml.

Actions

Copy link

Updated by Dirk Götz almost 7 years ago

You are rigth, again. So I added to the answers file and run the installer again to let it write the configuration.

"foreman_proxy::plugin::dynflow": true

Afterwards it changed it in the answer file to:

  "foreman_proxy::plugin::dynflow":
    enabled: true
    listen_on: https
    database_path: /var/lib/foreman-proxy/dynflow/dynflow.sqlite
    console_auth: true
    core_listen: "0.0.0.0" 
    core_port: 8008

So everything works fine now! Thanks for your help!

But I have no idea how we could change this to work out of the box or document it. Or if it is fine as most people will already be on 1.13 at least and so will not run in a upgrade-only bug.

Actions

Copy link

Updated by Adam Ruzicka almost 7 years ago

Status changed from New to Resolved

Glad to hear you got it working.

You're right, this is a rather tricky problem. Until we do come up with a OOTB solution we'll have to do it by hand like just now.

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Plugins » foreman-tasks

Custom queries

Bug #18171

Tasks (Play Ansible roles and Remote Execution) hang infinitly on Foreman 13.3/Tasks 0.8.2

Updated by Anonymous almost 7 years ago

Updated by Dirk Götz almost 7 years ago

Updated by Adam Ruzicka almost 7 years ago

Updated by Dirk Götz almost 7 years ago

Updated by Adam Ruzicka almost 7 years ago

Updated by Dirk Götz almost 7 years ago

Updated by Adam Ruzicka almost 7 years ago