Bug #35214
openUnable to run hosts job
Description
Since upgrading from foreman 3.1 to 3.2 and its plugins I'm unable to run host jobs. Upgrading from 3.2 to 3.3 didn't help too. I try to run a salt highstate:
Action:
Actions::RemoteExecution::RunHostsJob
Input:
{"job_invocation"=> {"id"=>27, "name"=>"Salt", "description"=>"Run Salt state.highstate on host"}, "job_category"=>"Salt", "job_invocation_id"=>27, "proxy_batch_size"=>100, "trigger_run_step_id"=>5, "current_request_id"=>"51735317-ef8c-4952-a286-b37f31a59a32", "current_timezone"=>"Berlin", "current_organization_id"=>2, "current_location_id"=>1, "current_user_id"=>3}
Output:
{"host_count"=>1, "remote_triggered_count"=>0, "planned_count"=>1, "cancelled_count"=>0, "total_count"=>1, "failed_count"=>1, "pending_count"=>0, "success_count"=>0}
Exception:
RuntimeError: A sub task failed
Backtrace:
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_polling_sub_plans.rb:19:in `poll' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_polling_sub_plans.rb:11:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_remote_execution-7.1.0/app/lib/actions/remote_execution/run_hosts_job.rb:140:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.4.8/lib/active_support/execution_wrapper.rb:91:in `wrap' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:17:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' /usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in `execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `catch' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `block in execute_run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `block in with_error_handling' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `catch' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `with_error_handling' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:575:in `execute_run' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in `execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/director.rb:94:in `execute' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in `run_user_code' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform' /usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels) in process' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in <module:Sidekiq>' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog' /usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread' /usr/share/foreman/vendor/ruby/2.7.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
foreman_remote_execution Version is: 7.1.0
Unfortunately I don't find any further information in the logs.
Files
Updated by Adam Ruzicka about 2 years ago
Please attach logs from the proxy (/var/log/foreman-proxy/proxy.log)
Updated by Lars Wagner about 2 years ago
- File foreman-proxy.log foreman-proxy.log added
I attached the requested log.
Updated by Adam Ruzicka about 2 years ago
- Project changed from Foreman Remote Execution to Salt
- Category set to Smart Proxy
- Target version deleted (
foreman_remote_execution-4.9.0)
Nothing suspicious in there. What does the job output for that host say?
Also judging from the logs this is more likely something in foreman-salt rather than generic remote execution.
Updated by Lars Wagner about 2 years ago
Which job output do you exactly mean? I added all task (job?) related output in the initial post.
Let me know if you need any further information. If this error doesn't belong to the remote execution module, I'll move that issue to the salt project.
Updated by Lars Wagner about 2 years ago
I investigated a little further and was able to find the following error:
Error message in salt master
2022-07-18 14:26:00,111 [salt.auth :364 ][WARNING ][6391] Authentication failure of type "user" occurred. 2022-07-18 14:26:00,112 [salt.master :2204][WARNING ][6391] Authentication failure of type "user" occurred.
Content of /etc/foreman-proxy/settings.d/salt.yml configuration file for foreman-proxy:
--- :enabled: true :autosign_file: /srv/autosign.conf :salt_command_user: root :use_api: true :api_auth: pam :api_url: https://localhost:9191 :api_username: saltuser :api_password: mypass
The API is working as expected if I try to connect by curl:
# curl -sSk https://localhost:9191/login -H 'Accept: application/x-yaml' -d username='saltuser' -d password='mypass' -d eauth='pam' return: - eauth: pam expire: 1658190454.1744072 perms: - '@runner' start: 1658147254.174407 token: 123 user: saltuser
Any help is highly appreciated.
Updated by Lars Wagner about 2 years ago
Any chance that somebody might have a look into that problem?
Updated by Lars Wagner about 2 years ago
I was able to track down the issue. The problem is that the salt commands are not executed by sudo:
File: /usr/lib/ruby/vendor_ruby/smart_proxy_salt/salt_runner.rb
--- salt_runner.rb 2022-08-15 15:47:05.578053805 +0200 +++ /usr/lib/ruby/vendor_ruby/smart_proxy_salt/salt_runner.rb 2022-08-15 15:41:52.688250986 +0200 @@ -43,7 +43,8 @@ def generate_command saltfile_path = ::Proxy::Salt::Plugin.settings[:saltfile] - command = %w[salt --show-jid] + command = %w[sudo salt --show-jid] + #command = %w[salt --show-jid] command << "--saltfile=#{saltfile_path}" if File.file?(saltfile_path) command << @options['name'] command << 'state.template_str'
Updated by Bastian Schmidt over 1 year ago
Did you pay attention to the installation instruction which adds a sudoers conf for foreman-proxy?
Have a look here: https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#salt_guide_configuring_foreman_managing-hosts
Updated by Lars Wagner over 1 year ago
Bastian Schmidt wrote:
Did you pay attention to the installation instruction which adds a sudoers conf for foreman-proxy?
Have a look here: https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#salt_guide_configuring_foreman_managing-hosts
Yes, I did. For that reason the patch including the sudo command is required. Otherwise commands will be executed by the foreman-proxy user and no sudo will be used.