Project

General

Profile

Actions
Copy link

Bug #35214

open

Unable to run hosts job

Added by Lars Wagner over 2 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Smart Proxy
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Since upgrading from foreman 3.1 to 3.2 and its plugins I'm unable to run host jobs. Upgrading from 3.2 to 3.3 didn't help too. I try to run a salt highstate:

Action:

Actions::RemoteExecution::RunHostsJob

Input:

{"job_invocation"=>
  {"id"=>27,
   "name"=>"Salt",
   "description"=>"Run Salt state.highstate on host"},
 "job_category"=>"Salt",
 "job_invocation_id"=>27,
 "proxy_batch_size"=>100,
 "trigger_run_step_id"=>5,
 "current_request_id"=>"51735317-ef8c-4952-a286-b37f31a59a32",
 "current_timezone"=>"Berlin",
 "current_organization_id"=>2,
 "current_location_id"=>1,
 "current_user_id"=>3}

Output:

{"host_count"=>1,
 "remote_triggered_count"=>0,
 "planned_count"=>1,
 "cancelled_count"=>0,
 "total_count"=>1,
 "failed_count"=>1,
 "pending_count"=>0,
 "success_count"=>0}

Exception:

RuntimeError: A sub task failed

Backtrace:

/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_polling_sub_plans.rb:19:in `poll'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/with_polling_sub_plans.rb:11:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman_remote_execution-7.1.0/app/lib/actions/remote_execution/run_hosts_job.rb:140:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/watch_delegated_proxy_sub_tasks.rb:17:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.0.4.8/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/foreman/vendor/ruby/2.7.0/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `catch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `catch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in `execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/director.rb:94:in `execute'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/foreman/vendor/ruby/2.7.0/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/foreman/vendor/ruby/2.7.0/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/foreman/vendor/ruby/2.7.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

foreman_remote_execution Version is: 7.1.0

Unfortunately I don't find any further information in the logs.

Files

foreman-proxy.log foreman-proxy.log 13.7 KB Lars Wagner, 07/14/2022 05:30 AM
Actions
Copy link
 #1

Updated by Adam Ruzicka over 2 years ago

Please attach logs from the proxy (/var/log/foreman-proxy/proxy.log)

Actions
Copy link
 #2

Updated by Lars Wagner over 2 years ago

I attached the requested log.

Actions
Copy link
 #3

Updated by Adam Ruzicka over 2 years ago

  • Project changed from Foreman Remote Execution to Salt
  • Category set to Smart Proxy
  • Target version deleted (foreman_remote_execution-4.9.0)

Nothing suspicious in there. What does the job output for that host say?

Also judging from the logs this is more likely something in foreman-salt rather than generic remote execution.

Actions
Copy link
 #4

Updated by Lars Wagner over 2 years ago

Which job output do you exactly mean? I added all task (job?) related output in the initial post.

Let me know if you need any further information. If this error doesn't belong to the remote execution module, I'll move that issue to the salt project.

Actions
Copy link
 #5

Updated by Lars Wagner over 2 years ago

I investigated a little further and was able to find the following error:

Error message in salt master

2022-07-18 14:26:00,111 [salt.auth        :364 ][WARNING ][6391] Authentication failure of type "user" occurred.
2022-07-18 14:26:00,112 [salt.master      :2204][WARNING ][6391] Authentication failure of type "user" occurred.

Content of /etc/foreman-proxy/settings.d/salt.yml configuration file for foreman-proxy:

---
:enabled: true
:autosign_file: /srv/autosign.conf
:salt_command_user: root
:use_api: true
:api_auth: pam
:api_url: https://localhost:9191
:api_username: saltuser
:api_password: mypass

The API is working as expected if I try to connect by curl:

# curl -sSk https://localhost:9191/login -H 'Accept: application/x-yaml' -d username='saltuser' -d password='mypass' -d eauth='pam'
return:
- eauth: pam
  expire: 1658190454.1744072
  perms:
  - '@runner'
  start: 1658147254.174407
  token: 123
  user: saltuser

Any help is highly appreciated.

Actions
Copy link
 #6

Updated by Lars Wagner over 2 years ago

Any chance that somebody might have a look into that problem?

Actions
Copy link
 #7

Updated by Lars Wagner over 2 years ago

I was able to track down the issue. The problem is that the salt commands are not executed by sudo:

File: /usr/lib/ruby/vendor_ruby/smart_proxy_salt/salt_runner.rb
--- salt_runner.rb    2022-08-15 15:47:05.578053805 +0200
+++ /usr/lib/ruby/vendor_ruby/smart_proxy_salt/salt_runner.rb    2022-08-15 15:41:52.688250986 +0200
@@ -43,7 +43,8 @@

       def generate_command
         saltfile_path = ::Proxy::Salt::Plugin.settings[:saltfile]
-        command = %w[salt --show-jid]
+        command = %w[sudo salt --show-jid]
+        #command = %w[salt --show-jid]
         command << "--saltfile=#{saltfile_path}" if File.file?(saltfile_path)
         command << @options['name']
         command << 'state.template_str'
Actions
Copy link
 #8

Updated by Bastian Schmidt about 2 years ago

Did you pay attention to the installation instruction which adds a sudoers conf for foreman-proxy?
Have a look here: https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#salt_guide_configuring_foreman_managing-hosts

Actions
Copy link
 #9

Updated by Lars Wagner about 2 years ago

Bastian Schmidt wrote:

Did you pay attention to the installation instruction which adds a sudoers conf for foreman-proxy?
Have a look here: https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#salt_guide_configuring_foreman_managing-hosts

Yes, I did. For that reason the patch including the sudo command is required. Otherwise commands will be executed by the foreman-proxy user and no sudo will be used.

Actions
Copy link

Also available in: Atom PDF