Plugins » Foreman Remote Execution

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=2217397

Description of problem:
Foreman shows the REX job has finished successfully but the script was actually failed with no space available on the client.

Based on the Dynflow task output below, the wrapper script failed to write the exit code to a file due to no space which caused the terminal to exit with 0.
----------------------
proxy_output:
result:
<snip>
- output_type: stdout
<snip>
Error Summary
-------------
Disk Requirements:
At least XXXX more space needed on the / filesystem. <=========================== Yum failed due to insufficient space

Uploading Enabled Repositories Report
      Loaded plugins: product-id, subscription-manager
    timestamp: xxxxxxx
  - output_type: stdout
    output: |
      Package action failed, exiting...
      sh: line 0: echo: write error: No space left on device  <============================ Wrapper script failed to write the exit code to the file
    timestamp: xxxxxxx
  runner_id: xxxxxx
  exit_status: 0   <==================== wrong exit code.
----------------------

Additional info:

Based on "sh: line 0: echo: write error: No space left on device" error above, I think the shell script which wrapped the command failed to redirect the Yum exit code to the "@exit_code_path" file due to completely ran out of space in "/" directory. Since the @exit_code_path" file is empty, the terminal exited with 0 status code.

--------------------------------
<<-SCRIPT.gsub(/^\s+\| /, '') | sh -c "(#{@user_method.cli_command_prefix}#{su_method ? "'#{@remote_script} < /dev/null '" : "#{@remote_script} < /dev/null"}; echo \\$?>#{@exit_code_path}) | /usr/bin/tee #{@output_path} <====================== redirect the YUM exit code to a file | exit \\$(cat #{@exit_code_path})" <=============== exit the script with the exit code in the file
SCRIPT
--------------------------------

I think we should be able to prevent this issue by checking the exit status of the wrapping script itself in the case that the wrapping script itself fail to write the exit code of the Yum command.