Ability to restart the machine while the remote execution job is still acting as running - backend
Some scenarios in rex involve need to disconnect from network and continue some work offline. While we have support for `async`, it still marks the job as failed when the managed hosts goes offline.
With this feature, we should be able to take control over the job status from the remote host, so that the job is marked as running even when the host goes down temporary as part of execution of the job.
Example of such a template:
$CONTROL_SCRIPT manual-mode cat <<HELP | $CONTROL_SCRIPT update >/dev/null The script has switched to manual-mode. It will be acting as running after this script finishes. The control script is available in \$CONTROL_SCRIPT env varaible. To send output data to the job, on can do something like this: echo Hello world | $CONTROL_SCRIPT update To mark the script as finished, one can do $CONTROL_SCRIPT finish 0 there the second argument should be the exit code the script ended with. HELP
After running this, one should be able to go to the remote host, reboot it and the job should still be running until
$CONTROL_SCRIPT finish 0 is finished. Additional output can be sent to the job with
echo Hello world | $CONTROL_SCRIPT update
the satellite needs to be installed with
--foreman-proxy-plugin-remote-execution-ssh-async-ssh=true in order for this to work.
- Fixes #26428 - exposed $CONTROL_SCRIPT and manual-mode
This patch addes ability to take control over the job, allowing things
like surviving restarts and other advanced stuff. The main commands
allowing this is:
- `$CONTROL_SCRIPT manual-control` : turn on the manual control
- `echo hello | $CONTROL_SCRIPT update` : send output to the job
- `$CONTROL_SCRIPT finish $EXIT_CODE` : mark the job as finished
with corresponding $EXIT_CODE
Currently, this only works when proxy rex plugin is configured with
It also involves refoctoring to make majority of the control scripts
static files, and dynamically generate only `env.sh` script, containing
dynamic data only, that other scripts can source from.
- Refs #26428 - refresh in non-manual mode and fix exit code
- Refs #26428 - more resiliency on failed curl
Make sure to move on with output only when the curl was successful.
- Refs #26428 - Make exit_code the first key in payload
To prevent it overriding other fields
- Refs #26428 - introduce retrieve script timeouts
- Refs #26428 - prevent loosing of output
#7 Updated by Ivan Necas about 1 year ago
- Status changed from Ready For Testing to Closed
Applied in changeset foreman_plugin|cc78145f78ecb2fcbae630b791b7ea1d68b4b02c.