Foreman Maintain

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1877466

There are cases where paused tasks fail to resume and foreman-maintain can not proceed with the upgrade. In most of these cases, the user will just delete the task with foreman-rake but this provides a significant barrier upgrades. We need to automate this and build it directly into foreman-maintain.

There are situations where the task is unable to be resumed and should just be deleted, eg:

1. foreman-maintain upgrade run --target-version=6.7.z y
   ...
   -------------------------------------------------------------------------------
   Check for paused tasks: [FAIL]
   There are currently 1 paused tasks in the system
   --------------------------------------------------------------------------------
   There are multiple steps to proceed:
   1) Resume paused tasks
   2) Investigate the tasks via UI
   (assuming first option)
   Resume paused tasks: [OK]
   Total tasks found paused in error state: 1
   Total tasks resumed: 1
   Resumed tasks:
   1) Task identifier: d775afce-232b-4310-9ff6-475f64648e79
   Task action: Publish
   Task errors: ERROR
   --------------------------------------------------------------------------------
   Rerunning the check after fix procedure
   Check for paused tasks: [FAIL]
   There are currently 1 paused tasks in the system
   --------------------------------------------------------------------------------
   ...

Scenario [Checks before upgrading to Satellite 6.7.z] failed.

The following steps ended up in failing state:

[foreman-tasks-not-paused]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="foreman-tasks-not-paused"

The steps in warning state itself might not mean there is an error,
but it should be reviewed to ensure the behavior is expected

#

We should update foreman-maintain to offer 3 choices:

1) Resume paused tasks <RECOMMENDED>
2) Investigate the tasks via UI
3) Delete paused tasks

and if (1) fails, then proceed to (3) delete and not require any user interaction. This will allow for a -y flag to upgrade to critical NOT FAIL the upgrade if there are paused tasks.

A run with -y and the above updated routine would like like:

--------------------------------------------------------------------------------
Check for paused tasks: [FAIL]
There are currently 1 paused tasks in the system
--------------------------------------------------------------------------------
There are multiple steps to proceed:
1) Resume paused tasks
2) Investigate the tasks via UI
3) Delete paused tasks
(assuming first option)
Resume paused tasks: [OK]
Total tasks found paused in error state: 1
Total tasks resumed: 1
Resumed tasks:
1) Task identifier: d775afce-232b-4310-9ff6-475f64648e79
Task action: Publish
Task errors: ERROR
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check for paused tasks: [FAIL]
There are currently 1 paused tasks in the system
--------------------------------------------------------------------------------
Deleting Paused tasks:

• WARNING: 1 task deleted
  --------------------------------------------------------------------------------
  Rerunning the check after fix procedure
  Check for old tasks in paused/stopped state: [OK]
  --------------------------------------------------------------------------------
  Check for tasks in planning state: [OK]
  --------------------------------------------------------------------------------
  Check to verify if any hotfix installed on system:
  / Checking for presence of hotfix(es). It may take some time to verify.
  ...