Bug #37285: REX jobs failing with Proxy task gone missing if proxy->foreman callback fails - foreman-tasks - Foreman

Actions

Copy link

Bug #37285

closed

REX jobs failing with Proxy task gone missing if proxy->foreman callback fails

Added by Adam Ruzicka 9 months ago. Updated 5 months ago.

Status:

Closed

Priority:

Normal

Assignee:

Adam Ruzicka

Category:

Foreman plugin

Target version:

Difficulty:

Triaged:

Bugzilla link:

Pull request:

https://github.com/theforeman/foreman-tasks/pull/749

Fixed in Releases:

foreman_tasks-9.2.2

Found in Releases:

Red Hat JIRA:

Description

Description of problem:
REX jobs failing with Proxy task gone missing if proxy->foreman callback fails. This bug manifests itself the same way as BZ2270295¹, the other bug is caused by changes done in smart_proxy_dynflow-0.9.1, this one is caused by changes that went out in foreman-tasks-5.3.0.

If you see the proxy task gone missing errors and find `[E] <RuntimeError> Failed performing callback to Foreman server` in proxy logs, then it is this one. Alternatively, if the job runs for more than 10 minutes, it is this one.

How reproducible:
The conditions to trigger the bug are sort of difficult to reproduce

Steps to Reproduce:
1. Run a job
2. Ensure that the callback from proxy to foreman fails
This is rather difficult to do naturally, in development I resorted to modifying code. It was discovered "in the wild" during scale tests
3. Wait until foreman checks on the proxy

Actual results:
The rex job fails with 'Proxy task gone missing'

Expected results:
The job succeeds or fails, depending on whether it succeeded or failed on the capsule

Additional info:

[1] - https://bugzilla.redhat.com/show_bug.cgi?id=2270295