Bug #24138
closedWhen running REX at scale, capsule is not able process all the incoming requests, leading to "Initialization error: RestClient::RequestTimeout - Request Timeout" in actions
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1431951
Description of problem:
about 7 sub-tasks from remote execution on 5000 systems failed with "Initialization error: RestClient::RequestTimeout - Request Timeout". Rest of tasks were +- OK
Version-Release number of selected component (if applicable):
Sat: satellite-6.2.8-4.0.el7sat.noarch
Capsule: satellite-capsule-6.2.8-4.0.el7sat.noarch
How reproducible:
usually
Steps to Reproduce:
1. Run `date` via ReX on 5k systems
Actual results:
Although there is this in the production.log on Satellite:
017-03-13 21:50:29 [app] [I] Started POST "/foreman_tasks/api/tasks/callback" for 172.17.1.6 at 2017-03-13 21:50:29 -0400
2017-03-13 21:50:29 [app] [I] Processing by ForemanTasks::Api::TasksController#callback as XML
2017-03-13 21:50:29 [app] [I] Parameters: {"callback"=>{"task_id"=>"3fb26249-36ec-4bf6-b812-7941d284fbdf", "step_id"=>3}, "data"=>{"result"=>[{"output_type"=>"stdout", "output"=>"Tue Mar 14 01:49:01 UTC 2017\r\n", "timestamp"=>1489456143.0842535}], "exit_status"=>0}, "task"=>{}}
2017-03-13 21:50:29 [app] [I] Completed 200 OK in 7ms (Views: 0.1ms | ActiveRecord: 1.4ms)
2017-03-13 21:50:29 [app] [I] Started POST "/foreman_tasks/api/tasks/callback" for 172.17.1.6 at 2017-03-13 21:50:29 -0400
there is this on the web:
Id: 3fb26249-36ec-4bf6-b812-7941d284fbdf
Label: Actions::RemoteExecution::RunHostJob
Name: Remote action:
Owner:
Execution type: Delayed
Start at: 2017-03-14 00:27:19 +0100
Start before: -
Started at: 2017-03-14 00:27:19 +0100
Ended at: 2017-03-14 04:28:43 +0100
State: stopped
Result: warning
Params: Run date on gprfc026container455.example.com
Parent task
100.0% Complete
100%
Output:
Initialization error: RestClient::RequestTimeout - Request Timeout
No output
Job finished with error: RuntimeError - Timeout exceeded.
Errors:
Timeout exceeded.
Expected results:
Should not be marked as timeouted
Updated by Adam Ruzicka over 5 years ago
- Category set to Smart Proxy Dynflow
- Target version set to 113
Updated by Adam Ruzicka over 5 years ago
- Blocked by Bug #25160: Allow batch triggering of tasks on the smart proxy added
Updated by The Foreman Bot over 5 years ago
- Status changed from New to Ready For Testing
- Assignee set to Adam Ruzicka
- Pull request https://github.com/theforeman/foreman_remote_execution/pull/392 added
Updated by Adam Ruzicka about 5 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset foreman_plugin|698e66a29f32fad93abbd396d59d6de05a3f11a3.
Updated by Adam Ruzicka almost 5 years ago
- Triaged set to No
- Fixed in Releases foreman_remote_execution 1.7.1, foreman_remote_execution_core 1.1.6 added