Project

General

Profile

Actions

Bug #24138

closed

When running REX at scale, capsule is not able process all the incoming requests, leading to "Initialization error: RestClient::RequestTimeout - Request Timeout" in actions

Added by Adam Ruzicka over 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Smart Proxy Dynflow
Target version:
-
Difficulty:
Triaged:
No
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1431951

Description of problem:
about 7 sub-tasks from remote execution on 5000 systems failed with "Initialization error: RestClient::RequestTimeout - Request Timeout". Rest of tasks were +- OK

Version-Release number of selected component (if applicable):
Sat: satellite-6.2.8-4.0.el7sat.noarch
Capsule: satellite-capsule-6.2.8-4.0.el7sat.noarch

How reproducible:
usually

Steps to Reproduce:
1. Run `date` via ReX on 5k systems

Actual results:
Although there is this in the production.log on Satellite:

017-03-13 21:50:29 [app] [I] Started POST "/foreman_tasks/api/tasks/callback" for 172.17.1.6 at 2017-03-13 21:50:29 -0400
2017-03-13 21:50:29 [app] [I] Processing by ForemanTasks::Api::TasksController#callback as XML
2017-03-13 21:50:29 [app] [I] Parameters: {"callback"=>{"task_id"=>"3fb26249-36ec-4bf6-b812-7941d284fbdf", "step_id"=>3}, "data"=>{"result"=>[{"output_type"=>"stdout", "output"=>"Tue Mar 14 01:49:01 UTC 2017\r\n", "timestamp"=>1489456143.0842535}], "exit_status"=>0}, "task"=>{}}
2017-03-13 21:50:29 [app] [I] Completed 200 OK in 7ms (Views: 0.1ms | ActiveRecord: 1.4ms)
2017-03-13 21:50:29 [app] [I] Started POST "/foreman_tasks/api/tasks/callback" for 172.17.1.6 at 2017-03-13 21:50:29 -0400

there is this on the web:

Id: 3fb26249-36ec-4bf6-b812-7941d284fbdf
Label: Actions::RemoteExecution::RunHostJob
Name: Remote action:
Owner:
Execution type: Delayed
Start at: 2017-03-14 00:27:19 +0100
Start before: -
Started at: 2017-03-14 00:27:19 +0100
Ended at: 2017-03-14 04:28:43 +0100
State: stopped
Result: warning
Params: Run date on gprfc026container455.example.com
Parent task
100.0% Complete
100%

Output:
Initialization error: RestClient::RequestTimeout - Request Timeout
No output
Job finished with error: RuntimeError - Timeout exceeded.

Errors:
Timeout exceeded.

Expected results:
Should not be marked as timeouted


Related issues 1 (0 open1 closed)

Blocked by foreman-tasks - Bug #25160: Allow batch triggering of tasks on the smart proxyClosedAdam RuzickaActions
Actions #1

Updated by Adam Ruzicka over 5 years ago

  • Category set to Smart Proxy Dynflow
  • Target version set to 113
Actions #2

Updated by Adam Ruzicka over 5 years ago

  • Blocked by Bug #25160: Allow batch triggering of tasks on the smart proxy added
Actions #3

Updated by The Foreman Bot over 5 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Adam Ruzicka
  • Pull request https://github.com/theforeman/foreman_remote_execution/pull/392 added
Actions #4

Updated by Adam Ruzicka about 5 years ago

  • Status changed from Ready For Testing to Closed
Actions #5

Updated by Adam Ruzicka almost 5 years ago

  • Triaged set to No
  • Fixed in Releases foreman_remote_execution 1.7.1, foreman_remote_execution_core 1.1.6 added
Actions

Also available in: Atom PDF