Project

General

Profile

Bug #29397

Delays when many clients upload tracer data simultaneously

Added by Jonathon Turel 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Category:
API
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1691416

Description of problem:
When customer patches around 60 servers simultaneously, many of them get to the "Uploading Tracer Profile" step around the same time, and some clients have delays of over 15 minutes. It is also possible to reproduce this by running a remote execution "katello-tracer-upload" command on multiple hosts.

How reproducible: Consistently

Steps to Reproduce:
1. Have around 160 clients with katello-host-tools-tracer installed.
2. Create a Remote Execution job to run on the 160 clients and run the command "time katello-tracer-upload", with a 15 minute time out.

Actual results: In customers environment, 3 of the servers timed out after 15 minutes, and 2 servers ran for over 5 minutes.

Expected results: Servers not taking over 15 minutes to run katello-host-tools-tracer

Additional info:
Customer ran the same Remote execution job, but set the concurrency level to 1, so that it only ran on one server at a time, and did not have any failures or delays.

See also: https://community.theforeman.org/t/katello-tracer-troubleshoot/8676

Associated revisions

Revision 709fe41f (diff)
Added by Jonathon Turel 7 months ago

Fixes #29397 - improve tracer upload performance

History

#1 Updated by Jonathon Turel 7 months ago

  • Target version set to Katello 3.16.0
  • Category set to API
  • Subject changed from Delays when many clients upload tracer data simultaneously to Delays when many clients upload tracer data simultaneously

#2 Updated by The Foreman Bot 7 months ago

  • Assignee set to Jonathon Turel
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/8625 added

#3 Updated by Samir Jha 7 months ago

  • Triaged changed from No to Yes

#4 Updated by The Foreman Bot 7 months ago

  • Fixed in Releases Katello 3.16.0 added

#5 Updated by Jonathon Turel 7 months ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF