Delays when many clients upload tracer data simultaneously
Description of problem:
When customer patches around 60 servers simultaneously, many of them get to the "Uploading Tracer Profile" step around the same time, and some clients have delays of over 15 minutes. It is also possible to reproduce this by running a remote execution "katello-tracer-upload" command on multiple hosts.
How reproducible: Consistently
Steps to Reproduce:
1. Have around 160 clients with katello-host-tools-tracer installed.
2. Create a Remote Execution job to run on the 160 clients and run the command "time katello-tracer-upload", with a 15 minute time out.
Actual results: In customers environment, 3 of the servers timed out after 15 minutes, and 2 servers ran for over 5 minutes.
Expected results: Servers not taking over 15 minutes to run katello-host-tools-tracer
Customer ran the same Remote execution job, but set the concurrency level to 1, so that it only ran on one server at a time, and did not have any failures or delays.