Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load
We previously switched from a sync task to an async task for host update API calls:
This temporarily reduced a bottleneck at the passenger layer but introduces a new issue:
During high traffic situations it is likely and common for clients to issue sequential calls from the client, eg:
subscription-manager repos disable ..
subscription-manager repos enable ..
these calls end up firing off an async task but it is common that the second task will fail because the Host is locked from the 1st.
With our new ability to have multiple Dynflow executors, we should explore moving this back to a sync task so we can avoid repeated failed Tasks calling host update.
Longer term, we need to:
1) move this out of Foreman Tasks into a direct API call like we did for registration
2) A better retry method
3) Queuing per-host so we can process multiple calls for the same host and process them in order.
Fixes #25422 - optional switch to link locking for Host::Update
link will queue conflicting actions and process the Host
updates in the order they were received. Tasks will sit
in planned:pending until the running task is completed.
This will reduce the error rate and prevent clients from
receiving errors because sequential API calls error
#2 Updated by Andrew Kofink over 3 years ago
- Triaged changed from No to Yes
- Target version set to Katello 3.10.0
- Subject changed from Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load to Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load