Bug #25422
Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1643649
We previously switched from a sync task to an async task for host update API calls:
https://github.com/Katello/katello/pull/6709/files
This temporarily reduced a bottleneck at the passenger layer but introduces a new issue:
During high traffic situations it is likely and common for clients to issue sequential calls from the client, eg:
subscription-manager repos disable ..
subscription-manager repos enable ..
these calls end up firing off an async task but it is common that the second task will fail because the Host is locked from the 1st.
With our new ability to have multiple Dynflow executors, we should explore moving this back to a sync task so we can avoid repeated failed Tasks calling host update.
Longer term, we need to:
1) move this out of Foreman Tasks into a direct API call like we did for registration
2) A better retry method
3) Queuing per-host so we can process multiple calls for the same host and process them in order.
Associated revisions
History
#1
Updated by The Foreman Bot over 3 years ago
- Assignee set to Mike McCune
- Status changed from New to Ready For Testing
- Pull request https://github.com/Katello/katello/pull/7825 added
#2
Updated by Andrew Kofink over 3 years ago
- Triaged changed from No to Yes
- Target version set to Katello 3.10.0
- Subject changed from Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load to Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load
#3
Updated by Christine Fouant over 3 years ago
- Target version changed from Katello 3.10.0 to Katello 3.11.0
#4
Updated by John Mitsch over 3 years ago
- Target version changed from Katello 3.11.0 to Katello 3.12.0
#5
Updated by Jonathon Turel about 3 years ago
- Target version changed from Katello 3.12.0 to Katello 3.12.1
#6
Updated by Anonymous about 3 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset katello|ae116f7fbd1210717368fd74af60829717f25302.
Fixes #25422 - optional switch to link locking for Host::Update
link will queue conflicting actions and process the Host
updates in the order they were received. Tasks will sit
in planned:pending until the running task is completed.
This will reduce the error rate and prevent clients from
receiving errors because sequential API calls error