Project

General

Profile

Bug #25422

Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load

Added by Mike McCune over 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Mike McCune
Category:
API
Target version:
Katello 3.12.1
Difficulty:
Triaged:
Yes
Bugzilla link:
1643649
Pull request:
https://github.com/Katello/katello/pull/7825
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1643649

We previously switched from a sync task to an async task for host update API calls:

https://github.com/Katello/katello/pull/6709/files

This temporarily reduced a bottleneck at the passenger layer but introduces a new issue:

During high traffic situations it is likely and common for clients to issue sequential calls from the client, eg:

subscription-manager repos disable ..
subscription-manager repos enable ..

these calls end up firing off an async task but it is common that the second task will fail because the Host is locked from the 1st.

With our new ability to have multiple Dynflow executors, we should explore moving this back to a sync task so we can avoid repeated failed Tasks calling host update.

Longer term, we need to:

1) move this out of Foreman Tasks into a direct API call like we did for registration
2) A better retry method
3) Queuing per-host so we can process multiple calls for the same host and process them in order.

Associated revisions

Revision ae116f7f (diff)
Added by Mike McCune about 3 years ago

Fixes #25422 - optional switch to link locking for Host::Update

link will queue conflicting actions and process the Host
updates in the order they were received. Tasks will sit
in planned:pending until the running task is completed.
This will reduce the error rate and prevent clients from
receiving errors because sequential API calls error

History

#1 Updated by The Foreman Bot over 3 years ago

  • Assignee set to Mike McCune
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/7825 added

#2 Updated by Andrew Kofink over 3 years ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.10.0
  • Subject changed from Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load to Sequential Actions::Katello::Host::Update calls from subscription-manager can fail under load

#3 Updated by Christine Fouant over 3 years ago

  • Target version changed from Katello 3.10.0 to Katello 3.11.0

#4 Updated by John Mitsch over 3 years ago

  • Target version changed from Katello 3.11.0 to Katello 3.12.0

#5 Updated by Jonathon Turel about 3 years ago

  • Target version changed from Katello 3.12.0 to Katello 3.12.1

#6 Updated by Anonymous about 3 years ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF