Project

General

Profile

Bug #19061

uploading facts involves synchronous dynflow task, can cause bottleneck

Added by Marek Hulán over 5 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Subscriptions
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1435370

Description of problem:

If a Satellite server is heavily loaded and has lots of tasks queued, you may start to see the web UI and API become unresponsive. Many of the passenger workers will be hung waiting on URLs like this:

PUT /rhsm/consumers/<UUID>

The root cause is that updating facts kicks off a synchronous dynflow task. If dynflow is backed up, client fact uploads will get slower and slower, eventually consuming all passenger workers.

Version-Release number of selected component (if applicable): 6.2.8

Steps to Reproduce:
1. create a large number of tasks that have to be run
2. on a registered client, delete /var/lib/rhsm/facts/facts.json and run subscription-manager facts --update

Actual results: slow response, possible timeouts

Expected results: the client should get a reply quickly and the host update can then happen asynchronously. Note that if the async task fails, the client would not be aware.

Associated revisions

Revision d80a3f67 (diff)
Added by Chris Duryee about 5 years ago

Fixes #19061 - kick off async fact update

Previously, the POST call to update facts waited on a synchronous
task. This meant that if there was a backlog of tasks, passenger
workers would start to fill up, causing katello to fall over.

Instead, just kick off an async task to update facts. If the task
fails, users may not know without checking the task status, but it
should be OK to do this given the recurring nature of fact updates.

History

#1 Updated by Marek Hulán over 5 years ago

  • Subject changed from uploading facts involves synchronous dynflow task, can cause bottleneck to uploading facts involves synchronous dynflow task, can cause bottleneck
  • Category set to Subscriptions

I was not sure about the right category. Unfortunately, core does not yet support background processing for facts importing yet but since the subscription manager facts are wrapped in foreman task, it could be probably triggered asynchronously as suggested in the patch available in the BZ.

#2 Updated by Justin Sherrill over 5 years ago

  • Assignee set to Chris Duryee
  • Legacy Backlogs Release (now unused) set to 226

#3 Updated by The Foreman Bot over 5 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/6709 added

#4 Updated by Chris Duryee about 5 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF