uploading facts involves synchronous dynflow task, can cause bottleneck
Description of problem:
If a Satellite server is heavily loaded and has lots of tasks queued, you may start to see the web UI and API become unresponsive. Many of the passenger workers will be hung waiting on URLs like this:
The root cause is that updating facts kicks off a synchronous dynflow task. If dynflow is backed up, client fact uploads will get slower and slower, eventually consuming all passenger workers.
Version-Release number of selected component (if applicable): 6.2.8
Steps to Reproduce:
1. create a large number of tasks that have to be run
2. on a registered client, delete /var/lib/rhsm/facts/facts.json and run subscription-manager facts --update
Actual results: slow response, possible timeouts
Expected results: the client should get a reply quickly and the host update can then happen asynchronously. Note that if the async task fails, the client would not be aware.
Fixes #19061 - kick off async fact update
Previously, the POST call to update facts waited on a synchronous
task. This meant that if there was a backlog of tasks, passenger
workers would start to fill up, causing katello to fall over.
Instead, just kick off an async task to update facts. If the task
fails, users may not know without checking the task status, but it
should be OK to do this given the recurring nature of fact updates.
#1 Updated by Marek Hulán over 5 years ago
- Subject changed from uploading facts involves synchronous dynflow task, can cause bottleneck to uploading facts involves synchronous dynflow task, can cause bottleneck
- Category set to Subscriptions
I was not sure about the right category. Unfortunately, core does not yet support background processing for facts importing yet but since the subscription manager facts are wrapped in foreman task, it could be probably triggered asynchronously as suggested in the patch available in the BZ.