Project

General

Profile

Actions

Bug #36921

closed

"Actions::Katello::Applicability::Hosts::BulkGenerate" tasks are processed in the default queue instead of hosts_queue causing congestion

Added by Hao Yu 7 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Performance
Target version:
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2249736

Description of problem:
"Actions::Katello::Applicability::Hosts::BulkGenerate" tasks can cause queue congestion in Dynflow, especially with this performance bug 2203077. The tasks are supposed to process by the "hosts_queue" worker but they go to the default queue instead.

This is causing slowness to other tasks which need to be processed by the default queue worker. Content view publish task is hanging waiting the task status to be polled from Pulp.

How reproducible:
Easy

Steps to Reproduce:
1. Go to Web UI -> Hosts -> Content Hosts -> click in to the host -> Content -> Errata -> click "recalculate"
2. Then on the Satellite command line, run the following command

watch -n 0.1 systemctl status

Actual results:
You should notice that 1 thread is busy like below:

# systemctl status dynflow-sidekiq@worker-1.service
● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-11-15 00:40:26 UTC; 3h 14min ago
     Docs: https://theforeman.org
 Main PID: xxxxx (sidekiq)
   Status: "Everything ready for world: 57b50c94-dbfe-4271-8033-76bc74138212" 
    Tasks: 15 (limit: 126231)
   Memory: 1.5G
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
           └─1104255 sidekiq 6.3.1  [1 of 5 busy] <==================

Expected results:
Expect to see 1 thread to busy in "worker-hosts-queue-1" worker.

Additional info:
Setting['host_tasks_workers_pool_size'] is used during the initialization but Katello settings are still not loaded at that time. Therefore, the value is always 0 and the condition will never met.

In lib/katello/engine.rb

    initializer "katello.register_actions", :before => :finisher_hook do |_app|
      ForemanTasks.dynflow.require!
      if (Setting.table_exists? rescue(false)) && Setting['host_tasks_workers_pool_size'].to_i > 0  <===============================
        ForemanTasks.dynflow.config.queues.add(HOST_TASKS_QUEUE, :pool_size => Setting['host_tasks_workers_pool_size'])
      end

We can also see the below warning in the production.log

2023-11-14T07:30:27 [W|app|] Setting host_tasks_workers_pool_size has no definition, please define it before using

In "/usr/share/foreman/app/registries/foreman/setting_manager.rb" we can see the plugin settings will only be fully loaded in "config.to_prepare" state.

Rails.application.config.to_prepare do
  Foreman::SettingManager.validations.setup!
end

Actions #1

Updated by The Foreman Bot 7 months ago

  • Status changed from New to Ready For Testing
  • Assignee set to Hao Yu
  • Pull request https://github.com/Katello/katello/pull/10799 added
Actions #2

Updated by Quinn James 7 months ago

  • Target version set to Katello 4.12.0
  • Triaged changed from No to Yes
Actions #3

Updated by The Foreman Bot 7 months ago

  • Pull request https://github.com/Katello/katello/pull/10807 added
Actions #4

Updated by The Foreman Bot 7 months ago

  • Pull request deleted (https://github.com/Katello/katello/pull/10799, https://github.com/Katello/katello/pull/10807)
Actions #5

Updated by The Foreman Bot 7 months ago

  • Fixed in Releases Katello 4.11.0 added
Actions #6

Updated by Hao Yu 7 months ago

  • Status changed from Ready For Testing to Closed
Actions #7

Updated by Jeremy Lenz 5 months ago

  • Pull request https://github.com/Katello/katello/pull/10799 added
Actions

Also available in: Atom PDF