Plugins » Katello

Clone from https://bugzilla.redhat.com/show_bug.cgi?id=2249736

Description of problem:
"Actions::Katello::Applicability::Hosts::BulkGenerate" tasks can cause queue congestion in Dynflow, especially with this performance bug 2203077. The tasks are supposed to process by the "hosts_queue" worker but they go to the default queue instead.

This is causing slowness to other tasks which need to be processed by the default queue worker. Content view publish task is hanging waiting the task status to be polled from Pulp.

How reproducible:
Easy

Steps to Reproduce:
1. Go to Web UI -> Hosts -> Content Hosts -> click in to the host -> Content -> Errata -> click "recalculate"
2. Then on the Satellite command line, run the following command

watch -n 0.1 systemctl status dynflow-sidekiq@worker-1.service

Actual results:
You should notice that 1 thread is busy like below:

# systemctl status dynflow-sidekiq@worker-1.service
● dynflow-sidekiq@worker-1.service - Foreman jobs daemon - worker-1 on sidekiq
   Loaded: loaded (/usr/lib/systemd/system/dynflow-sidekiq@.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-11-15 00:40:26 UTC; 3h 14min ago
     Docs: https://theforeman.org
 Main PID: xxxxx (sidekiq)
   Status: "Everything ready for world: 57b50c94-dbfe-4271-8033-76bc74138212" 
    Tasks: 15 (limit: 126231)
   Memory: 1.5G
   CGroup: /system.slice/system-dynflow\x2dsidekiq.slice/dynflow-sidekiq@worker-1.service
           └─1104255 sidekiq 6.3.1  [1 of 5 busy] <==================

Expected results:
Expect to see 1 thread to busy in "worker-hosts-queue-1" worker.

Additional info:
Setting['host_tasks_workers_pool_size'] is used during the initialization but Katello settings are still not loaded at that time. Therefore, the value is always 0 and the condition will never met.

In lib/katello/engine.rb

    initializer "katello.register_actions", :before => :finisher_hook do |_app|
      ForemanTasks.dynflow.require!
      if (Setting.table_exists? rescue(false)) && Setting['host_tasks_workers_pool_size'].to_i > 0  <===============================
        ForemanTasks.dynflow.config.queues.add(HOST_TASKS_QUEUE, :pool_size => Setting['host_tasks_workers_pool_size'])
      end

We can also see the below warning in the production.log

2023-11-14T07:30:27 [W|app|] Setting host_tasks_workers_pool_size has no definition, please define it before using

In "/usr/share/foreman/app/registries/foreman/setting_manager.rb" we can see the plugin settings will only be fully loaded in "config.to_prepare" state.

Rails.application.config.to_prepare do
  Foreman::SettingManager.validations.setup!
end