Plugins » Foreman Remote Execution

• We are a extremely heavy user of Foreman and Dynflow. I would guess maybe even the largest in the world. 70,000 hosts, around 2M facts processed daily, and around 150,000-300,000 daily jobs.
  The new job invocations page is quite a bit (to put it nicely) slower in loading than the previous old page.
  In addition, it doesn’t appear to update live like the old one did. I see hosts that show Pending, but if you click on the arrow to see the output, they have already finished their job.

When loading the Legacy page (32,000 job test) it loads in about 3-6 seconds, that includes all the hosts and their statuses. And it updates live, and quickly.

The new UI isn’t even useable. If I load the new UI on the same parent job (32,000 jobs), it loads and loads and loads and never seems to complete. Each load I have waited about 12-17 mins. After 2-3 mins it loads the job page itself. After another 10-15 mins the host lists still shows loading. It took me around 20 mins to write this post, the hosts are still stuck loading.

On occasion, we also see this error:

TypeError: Cannot read properties of undefined (reading 'length')     in g     in t     in main     in div     in Page     in div     in FlexItem     in div     in Flex     in p     in b     in t     in o     in a     in Connect(a)     in vo     in div     in h     in IntlProvider     in I18nProviderWrapper(h)     in d     in StoreProvider(I18nProviderWrapper(h))     in DataProvider(StoreProvider(I18nProviderWrapper(h)))     in Unknown

Expected outcome:
The new jobs page loads as quickly (if not quicker) than the current Legacy page.

Foreman and Proxy versions:
All latest (3.18.1)

$ rpm -qa | grep foreman
rubygem-foreman_salt-17.0.2-1.fm3_15.el9.noarch
rubygem-foreman_vault-3.0.0-1.fm3_15.el9.noarch
rubygem-foreman_statistics-2.1.0-4.fm3_16.el9.noarch
rubygem-hammer_cli_foreman_kubevirt-0.2.0-2.fm3_16.el9.noarch
rubygem-hammer_cli_foreman_salt-0.1.0-3.fm3_16.el9.noarch
rubygem-hammer_cli_foreman_ssh-0.0.3-2.fm3_16.el9.noarch
foreman-obsolete-packages-1.11-1.el9.noarch
rubygem-hammer_cli_foreman_tasks-0.0.24-1.fm3_17.el9.noarch
rubygem-foreman_puppet-9.1.0-1.fm3_17.el9.noarch
foreman-release-3.18.1-1.el9.noarch
foreman-proxy-3.18.1-1.el9.noarch
rubygem-hammer_cli_foreman-3.18.1-1.el9.noarch
foreman-selinux-3.18.1-1.el9.noarch
foreman-3.18.1-1.el9.noarch
rubygem-foreman-tasks-11.1.1-1.fm3_18.el9.noarch
rubygem-foreman_remote_execution-16.5.3-1.fm3_18.el9.noarch
foreman-dynflow-sidekiq-3.18.1-1.el9.noarch
foreman-libvirt-3.18.1-1.el9.noarch
foreman-postgresql-3.18.1-1.el9.noarch
foreman-redis-3.18.1-1.el9.noarch
foreman-service-3.18.1-1.el9.noarch
foreman-telemetry-3.18.1-1.el9.noarch
foreman-vmware-3.18.1-1.el9.noarch
rubygem-foreman_kubevirt-0.6.0-1.fm3_18.el9.noarch
rubygem-foreman_ovirt-2.0.3-1.fm3_18.el9.noarch
rubygem-foreman_templates-11.0.1-1.fm3_18.el9.noarch
rubygem-foreman_webhooks-5.0.2-1.fm3_18.el9.noarch
foreman-cli-3.18.1-1.el9.noarch
rubygem-hammer_cli_foreman_puppet-0.1.2-1.fm3_18.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.4.2-1.fm3_18.el9.noarch
rubygem-foreman_maintain-1.14.3-1.el9.noarch
foreman-installer-3.18.1-1.el9.noarch

Distribution and version:
Alma Linux 9