Bug #33124
openSidekiq Orchestrator gets killed by systemd if Redis lock can't be aquired
Description
Currently, the sidekiq orchestrator uses a Redis lock to mark itself as the active instance.
If a second orchestrator is started, the lock is not available and it therefore blocks here: https://github.com/theforeman/foreman/blob/develop/extras/dynflow-sidekiq.rb#L34
As a result, systemd will never get notified and there kill the service after the timeout: https://github.com/theforeman/foreman/blob/develop/extras/systemd/dynflow-sidekiq%40.service#L9
This results in the second instance being either completely down or restarted everytime Puppet runs (which results in a loop).
There should be a state where the service is running as standby (as in "initialized correctly but waiting for lock") and systemd is notified about that (so the process does not get killed).
Foreman Community Thread: https://community.theforeman.org/t/dynflow-and-high-availability/24558/4
No data to display