Project

General

Profile

Bug #33124

Sidekiq Orchestrator gets killed by systemd if Redis lock can't be aquired

Added by Manuel Laug over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Currently, the sidekiq orchestrator uses a Redis lock to mark itself as the active instance.
If a second orchestrator is started, the lock is not available and it therefore blocks here: https://github.com/theforeman/foreman/blob/develop/extras/dynflow-sidekiq.rb#L34

As a result, systemd will never get notified and there kill the service after the timeout: https://github.com/theforeman/foreman/blob/develop/extras/systemd/dynflow-sidekiq%40.service#L9

This results in the second instance being either completely down or restarted everytime Puppet runs (which results in a loop).

There should be a state where the service is running as standby (as in "initialized correctly but waiting for lock") and systemd is notified about that (so the process does not get killed).

Foreman Community Thread: https://community.theforeman.org/t/dynflow-and-high-availability/24558/4

Also available in: Atom PDF