Project

General

Profile

Actions

Bug #33124

open

Sidekiq Orchestrator gets killed by systemd if Redis lock can't be aquired

Added by Manuel Laug over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

Currently, the sidekiq orchestrator uses a Redis lock to mark itself as the active instance.
If a second orchestrator is started, the lock is not available and it therefore blocks here: https://github.com/theforeman/foreman/blob/develop/extras/dynflow-sidekiq.rb#L34

As a result, systemd will never get notified and there kill the service after the timeout: https://github.com/theforeman/foreman/blob/develop/extras/systemd/dynflow-sidekiq%40.service#L9

This results in the second instance being either completely down or restarted everytime Puppet runs (which results in a loop).

There should be a state where the service is running as standby (as in "initialized correctly but waiting for lock") and systemd is notified about that (so the process does not get killed).

Foreman Community Thread: https://community.theforeman.org/t/dynflow-and-high-availability/24558/4

No data to display

Actions

Also available in: Atom PDF