Plugins » foreman-tasks

A user (rashkinadze and gkoch) in #theforeman saw this issue. Immediately after upgrading to 1.16.1 Katello Host::Update tasks started piling up in planned/pending state and were never being processed. Restarting all the services seemed to do nothing to resolve the issue.

Looking through their logs I really did not see much, but noticed in their postgresql logs they seemed to see this anytime foreman-tasks was restarted:

2018-04-17 10:36:41 EDT ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
2018-04-17 10:36:41 EDT DETAIL: Key (id, class)=(delayed-executor, Dynflow::Coordinator::DelayedExecutorLock) already exists.
2018-04-17 10:36:41 EDT STATEMENT: INSERT INTO "dynflow_coordinator_records" ("data", "id", "owner_id", "class") VALUES ('{"class":"Dynflow::Coordinator::DelayedExecutorLock","owner_id":"world:89c713a9-ffd3-49b2-ad84-c921fad5e75e","world_id":"89c713a9-ffd3-49b2-ad84-c921fad5e75e","id":"delayed-executor"}', 'delayed-executor', 'world:89c713a9-ffd3-49b2-ad84-c921fad5e75e', 'Dynflow::Coordinator::DelayedExecutorLock') RETURNING "id"

I had them go to dynflow status and invalidate their worlds. 4 worlds were invalidated and after a foreman-tasks restart, things seemed to be processed.

Any idea why this happened or what we can do to prevent manual intervention?