Bug #24278
Updated by Ivan Necas almost 7 years ago
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1602110 *How reproducible:* occasionally *Steps to Reproduce:* 1. schedule remote execution job against larger number of hosts (300+) - to ensure some tasks will be actually running during the cleanup 2. foreman-rake foreman_tasks:cleanup TASK_SEARCH='' STATES='running' 3. schedule new remote execution job *Expected results:* There is no info about "PersistenceError in executor: terminating" in production.log, the executor is able to process new requests after this incident without restart. *Actual results:* There is the following error, followed by the executor either restarting or getting stuck. <pre> 2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] The number of MAX_RETRIES exceeded 2018-06-20 18:36:27 [foreman-tasks/dynflow] [F] PersistenceError in executor: terminating 2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] PG::Error: ERROR: insert or update on table "dynflow_actions" violates foreign key constraint "dynflow_actions_execution_plan_uuid_fkey" | DETAIL: Key (execution_plan_uuid)=(6efc7c85-46ed-4119-9f35-710d12e4f72b) is not present in table "dynflow_execution_plans". | (Sequel::ForeignKeyConstraintViolation) | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `exec' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `block in execute_query' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `execute_query' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:143:in `check_disconnect_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511:in `_execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532:in `check_database_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize' </pre> Also trying to rerun other jobs after this incided without restarting foreman-tasks might not work. *Additional info:*