Bug #24278
closedThe number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1602110
How reproducible:
occasionally
Steps to Reproduce:
1. schedule remote execution job against larger number of hosts (300+) - to ensure some tasks will be actually running during the cleanup
2. foreman-rake foreman_tasks:cleanup TASK_SEARCH='' STATES='running'
3. schedule new remote execution job
Expected results:
There is no info about "PersistenceError in executor: terminating" in production.log, the executor is able to process new requests after this incident without restart.
Actual results:
There is the following error, followed by the executor either restarting or getting stuck.
2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] The number of MAX_RETRIES exceeded 2018-06-20 18:36:27 [foreman-tasks/dynflow] [F] PersistenceError in executor: terminating 2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] PG::Error: ERROR: insert or update on table "dynflow_actions" violates foreign key constraint "dynflow_actions_execution_plan_uuid_fkey" | DETAIL: Key (execution_plan_uuid)=(6efc7c85-46ed-4119-9f35-710d12e4f72b) is not present in table "dynflow_execution_plans". | (Sequel::ForeignKeyConstraintViolation) | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `exec' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `block in execute_query' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `execute_query' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:143:in `check_disconnect_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511:in `_execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block (2 levels) in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532:in `check_database_errors' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block in execute' | /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize'
Also trying to rerun other jobs after this incided without restarting foreman-tasks might not work.
Additional info:
Updated by Ivan Necas over 5 years ago
- Subject changed from The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database to The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database
- Description updated (diff)
Updated by Adam Ruzicka over 5 years ago
- Category set to Dynflow
- Status changed from New to Closed
- Assignee set to Ivan Necas
- Pull request https://github.com/Dynflow/dynflow/pull/292 added