Project

General

Profile

Actions

Bug #24278

closed

The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database

Added by Ivan Necas over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Dynflow
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1602110

How reproducible:
occasionally

Steps to Reproduce:
1. schedule remote execution job against larger number of hosts (300+) - to ensure some tasks will be actually running during the cleanup
2. foreman-rake foreman_tasks:cleanup TASK_SEARCH='' STATES='running'
3. schedule new remote execution job

Expected results:

There is no info about "PersistenceError in executor: terminating" in production.log, the executor is able to process new requests after this incident without restart.

Actual results:

There is the following error, followed by the executor either restarting or getting stuck.

2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] The number of MAX_RETRIES exceeded
2018-06-20 18:36:27 [foreman-tasks/dynflow] [F] PersistenceError in executor: terminating
2018-06-20 18:36:27 [foreman-tasks/dynflow] [E] PG::Error: ERROR: insert or update on table "dynflow_actions" violates foreign key constraint "dynflow_actions_execution_plan_uuid_fkey" 
| DETAIL: Key (execution_plan_uuid)=(6efc7c85-46ed-4119-9f35-710d12e4f72b) is not present in table "dynflow_execution_plans". 
| (Sequel::ForeignKeyConstraintViolation) 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `exec' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `block in execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/logging.rb:33:in `log_yield' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:180:in `execute_query' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:143:in `check_disconnect_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:167:in `execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:511:in `_execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block (2 levels) in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:532:in `check_database_errors' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/adapters/postgres.rb:335:in `block in execute' 
| /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/connecting.rb:250:in `block in synchronize'

Also trying to rerun other jobs after this incided without restarting foreman-tasks might not work.

Additional info:

Actions #1

Updated by Ivan Necas over 5 years ago

  • Subject changed from The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database to The number of MAX_RETRIES exceeded: PersistenceError in executor: terminating when running tasks are removed from database
  • Description updated (diff)
Actions #2

Updated by Adam Ruzicka over 5 years ago

  • Category set to Dynflow
  • Status changed from New to Closed
  • Assignee set to Ivan Necas
  • Pull request https://github.com/Dynflow/dynflow/pull/292 added
Actions

Also available in: Atom PDF