Bug #25021
closed
The termination procedure after memory threshold exceeded can get stuck, waiting infinitely for some events to occur
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1628638
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Occasionally
Steps to Reproduce:
1. set a memory threshold on dynflow executor (EXECUTOR_MEMORY_LIMIT=2gb in /etc/sysconfig/foreman-tasks in sat 6.3, /etc/sysconfig/dynflowd in 6.4
2. run some substantial load on the tasking system continuous resyncing and publishing of CVs might be a good start
3. wait for the memory limit of the dynflow process to cross the limit - watch production.log for 'Memory level exceeded' message
Actual results:
The process still keeps running, termination is not finished, the tasks are not proceeding anymore
Expected results:
The process exits in timely manner and new one gets started
Additional info:
I've written the reproduced based on customer observations, haven't reproducer it locally in production just yet, we however know about places where we wait in the termination phase without any timeout
Updated by