Bug #25021
closed
The termination procedure after memory threshold exceeded can get stuck, waiting infinitely for some events to occur
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1628638
Description of problem:
Version-Release number of selected component (if applicable):
How reproducible:
Occasionally
Steps to Reproduce:
1. set a memory threshold on dynflow executor (EXECUTOR_MEMORY_LIMIT=2gb in /etc/sysconfig/foreman-tasks in sat 6.3, /etc/sysconfig/dynflowd in 6.4
2. run some substantial load on the tasking system continuous resyncing and publishing of CVs might be a good start
3. wait for the memory limit of the dynflow process to cross the limit - watch production.log for 'Memory level exceeded' message
Actual results:
The process still keeps running, termination is not finished, the tasks are not proceeding anymore
Expected results:
The process exits in timely manner and new one gets started
Additional info:
I've written the reproduced based on customer observations, haven't reproducer it locally in production just yet, we however know about places where we wait in the termination phase without any timeout
Updated by Ivan Necas about 6 years ago
- Subject changed from The termination procedure after memory threshold exceeded can get stuck, waiting infinitely for some events to occur to The termination procedure after memory threshold exceeded can get stuck, waiting infinitely for some events to occur
- Status changed from New to Ready For Testing
Updated by Ivan Necas about 6 years ago
- Status changed from Ready For Testing to Closed
- Pull request https://github.com/Dynflow/dynflow/pull/297 added
Updated by Ivan Necas almost 6 years ago
- Blocked by Bug #25593: Dynflow executor termination may hang if there is an action which keeps the executor occupied added