Bug #30192
openUnable to recycle the dynflow executor
Description
Cloned from: https://bugzilla.redhat.com/show_bug.cgi?id=1849869
Description of problem:
Unable to recycle executor when the memory limit is reached. The world got terminated but no new world is created. The executor process doesn't exit.
Steps to Reproduce:
Setting the "EXECUTOR_MEMORY_LIMIT" to something very low which will trigger the refresh pretty much 60 seconds after the starting the executor.
----------------------------
#Set memory limit for executor process, before it's restarted automatically
EXECUTOR_MEMORY_LIMIT=200mb
#Set delay before first memory polling to let executor initialize (in sec)
EXECUTOR_MEMORY_MONITOR_DELAY=60 #default: 2 hours
----------------------------
Observed that:
- the world got terminated when the memory limit is reached.
- memory is not freed and no more executor showing in the Dynflow status page.
- running task will enter paused stated and never able to resume/cancel even after restarting the dynflowd.
Log:
Everything ready for world: da8f1461-7891-47a1-a39a-2ff44c0492bd
World has been terminated
<getting dead letter error>
- ps -aux | grep -i dynflow
foreman 18794 22.8 2.0 2572916 409472 ? Sl 17:37 1:05 dynflow_executor
foreman 18795 0.0 0.6 581248 126556 ? Sl 17:37 0:00 dynflow_executor_monitor
Same behaviour on both Satellite 6.6 and 6.7 but working on Satellite 6.5.
Actual results:
failed to recycle executor
Expected results:
executor can be recycled successfully
Additonal info:
I think I found the issue:
In "world.rb"
...
end.on_resolution do
@terminated.complete <==== This is wrong. Remove this line will solve the issue.
Thread.new { Kernel.exit } if @exit_on_terminate.true?
end
Updated by Adam Ruzicka about 5 years ago
- Assignee set to Adam Ruzicka
- Pull request https://github.com/Dynflow/dynflow/pull/357 added