Bug #10362
closeddynflow_executor_monitor taking up 100% CPU
Description
I recently installed the latest version of Katello which comes with Foreman version 1.9.0. When the system is idle (no sync or content view creations in process) the process dynflow_executor_monitor constantly takes up 100% CPU.
I verified there is no activity in the /var/log/foreman/* logs. In addition I verified there was no recent output in /usr/share/foreman/tmp/pids where the dynflow_executor.output and log file is located.
I then did a strace on the monitor PID and it appears to be constantly spinning on stating the dynflow_executor.pid file, opening it, reading it and then closing it. It then attempts to send SIG_0 to the dynflow_executor process (so checking whether the process exists).
Attached is the output from a strace run.
stat("/usr/share/foreman/tmp/pids/dynflow_executor.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
open("/usr/share/foreman/tmp/pids/dynflow_executor.pid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff8da280e0) = -1 ENOTTY (Inappropriate ioctl for device)
read(5, "27794\n", 8192) = 6
close(5) = 0
I also noted that there are two dynflow_executor processes running, with multiple threads for each one.
Files
Updated by Dominic Cleal about 10 years ago
- Project changed from Foreman to foreman-tasks
Updated by Ivan Necas about 10 years ago
What OS version are you running on? The fact that who dynflow_executor processes are running there is
an indication something went wrong (and that might be the cause for the 100% CPU): such
a situation should not happen. Are you able to reproduce this state, when more dynflow executors
are getting started?
Updated by Matthew Ceroni about 10 years ago
I am running on CentOS version 6.6.
The issue persists all the time. I simply have to start up Foreman (and foreman-tasks) and dynflow_executor kicks off and stays running. For now I kill the monitor process but left the executor processes running. Foreman seems to function just fine and the CPU usage is back to normal.
Updated by Ivan Necas about 10 years ago
- Status changed from New to Assigned
I've finally discovered the problem: we've updated the daemons gem recently to 1.2.1 in nightlies, which introduced this bug, should be fixed in daemons 1.2.2: I will open a PRs to updated that gem in nightlies
Thanks for reporting the issue
Updated by The Foreman Bot about 10 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman-packaging/pull/650 added
- Pull request deleted (
)
Updated by Dominic Cleal about 10 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
New package is building and should be published in the next 24 hours. Thanks for the report.
In the meantime, pick it up from http://koji.katello.org/koji/taskinfo?taskID=260634.