Bug #10362
closeddynflow_executor_monitor taking up 100% CPU
Description
I recently installed the latest version of Katello which comes with Foreman version 1.9.0. When the system is idle (no sync or content view creations in process) the process dynflow_executor_monitor constantly takes up 100% CPU.
I verified there is no activity in the /var/log/foreman/* logs. In addition I verified there was no recent output in /usr/share/foreman/tmp/pids where the dynflow_executor.output and log file is located.
I then did a strace on the monitor PID and it appears to be constantly spinning on stating the dynflow_executor.pid file, opening it, reading it and then closing it. It then attempts to send SIG_0 to the dynflow_executor process (so checking whether the process exists).
Attached is the output from a strace run.
stat("/usr/share/foreman/tmp/pids/dynflow_executor.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
open("/usr/share/foreman/tmp/pids/dynflow_executor.pid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff8da280e0) = -1 ENOTTY (Inappropriate ioctl for device)
read(5, "27794\n", 8192) = 6
close(5) = 0
I also noted that there are two dynflow_executor processes running, with multiple threads for each one.
Files
Updated by Dominic Cleal over 9 years ago
- Project changed from Foreman to foreman-tasks
Updated by Ivan Necas over 9 years ago
What OS version are you running on? The fact that who dynflow_executor processes are running there is
an indication something went wrong (and that might be the cause for the 100% CPU): such
a situation should not happen. Are you able to reproduce this state, when more dynflow executors
are getting started?
Updated by Matthew Ceroni over 9 years ago
I am running on CentOS version 6.6.
The issue persists all the time. I simply have to start up Foreman (and foreman-tasks) and dynflow_executor kicks off and stays running. For now I kill the monitor process but left the executor processes running. Foreman seems to function just fine and the CPU usage is back to normal.
Updated by Ivan Necas over 9 years ago
- Status changed from New to Assigned
I've finally discovered the problem: we've updated the daemons gem recently to 1.2.1 in nightlies, which introduced this bug, should be fixed in daemons 1.2.2: I will open a PRs to updated that gem in nightlies
Thanks for reporting the issue
Updated by The Foreman Bot over 9 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman-packaging/pull/650 added
- Pull request deleted (
)
Updated by Dominic Cleal over 9 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
New package is building and should be published in the next 24 hours. Thanks for the report.
In the meantime, pick it up from http://koji.katello.org/koji/taskinfo?taskID=260634.