Project

General

Profile

Actions

Bug #10362

closed

dynflow_executor_monitor taking up 100% CPU

Added by Matthew Ceroni about 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

I recently installed the latest version of Katello which comes with Foreman version 1.9.0. When the system is idle (no sync or content view creations in process) the process dynflow_executor_monitor constantly takes up 100% CPU.

I verified there is no activity in the /var/log/foreman/* logs. In addition I verified there was no recent output in /usr/share/foreman/tmp/pids where the dynflow_executor.output and log file is located.

I then did a strace on the monitor PID and it appears to be constantly spinning on stating the dynflow_executor.pid file, opening it, reading it and then closing it. It then attempts to send SIG_0 to the dynflow_executor process (so checking whether the process exists).

Attached is the output from a strace run.

stat("/usr/share/foreman/tmp/pids/dynflow_executor.pid", {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
open("/usr/share/foreman/tmp/pids/dynflow_executor.pid", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0
ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff8da280e0) = -1 ENOTTY (Inappropriate ioctl for device)
read(5, "27794\n", 8192) = 6
close(5) = 0

I also noted that there are two dynflow_executor processes running, with multiple threads for each one.


Files

dynflow_executor_monitor.strace dynflow_executor_monitor.strace 8.94 KB Matthew Ceroni, 05/04/2015 01:41 PM
Actions #1

Updated by Dominic Cleal about 9 years ago

  • Project changed from Foreman to foreman-tasks
Actions #2

Updated by Ivan Necas about 9 years ago

What OS version are you running on? The fact that who dynflow_executor processes are running there is
an indication something went wrong (and that might be the cause for the 100% CPU): such
a situation should not happen. Are you able to reproduce this state, when more dynflow executors
are getting started?

Actions #3

Updated by Matthew Ceroni about 9 years ago

I am running on CentOS version 6.6.

The issue persists all the time. I simply have to start up Foreman (and foreman-tasks) and dynflow_executor kicks off and stays running. For now I kill the monitor process but left the executor processes running. Foreman seems to function just fine and the CPU usage is back to normal.

Actions #4

Updated by Ivan Necas about 9 years ago

  • Status changed from New to Assigned

I've finally discovered the problem: we've updated the daemons gem recently to 1.2.1 in nightlies, which introduced this bug, should be fixed in daemons 1.2.2: I will open a PRs to updated that gem in nightlies

Thanks for reporting the issue

Actions #5

Updated by Ivan Necas about 9 years ago

  • Assignee set to Ivan Necas
Actions #6

Updated by The Foreman Bot about 9 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman-packaging/pull/650 added
  • Pull request deleted ()
Actions #7

Updated by Dominic Cleal about 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

New package is building and should be published in the next 24 hours. Thanks for the report.

In the meantime, pick it up from http://koji.katello.org/koji/taskinfo?taskID=260634.

Actions

Also available in: Atom PDF