Bug #12650

dynflow_executor memory usage continues to grow, causing performance degredation

Added by Bryan Kearney over 2 years ago. Updated almost 2 years ago.

Assigned To:Ivan Necas
Target version:-
Difficulty: Pull request:https://github.com/Katello/katello/pull/6065
Bugzilla link:1283582
Story points-
Velocity based estimate-
ReleaseKatello 3.1.0Release relationshipAuto


Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1283582
Description of problem:

The memory usage of dynflow_executor continues to grow over time.
As a conequence of this, operations like content view publishes
take longer and longer to complete.

Restarting the Satellite improves the performance, which then
continues to degrade over time

Version-Release number of selected component (if applicable):

6.1.3, 6.1.4

How reproducible:


Steps to Reproduce:
1. Use Satellite

Actual results:

dynflow_executor memory usage continues to grow over time

Expected results:

memory usage should be reach a steady state and rise and
fall as work is done.

Additional info:

I used the following script to collect process mem stats
every 15 minutes


PID=`pidof dynflow_executor`
cat /proc/$PID/status | grep ^Vm >> $LOGFILE

The I can review the VmPeak, VmData etc sizes.

The only time these figures go down is on a process restart.

Associated revisions

Revision 7f661597
Added by Ivan Necas almost 2 years ago

Fixes #12650 - make sure we sync on QPID acknowledge (#6065)

It seems the way we use the QPID client library causes the memory leaks
described in https://issues.apache.org/jira/browse/QPID-3321

Steps to test:

while subscription-manager register --username admin\
--password changeme --org 'Default Organization'\
--environment Library --force; do I=$((I+1)); echo
$I; done

In other shell:

PID=`/usr/sbin/pidof dynflow_executor`
watch "cat /proc/$PID/status | grep ^VmData"

and watch the number grow without the fix and stay still with the fix


#1 Updated by Ivan Necas almost 2 years ago

  • Project changed from foreman-tasks to Katello
  • Category set to Subscriptions

#2 Updated by The Foreman Bot almost 2 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/6065 added

#3 Updated by Eric Helms almost 2 years ago

  • Release set to Katello 3.1.0

#4 Updated by Ivan Necas almost 2 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF