Plugins » foreman-tasks

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1368103
I'm doing some ruby mem experiments and sometime around 07:59:07 (+-10s) ruby went from:

3290628 ruby

to:

2149180 ruby

this is in the production.log:

[...]
  2016-08-18 07:57:28 [app] [I] Completed 200 OK in 998ms (Views: 257.5ms | ActiveRecord: 730.5ms)
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating throttle_limiter...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating throttle_limiter...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating client dispatcher...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] stop listening for new events...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating client dispatcher...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] stop listening for new events...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating clock...
  2016-08-18 07:59:03 [foreman-tasks/dynflow] [I] start terminating clock...
  2016-08-18 07:59:43 [app] [I] Started GET "/foreman_tasks/tasks/3cee987c-7266-4532-a222-bac6b8355029/sub_tasks" for 213.211.43.129 at 2016-08-18 07:59:43 -0400
  [...]

Any idea on what it might be?

I use this crazy stuff to "measure" ruby RSS usage (well, not only ruby,
but it is in top 10):

ps --no-headers -eo rss,comm>a; for comm in $( sed 's/^\s*[0-9]\+\s*\(.*\)$/\1/' a | sort -u ); do size=$( grep "\s$comm" a | sed 's/^\s*\([0-9]\+\)\s*.*$/\1/' | paste -sd+ - | bc ); echo "$size $comm"; done | sort -n | tail

and here is what I have done:

1. I have clients registered through capsules with gofferd/katello-agent running
  2. I have shut down capsules, so clients can not reach satellite
  3. I have scheduled errata update on 400 clients
  4. I was sitting and looking at the ruby mem usage

Around that 07:59:07, I have not touched satellite at all.

I'm looking into this, because Pradeep noticed at some point ruby was
consuming 43GB RSS and we believe this is connected with some stuck
tasks (had some problems with qpidd/qdrouterd/clients itself - so our
10k errata update task was a bit mess at first two tries).