Project

General

Profile

Bug #5568

Possible memory leak in trends

Added by Lukas Zapletal about 5 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Trends
Target version:
Difficulty:
Triaged:
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

By Sean Alderman (https://groups.google.com/forum/#!msg/foreman-users/hi7TagIRE8A/5ZRzlI8ChxEJ):

I am collecting a few trends in foreman using host facts. I'm using a cron entry for the foreman user: /usr/sbin/foreman-rake trends:counter every half hour.

My foreman machine is running puppet, puppetdb, foreman, foreman-proxy, and postgres, 2 vCPU and 6GB of RAM. It seems that the rake commands are running away with all the memory and swap. Is there a way to control this?

Here's what top shows currently:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                               
30002 foreman   20   0 6602m 3.3g  104 D 13.3 58.3  17:06.22 /opt/rh/ruby193/root/usr/bin/ruby /opt/rh/ruby193/root/usr/bin/rake trends:clean                                                                      

Seems like a lot of RAM and accumulated CPU time. I can add more RAM, but other than rake commands like this one, the server doesn't seem to utilize the memory allocated.


Related issues

Related to Foreman - Bug #4114: Trends don't scale well and become very slowDuplicate2014-01-17

Associated revisions

Revision 040abfa3 (diff)
Added by Jon McKenzie about 4 years ago

Fixes #5568 - Improves performance of trends:clean rake task

Perform trend counter dupe counting within the database rather than in the Ruby code.

Revision 5b7ac025 (diff)
Added by Jon McKenzie about 4 years ago

Fixes #5568 - Improves performance of trends:clean rake task

Perform trend counter dupe counting within the database rather than in the Ruby code.

(cherry picked from commit 040abfa32f326de7cd488f34f243377f76fd70ae)

Conflicts:
test/lib/tasks/trends_test.rb

History

#1 Updated by Dominic Cleal about 5 years ago

  • Related to Bug #4114: Trends don't scale well and become very slow added

#2 Updated by Edson Manners almost 5 years ago

Seem to have a similar issue. Currently running 1.4.5 on RHEL 6.5.
We have 545 hosts and I'm not running and anytrends manually foreman seems ot be running these trends on it's own.
I'm running this in a KVM VM with 6 cores and 16GB RAM ot this point but it's obviously never enough.
I'll be upgrading through 1.5 and 1.6 in the coming days and looking to see if that helps.

Top show this after 'service foreman stop; service httpd stop'
30143 foreman 20 0 8407m 7.9g 6544 D 11.6 50.9 42:44.19 rake

I have to 'kill -9' the rake process as it doesn't die on it's own.

'ps -lef | grep 30143'
/opt/rh/ruby193/root/usr/bin/ruby /opt/rh/ruby193/root/usr/bin/rake trends:clean

#3 Updated by Shimon Shtein over 4 years ago

Can it be related to the amount of data in the trend_counters table?

#4 Updated by Jon McKenzie about 4 years ago

We're running into this issue too. I haven't done a ton with ActiveRecord, but the issue are these lines in the trends:clean rake task:

counts = TrendCounter.group([:trend_id, :created_at]).count

dupes = counts.select{ |attrs, count| count > 1}

Rather than doing the aggregation inside the database, the task first executes a very expensive GROUP and then filters the results it needs in plain Ruby.

Again, not an ActiveRecord expert, but I think replacing those two lines with the following should significantly speed it up:

dupes = TrendCounter.having('count(*) > 1').group([:trend_id, :created_at]).count

This translates the SQL from this:

SELECT COUNT(*) AS count_all, trend_id AS trend_id, created_at AS created_at FROM "trend_counters" GROUP BY trend_id, created_at ORDER BY created_at

...to this:

SELECT COUNT(*) AS count_all, trend_id AS trend_id, created_at AS created_at FROM "trend_counters" GROUP BY trend_id, created_at HAVING count(*) > 1 ORDER BY created_at

#5 Updated by Jon McKenzie about 4 years ago

Huh, I must have misread this issue initially, which is for trends:counter not trends:clean. Oh well, I submitted a PR for the trends:clean issue anyways

#6 Updated by Dominic Cleal about 4 years ago

  • Status changed from Assigned to Ready For Testing
  • Assignee deleted (Lukas Zapletal)
  • Pull request https://github.com/theforeman/foreman/pull/2365 added
  • Pull request deleted ()

#7 Updated by Jon McKenzie about 4 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#8 Updated by Dominic Cleal about 4 years ago

  • Assignee set to Jon McKenzie
  • Legacy Backlogs Release (now unused) set to 50

Also available in: Atom PDF