Project

General

Profile

Actions

Feature #19496

closed

Passenger graceful killer cron job

Added by Lukas Zapletal almost 7 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
-

Description

Passenger open-source version does support process restarting on memory threshold. It looks like passenger have reserved SIGUSR1 for graceful restart. A cron job could do the job easily:

https://gist.github.com/kascote/366441

http://mensfeld.pl/2012/08/simple-rubyrails-passenger-memory-consumption-limit-monitoring/

Actions #1

Updated by Lukas Zapletal almost 7 years ago

I wrote initial version of a script that gracefully terminates Passenger processes consuming more than 2.5 GB of private RSS memory. Typical consumption of Foreman with Katello and other basic plugins is around 1.5 GB and since only Passenger Enterprise allows to limit maximum amount of RSS memory for processes, this simple script does exactly that:

https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17

Put it into your /etc/cron.hourly/ and make it executable, make sure you read root emails or forward it properly, the script reports all terminations performed on STDOUT. I would like you to test this script in production and get back to me with feedback about what you think.

Motivation is simple - we often introduce bugs in our Rails codebase which performs some eager loading or there are memory leaks in our code or dependencies and Passenger processes can grow up to dozens gigabytes. Unfortunately, there is no other way of getting out other than restarting httpd with passenger. This script could help to avoid situations when production instance starts to swap hard thank to some small regression we introduced. Also administrator will be noticed early via email when this happens so we will keep track of these regressions in production.

Actions #2

Updated by The Foreman Bot almost 7 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Lukas Zapletal
  • Pull request https://github.com/theforeman/foreman/pull/4540 added
Actions #3

Updated by Bryan Kearney almost 7 years ago

  • Bugzilla link set to 1452772
Actions #4

Updated by Anonymous over 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #5

Updated by Marek Hulán over 6 years ago

  • translation missing: en.field_release set to 296
Actions #6

Updated by Marek Hulán over 6 years ago

  • translation missing: en.field_release deleted (296)

Removing release again, foreman maintain is async to foreman releases. I suppose it deserves its own redmine project.

Actions #7

Updated by Ivan Necas over 6 years ago

  • Project changed from Foreman to Foreman Maintain
  • Category deleted (Rake tasks)

Yes, it deserves and it actually has :)

Actions #8

Updated by The Foreman Bot over 6 years ago

  • Pull request https://github.com/theforeman/foreman_maintain/pull/89 added
Actions #9

Updated by The Foreman Bot over 6 years ago

  • Pull request https://github.com/theforeman/foreman_maintain/pull/133 added
Actions

Also available in: Atom PDF