Project

General

Profile

Actions

Feature #18821

closed

Set PassengerMaxRequests and PassengerMemoryLimit to reasonable values

Added by Lukas Zapletal about 7 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Category:
Foreman modules
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

We leave these values unconfigured which means no restarting is performed. Ruby VM is known to grow on memory and due nature of it's memory model, it never returns memory back to OS and due fragmentation it grows slowly. The only way out on CRuby (MRI) is to restart processes after some time, it should not happend more than once per day as a generic rule of thumb.

Now, Passenger provides tools to do this automatically, it can restart after N served requests and/or after memory goes above threshold. We should be able to setup reasonable default values (100,000 requests and 1 GB memory per instance) to prevent OOM kills which happens on production installs.

https://www.phusionpassenger.com/library/config/apache/reference/#passengermemorylimit

https://www.phusionpassenger.com/library/config/apache/reference/#passengermaxrequests

Actions #1

Updated by Ewoud Kohl van Wijngaarden about 7 years ago

Given the PassengerMemoryLimit has a big warning that it's enterprise only we can't use that. For the other setting they mention 'This option should be considered as a workaround for misbehaving applications. It is advised that you fix the problem in your application rather than relying on this option as a measure to avoid memory leaks.' so setting this by default doesn't feel right.

As a workaround you can add the following to /etc/foreman-installer/custom-hiera.yaml

---
apache::mod::passenger::passenger_max_requests: 100000

This will set it globally for the server, but currently there's no parameter to easily set for the single vhost.

Actions #2

Updated by Ewoud Kohl van Wijngaarden about 7 years ago

I just talked to Timo and they deploy it with max requests set to 10000, at least on their puppet masters. Please submit a patch to https://github.com/theforeman/foreman-installer/blob/develop/config/foreman.hiera/tuning.yaml to fix it for Foreman. I don't know if Katello will also use it then but that should be easy to verify.

Actions #3

Updated by Lukas Zapletal about 7 years ago

Ewoud, this is not a memory leak, every single Ruby application eventually needs restarting, if it allocates some memory. There was a great talk at FOSDEM about this. Ruby will effectively never return allocated memory to OS, so GC will not help at all and the only way to free memory is to terminate the process.

I don't understand why you accept 100000 requests limit but not like 1 GB top limit, which is IMHO much more useful than 100000 which is little bit artificial (while we know that Foreman w/ Katello takes about 400 MB so 1 GB is sane default to initiate restart).

Actions #4

Updated by Ewoud Kohl van Wijngaarden about 7 years ago

The reason I wouldn't accept the memory limit is that as I read it, it's not available in the open source release, only in the enterprise release. That means it's not available to us.

Actions #5

Updated by Lukas Zapletal about 7 years ago

Oh really is it? Is this cripple-ware? Jeeez. I had no idea, I live in a perfect world I guess :-)

I will make the patch then.

Actions #6

Updated by The Foreman Bot about 7 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Lukas Zapletal
  • Pull request https://github.com/theforeman/foreman-installer/pull/221 added
Actions #7

Updated by Lukas Zapletal about 7 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #8

Updated by Dominic Cleal almost 7 years ago

  • translation missing: en.field_release set to 209
Actions

Also available in: Atom PDF