Project

General

Profile

Actions

Bug #38085

closed

On large deployments puma auto tuning results in too many workers for PostgreSQL connections

Added by Ewoud Kohl van Wijngaarden 23 days ago. Updated 21 days ago.

Status:
Closed
Priority:
Normal
Category:
Foreman modules
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

The current code we have for this:

https://github.com/theforeman/puppet-foreman/blob/907b63962ef54842a166951cdb6c93a22181e907/manifests/config.pp#L89-L99

Or in words:

Take the minimum value of CPU count * 1.5 and RAM in GB - 1.5, then round that down.

So for the specific case of 128 CPU cores (output of nproc) and 250 GB RAM: min(128 * 1.5, 250 - 1.5) = 192.0.

Now that's the number of Puma workers. Each worker uses up some database connections:

https://github.com/theforeman/puppet-foreman/blob/907b63962ef54842a166951cdb6c93a22181e907/manifests/config.pp#L71-L79

The default for puma_threads_max is 5, so it results in 9 database connections per worker.

9 * 192 = 1728

That is a lot. In practice only one worker should use that + 4 part, but let's assume the worst case.

PostgreSQL is configured with a max_connections, which we set in foreman-installer using Hiera:

$ grep -R max_connections config/foreman.hiera/tuning/
config/foreman.hiera/tuning/sizes/extra-extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/medium.yaml: max_connections: 1000
config/foreman.hiera/tuning/common.yaml: max_connections: 500

Note that's server wide and also needs to account for anything that Pulp and Candlepin use.

My guess is that any server with more than 64 cores ends up in a problematic case, possibly even sooner.

Actions

Also available in: Atom PDF