Bug #38085
closedOn large deployments puma auto tuning results in too many workers for PostgreSQL connections
Description
The current code we have for this:
Or in words:
Take the minimum value of CPU count * 1.5 and RAM in GB - 1.5, then round that down.
So for the specific case of 128 CPU cores (output of nproc) and 250 GB RAM: min(128 * 1.5, 250 - 1.5) = 192.0.
Now that's the number of Puma workers. Each worker uses up some database connections:
The default for puma_threads_max is 5, so it results in 9 database connections per worker.
9 * 192 = 1728
That is a lot. In practice only one worker should use that + 4 part, but let's assume the worst case.
PostgreSQL is configured with a max_connections, which we set in foreman-installer using Hiera:
$ grep -R max_connections config/foreman.hiera/tuning/
config/foreman.hiera/tuning/sizes/extra-extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/medium.yaml: max_connections: 1000
config/foreman.hiera/tuning/common.yaml: max_connections: 500
Note that's server wide and also needs to account for anything that Pulp and Candlepin use.
My guess is that any server with more than 64 cores ends up in a problematic case, possibly even sooner.
Updated by The Foreman Bot 20 days ago
- Status changed from New to Ready For Testing
- Assignee set to Ewoud Kohl van Wijngaarden
- Pull request https://github.com/theforeman/puppet-foreman/pull/1197 added
Updated by Ewoud Kohl van Wijngaarden 18 days ago
- Status changed from Ready For Testing to Closed
Applied in changeset puppet-foreman|0e8bb06de57a86d087fce76058d9e40748afd52a.