Project

General

Profile

Actions

Bug #38085

closed

On large deployments puma auto tuning results in too many workers for PostgreSQL connections

Added by Ewoud Kohl van Wijngaarden 20 days ago. Updated 18 days ago.

Status:
Closed
Priority:
Normal
Category:
Foreman modules
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

The current code we have for this:

https://github.com/theforeman/puppet-foreman/blob/907b63962ef54842a166951cdb6c93a22181e907/manifests/config.pp#L89-L99

Or in words:

Take the minimum value of CPU count * 1.5 and RAM in GB - 1.5, then round that down.

So for the specific case of 128 CPU cores (output of nproc) and 250 GB RAM: min(128 * 1.5, 250 - 1.5) = 192.0.

Now that's the number of Puma workers. Each worker uses up some database connections:

https://github.com/theforeman/puppet-foreman/blob/907b63962ef54842a166951cdb6c93a22181e907/manifests/config.pp#L71-L79

The default for puma_threads_max is 5, so it results in 9 database connections per worker.

9 * 192 = 1728

That is a lot. In practice only one worker should use that + 4 part, but let's assume the worst case.

PostgreSQL is configured with a max_connections, which we set in foreman-installer using Hiera:

$ grep -R max_connections config/foreman.hiera/tuning/
config/foreman.hiera/tuning/sizes/extra-extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/extra-large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/large.yaml: max_connections: 1000
config/foreman.hiera/tuning/sizes/medium.yaml: max_connections: 1000
config/foreman.hiera/tuning/common.yaml: max_connections: 500

Note that's server wide and also needs to account for anything that Pulp and Candlepin use.

My guess is that any server with more than 64 cores ends up in a problematic case, possibly even sooner.

Actions #1

Updated by The Foreman Bot 20 days ago

  • Status changed from New to Ready For Testing
  • Assignee set to Ewoud Kohl van Wijngaarden
  • Pull request https://github.com/theforeman/puppet-foreman/pull/1197 added
Actions #2

Updated by The Foreman Bot 18 days ago

  • Fixed in Releases 3.14.0 added
Actions #3

Updated by Ewoud Kohl van Wijngaarden 18 days ago

  • Status changed from Ready For Testing to Closed
Actions

Also available in: Atom PDF