Project

General

Profile

Actions

Bug #33974

closed

Getting http 500 internal server error due to "ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds"

Added by William Clark over 2 years ago. Updated 23 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1976728

Description of problem:
Getting the following error on default 2 workers 16 threads Puma configurations when sending more than 16 query requests at the same time.
----------------------------------------------
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use
----------------------------------------------

The error is gone after settings the connection pools to 32 (number of worker * number of threads)

/usr/share/foreman/config/database.yml
  1. Database is managed by foreman::database::postgresql
    production:
    adapter: postgresql
    database: foreman
    username: foreman
    password: ***
    pool: 32 <==============

Steps to Reproduce:
1. On any remote host or Satellite itself, run the command to send many concurrent requests to the web server using irb console

irb
require 'rest_client'
50.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/v2/hosts?installed_package_name=kernel&page=1&per_page=200", user: "admin", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end } }

2. On Satellite, tail the /var/log/foreman/production.log

Actual results:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 6.756 seconds); all pooled connections were in use

Requests getting 500 internal server error

Expected results:
No error

Additional info:
I thought each worker process has a separate connection pool but somehow it seems like it is not. Not sure.

Actions #1

Updated by The Foreman Bot 3 months ago

  • Status changed from New to Ready For Testing
  • Assignee set to Shimon Shtein
  • Pull request https://github.com/theforeman/puppet-foreman/pull/1161 added
Actions #2

Updated by The Foreman Bot 2 months ago

  • Pull request https://github.com/theforeman/foreman-installer/pull/937 added
Actions #3

Updated by The Foreman Bot 2 months ago

  • Fixed in Releases 3.11.0 added
Actions #4

Updated by Shimon Shtein 2 months ago

  • Status changed from Ready For Testing to Closed
Actions #5

Updated by Ewoud Kohl van Wijngaarden 23 days ago

  • Triaged changed from No to Yes
Actions

Also available in: Atom PDF