Project

General

Profile

Actions
Copy link

Bug #37662

open

rubygem-redis leaks connections which can result in 'Errno::EMFILE: Too many open files'

Added by Markus Bucher 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
3.9.3
Red Hat JIRA:

Description

rubygem-redis issue: https://github.com/redis/redis-rb/issues/524

We seem to have hit this now multiple times during the foreman-rake db:migrate-step in foreman-installer, when doing a foreman upgrade.
It usually works, when being run the second time.

According to the redis-rb issue mentioned above this is fixed with version >= 5.0.0. Currently in use by foreman is rubygem redis-4.5.1.
I tried upgrading the gem in forklift, but sidekiq seems to be incompatible with redis>=5.0.0:

Bundler could not find compatible versions for gem "redis":
  In Gemfile:
    redis (~> 5.0.0)

    sidekiq (~> 6.5) was resolved to 6.5.12, which depends on
      redis (>= 4.5.0, < 5)

foreman-installer output

2024-07-02 14:08:36 [NOTICE] [configure] 1250 configuration steps out of 1689 steps complete.
2024-07-02 14:09:00 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-07-02 14:09:00 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-07-02 14:09:12 [NOTICE] [configure] 1500 configuration steps out of 1689 steps complete.
2024-07-02 14:09:28 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
    Adding autorequire relationship with User[foreman]
    Starting to evaluate the resource (1432 of 1689)
    Evaluated in 23.46 seconds
  Exec[foreman-rake-db:migrate](provider=posix)
    Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
    Executing '/usr/sbin/foreman-rake db:migrate'
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
    rake aborted!
    Errno::EMFILE: Too many open files - socket(2)
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
    /usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
    /usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:25:in `with'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:487:in `block in delete'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `block in instrument'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/notifications.rb:205:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:726:in `instrument'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/cache.rb:486:in `delete'
    /usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
    /usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
    /usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `instance_exec'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:427:in `block in make_lambda'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `catch'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:603:in `block in default_terminator'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:199:in `block in halting'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `block in invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `each'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:512:in `invoke_before'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/callbacks.rb:105:in `run_callbacks'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/reloader.rb:88:in `prepare!'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `instance_exec'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:32:in `run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
    /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:61:in `block in run_initializers'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/initializable.rb:60:in `run_initializers'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:391:in `initialize!'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `public_send'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/railtie.rb:207:in `method_missing'
    /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
    /usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
    /usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:332:in `block in require'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:299:in `load_dependency'
    /usr/share/gems/gems/activesupport-6.1.7.8/lib/active_support/dependencies.rb:332:in `require'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:367:in `require_environment!'
    /usr/share/gems/gems/railties-6.1.7.8/lib/rails/application.rb:533:in `block in run_tasks_blocks'
    /usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
    Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment
    (See full trace by running task with --trace)
  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
    rake aborted!

No data to display

Actions
Copy link

Also available in: Atom PDF