Bug #28931: If seeding fails, Passenger refuses to start - Foreman

Actions

Copy link

Bug #28931

closed

If seeding fails, Passenger refuses to start

Added by Ewoud Kohl van Wijngaarden about 4 years ago. Updated about 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Tomer Brisker

Category:

DB migrations

Target version:

2.0.0

Difficulty:

Triaged:

Bugzilla link:

Pull request:

https://github.com/theforeman/foreman/pull/7459

Fixed in Releases:

2.0.0

Found in Releases:

Red Hat JIRA:

Description

When running the installer, I got the following error

 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[centos7-foreman-nightly.wisse.example.com]: Could not evaluate: Proxy centos7-foreman-nightly.wisse.example.com cannot be retrieved: unknown error (response 500)

After looking for the error in /var/log/foreman/production.log, I couldn't find anything. A closer look showed me the following in /var/log/httpd/error_log:

[ 2020-02-05 10:39:03.8030 21530/7f0770143700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
  Error ID: 91ebc548
  Error details saved to: /tmp/passenger-error-vJ3j47.html
  Message from application: Unable to update template Kickstart default PXEGrub: This template is locked. Please clone it to a new template to customize. (RuntimeError)
  /usr/share/foreman/lib/seed_helper.rb:129:in `import_raw_template'
  /usr/share/foreman/lib/seed_helper.rb:138:in `block in import_templates'
  /usr/share/foreman/lib/seed_helper.rb:137:in `each'
  /usr/share/foreman/lib/seed_helper.rb:137:in `import_templates'
  /usr/share/foreman/db/seeds.d/070-provisioning_templates.rb:9:in `block in <top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
  /usr/share/foreman/db/seeds.d/070-provisioning_templates.rb:2:in `<top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `block in load'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
  /usr/share/foreman/app/services/foreman_seeder.rb:39:in `block (2 levels) in execute'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
  /usr/share/foreman/app/services/foreman_seeder.rb:38:in `block in execute'
  /usr/share/foreman/app/services/foreman_seeder.rb:32:in `each'
  /usr/share/foreman/app/services/foreman_seeder.rb:32:in `execute'
  /usr/share/foreman/config/initializers/seeds.rb:8:in `block (2 levels) in <top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:69:in `block in execute_hook'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application/finisher.rb:75:in `block in <module:Finisher>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `instance_exec'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `run'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `call'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:60:in `run_initializers'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application.rb:361:in `initialize!'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
  /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  config.ru:5:in `block in <main>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `instance_eval'
  /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'

[ 2020-02-05 10:39:03.8076 21530/7f076dbba700 agents/HelperAgent/RequestHandler.h:2306 ]: [Client 20] Cannot checkout session because a spawning error occurred. The identifier of the error is 91ebc548. Please see earlier logs for details about the error.

This is a bad user experience because before we rarely if ever got users to look in that file but I'm not sure where we should fix this (Foreman, Installer, Packaging) so I created the issue in Foreman for now.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Eric Helms about 4 years ago

Looking at the code, this is the result of a 'raise' with nothing catching it which is fine. However, I could this being more obvious if we add logger statements before the raise to ensure it appears in the production log?

Actions

Copy link

Updated by Ewoud Kohl van Wijngaarden about 4 years ago

Another thing you see is that both dynflow workers get in a restart loop because systemd doesn't figure out it'll never work. On Katello it'll be 3 processes meaning you get 300% CPU usage.

I've now worked around this by changing the seeding script to ignore it, but as a normal user I wouldn't know how to solve this.

Actions

Copy link

Updated by Ewoud Kohl van Wijngaarden about 4 years ago

Related to Feature #28519: Allow storing if db:seed is required within the DB. added

Actions

Copy link

Updated by Tomer Brisker about 4 years ago

Target version set to 2.0.0

Actions

Copy link

Updated by Tomer Brisker about 4 years ago

Category set to Rake tasks

Would it make sense to have a generic catch in the initializer to catch any seed errors that aren't properly handled? maybe show a ui notification with the message and telling users they need to manually rerun the seed?

Actions

Copy link