Project

General

Profile

Bug #28931

If seeding fails, Passenger refuses to start

Added by Ewoud Kohl van Wijngaarden 9 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
DB migrations
Target version:
Difficulty:
Triaged:
No
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

When running the installer, I got the following error

 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[centos7-foreman-nightly.wisse.example.com]: Could not evaluate: Proxy centos7-foreman-nightly.wisse.example.com cannot be retrieved: unknown error (response 500)

After looking for the error in /var/log/foreman/production.log, I couldn't find anything. A closer look showed me the following in /var/log/httpd/error_log:

[ 2020-02-05 10:39:03.8030 21530/7f0770143700 Pool2/Implementation.cpp:287 ]: Could not spawn process for application /usr/share/foreman: An error occured while starting up the preloader.
  Error ID: 91ebc548
  Error details saved to: /tmp/passenger-error-vJ3j47.html
  Message from application: Unable to update template Kickstart default PXEGrub: This template is locked. Please clone it to a new template to customize. (RuntimeError)
  /usr/share/foreman/lib/seed_helper.rb:129:in `import_raw_template'
  /usr/share/foreman/lib/seed_helper.rb:138:in `block in import_templates'
  /usr/share/foreman/lib/seed_helper.rb:137:in `each'
  /usr/share/foreman/lib/seed_helper.rb:137:in `import_templates'
  /usr/share/foreman/db/seeds.d/070-provisioning_templates.rb:9:in `block in <top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:376:in `without_auditing'
  /usr/share/foreman/db/seeds.d/070-provisioning_templates.rb:2:in `<top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `block in load'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:253:in `load_dependency'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies.rb:281:in `load'
  /usr/share/foreman/app/services/foreman_seeder.rb:39:in `block (2 levels) in execute'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
  /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
  /usr/share/foreman/app/services/foreman_seeder.rb:38:in `block in execute'
  /usr/share/foreman/app/services/foreman_seeder.rb:32:in `each'
  /usr/share/foreman/app/services/foreman_seeder.rb:32:in `execute'
  /usr/share/foreman/config/initializers/seeds.rb:8:in `block (2 levels) in <top (required)>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:69:in `block in execute_hook'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application/finisher.rb:75:in `block in <module:Finisher>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `instance_exec'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:32:in `run'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:61:in `block in run_initializers'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `call'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
  /opt/rh/rh-ruby25/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/initializable.rb:60:in `run_initializers'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/application.rb:361:in `initialize!'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
  /opt/theforeman/tfm/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
  /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  /opt/rh/rh-ruby25/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
  config.ru:5:in `block in <main>'
  /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `instance_eval'
  /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.0.6/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:112:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:158:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:28:in `<main>'

[ 2020-02-05 10:39:03.8076 21530/7f076dbba700 agents/HelperAgent/RequestHandler.h:2306 ]: [Client 20] Cannot checkout session because a spawning error occurred. The identifier of the error is 91ebc548. Please see earlier logs for details about the error.

This is a bad user experience because before we rarely if ever got users to look in that file but I'm not sure where we should fix this (Foreman, Installer, Packaging) so I created the issue in Foreman for now.


Related issues

Related to Foreman - Feature #28519: Allow storing if db:seed is required within the DB.Closed

Associated revisions

Revision 3c6b140f (diff)
Added by Tomer Brisker 8 months ago

Fixes #28838, #28931 - Safer seeding on initialization

Make sure that migartions have run before attempting to run seeds, to
ensure we don't get into a wierd state of the database if the app is
started before the db has been migrated.
Also, catch and log errors during seeding to prevent passanger from
failing to start if an error occurs.

History

#1 Updated by Eric Helms 9 months ago

Looking at the code, this is the result of a 'raise' with nothing catching it which is fine. However, I could this being more obvious if we add logger statements before the raise to ensure it appears in the production log?

#2 Updated by Ewoud Kohl van Wijngaarden 9 months ago

Another thing you see is that both dynflow workers get in a restart loop because systemd doesn't figure out it'll never work. On Katello it'll be 3 processes meaning you get 300% CPU usage.

I've now worked around this by changing the seeding script to ignore it, but as a normal user I wouldn't know how to solve this.

#3 Updated by Ewoud Kohl van Wijngaarden 9 months ago

  • Related to Feature #28519: Allow storing if db:seed is required within the DB. added

#4 Updated by Tomer Brisker 9 months ago

  • Target version set to 2.0.0

#5 Updated by Tomer Brisker 9 months ago

  • Category set to Rake tasks

Would it make sense to have a generic catch in the initializer to catch any seed errors that aren't properly handled? maybe show a ui notification with the message and telling users they need to manually rerun the seed?

#6 Updated by The Foreman Bot 8 months ago

  • Assignee set to Tomer Brisker
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/7459 added

#7 Updated by The Foreman Bot 8 months ago

  • Fixed in Releases 2.1.0 added

#8 Updated by Tomer Brisker 8 months ago

  • Status changed from Ready For Testing to Closed

#9 Updated by Tomer Brisker 8 months ago

  • Fixed in Releases 2.0.0 added
  • Fixed in Releases deleted (2.1.0)

#10 Updated by Tomer Brisker 8 months ago

  • Category changed from Rake tasks to DB migrations

Also available in: Atom PDF