Project

General

Profile

Bug #23529

Multiple Issues Upgrading from 1.16 to 1.17

Added by Jeff Sparrow over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
Difficulty:
hard
Triaged:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

I am trying to upgrade from 1.16 to 1.17 following the standard upgrade procedure as listed in the documentation.

I am running in to multiple issues trying to run the foreman-rake db:migrate portion of the upgrade and am not sure where I should start this ticket. The first issue I hit is related to plugins:
I should mention that when doing the upgrade I did run yum upgrade tfm\* ruby\* foreman\* --exclude=rubygem-rkerberos-0.1.3-5.el7.x86_64 due to - https://projects.theforeman.org/issues/23241 - we dont use kerberos so I figured i'd be ok. maybe?

[*PROD* root@puppet 1 /tmp]# foreman-rake db:migrate --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
rake aborted!
Bundler::Dsl::DSLError:
[!] There was an error parsing `Gemfile.in`: cannot load such file -- foreman_host_overview/engine. Bundler cannot continue.

 #  from /usr/share/foreman/Gemfile.in:59
 #  -------------------------------------------
 #  Dir["#{File.dirname(FOREMAN_GEMFILE)}/bundler.d/*.rb"].each do |bundle|
 >    self.instance_eval(Bundler.read_file(bundle))
 #  end
 #  -------------------------------------------
/usr/share/foreman/Gemfile.in:59:in `instance_eval'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
(eval):1:in `block in eval_gemfile'
/usr/share/foreman/Gemfile.in:59:in `instance_eval'
/usr/share/foreman/Gemfile.in:59:in `block in eval_gemfile'
/usr/share/foreman/Gemfile.in:58:in `each'
/usr/share/foreman/Gemfile.in:58:in `eval_gemfile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:41:in `instance_eval'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:41:in `eval_gemfile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:11:in `evaluate'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/gemfile.rb:46:in `parse'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
LoadError: cannot load such file -- foreman_host_overview/engine
/usr/share/foreman/Gemfile.in:59:in `instance_eval'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
(eval):1:in `block in eval_gemfile'
/usr/share/foreman/Gemfile.in:59:in `instance_eval'
/usr/share/foreman/Gemfile.in:59:in `block in eval_gemfile'
/usr/share/foreman/Gemfile.in:58:in `each'
/usr/share/foreman/Gemfile.in:58:in `eval_gemfile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:41:in `instance_eval'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:41:in `eval_gemfile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/bundler-1.13.7/lib/bundler/dsl.rb:11:in `evaluate'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/gemfile.rb:46:in `parse'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
You have mail in /var/spool/mail/root

This issue is obviously related to a plugin foreman_host_overview/engine and a file in /usr/share/foreman/bundler.d. I removed that file and am able to get past the error. Then I hit another error:

https://gist.github.com/tek0011/29cb9ba40a21dcb771b004cdc75d21d0

[*PROD* root@puppet 0 /usr/share/foreman/bundler.d]# foreman-rake db:migrate --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
rake aborted!
Gem loading error: Could not find 'prawn' (~> 2.1) among 152 total gem(s)
Checked in 'GEM_PATH=/opt/theforeman/tfm/root/usr/share/gems:/opt/theforeman/tfm-ror51/root/usr/share/gems:/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby24/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems/opt/theforeman/tfm-ror51/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems', execute `gem env` for more information
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/output.rb:13:in `strict_err'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:46:in `rescue in block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:39:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
Gem::MissingSpecError: Could not find 'prawn' (~> 2.1) among 152 total gem(s)
Checked in 'GEM_PATH=/opt/theforeman/tfm/root/usr/share/gems:/opt/theforeman/tfm-ror51/root/usr/share/gems:/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby24/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems/opt/theforeman/tfm-ror51/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems', execute `gem env` for more information
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/dependency.rb:308:in `to_specs'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1439:in `block in activate_dependencies'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1428:in `each'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1428:in `activate_dependencies'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1410:in `activate'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems.rb:220:in `rescue in try_activate'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems.rb:213:in `try_activate'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:126:in `rescue in require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:40:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
Gem::MissingSpecError: Could not find 'prawn' (~> 2.1) among 152 total gem(s)
Checked in 'GEM_PATH=/opt/theforeman/tfm/root/usr/share/gems:/opt/theforeman/tfm-ror51/root/usr/share/gems:/usr/share/foreman/.gem/ruby:/opt/rh/rh-ruby24/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems/opt/theforeman/tfm-ror51/root/usr/share/gems:/opt/rh/rh-ruby24/root/usr/local/share/gems', execute `gem env` for more information
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/dependency.rb:308:in `to_specs'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1439:in `block in activate_dependencies'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1428:in `each'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1428:in `activate_dependencies'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/specification.rb:1410:in `activate'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems.rb:214:in `try_activate'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:126:in `rescue in require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:40:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
LoadError: cannot load such file -- foreman_datacenter
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:41:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext/runtime.rb:37:in `system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:19:in `block in system_require'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/bundler_ext-0.4.1/lib/bundler_ext.rb:14:in `system_require'
/usr/share/foreman/config/application.rb:17:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/share/foreman/Rakefile:1:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'

At this point I tried to remove all plugins, and then I hit the following error:
https://gist.github.com/tek0011/7cfb949f168a5f531ecbb74ddac00b62

[*PROD* root@puppet 0 /usr/share/foreman/bundler.d]# foreman-rake db:migrate --trace
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
/usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/appender.rb:139: warning: constant ::Fixnum is deprecated
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/logger.rb:295: warning: constant ::Fixnum is deprecated
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
fatal: No live threads left. Deadlock?
5 threads, 5 sleeps current:0x0000000c5d77e0 main thread:0x00000001be05d0
* #<Thread:0x00000001c13878 sleep_forever>
   rb_thread_t:0x00000001be05d0 native:0x007f2c24753740 int:1
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `sleep'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `wait'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `block in wait_until_complete'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:400:in `synchronize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:400:in `wait_until_complete'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:242:in `wait'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/world.rb:440:in `spawn_and_wait'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/world.rb:15:in `initialize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails/configuration.rb:66:in `new'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails/configuration.rb:66:in `initialize_world'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails.rb:101:in `init_world'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails.rb:37:in `initialize!'
   /usr/share/foreman/config/application.rb:281:in `block in <class:Application>'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `call'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
   /opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
   /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
   /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
   /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
   /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:329:in `require_environment!'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:445:in `block in run_tasks_blocks'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
   /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `each'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in `block in invoke_with_call_chain'
   /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
   /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
   /opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
   /opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
* #<Thread:0x0000000a1e7300@/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:320 sleep_forever>
   rb_thread_t:0x0000000c5d77e0 native:0x007f2c16c12700 int:0
   /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:187:in `lock'
   /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:187:in `mon_enter'
   /opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:40:in `require'
   /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
   /opt/rh/rh-ruby24/root/usr/share/ruby/set.rb:632:in `setup'
   /opt/rh/rh-ruby24/root/usr/share/ruby/set.rb:719:in `initialize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/clock.rb:52:in `new'
   /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/clock.rb:52:in `initialize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:156:in `build_context'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:199:in `block in ns_initialize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
   /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
* #<Thread:0x0000000a1eb180@/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:320 sleep_forever>
   rb_thread_t:0x007f2c10002b60 native:0x007f2c16b11700 int:0
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `pop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
   /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
* #<Thread:0x0000000a1ee498@/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:320 sleep_forever>
   rb_thread_t:0x007f2c080022b0 native:0x007f2c16119700 int:0
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `pop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
   /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
* #<Thread:0x0000000c921470@/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:320 sleep_forever>
   rb_thread_t:0x0000000b959370 native:0x007f2c0f93a700 int:0
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `pop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
   /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
   /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `sleep'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `wait'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:404:in `block in wait_until_complete'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:400:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:400:in `wait_until_complete'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/edge/future.rb:242:in `wait'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/world.rb:440:in `spawn_and_wait'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/world.rb:15:in `initialize'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails/configuration.rb:66:in `new'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails/configuration.rb:66:in `initialize_world'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails.rb:101:in `init_world'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/rails.rb:37:in `initialize!'
/usr/share/foreman/config/application.rb:281:in `block in <class:Application>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:67:in `block in execute_hook'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60:in `with_execution_control'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65:in `execute_hook'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:50:in `block in run_load_hooks'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:49:in `run_load_hooks'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application/finisher.rb:73:in `block in <module:Finisher>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `instance_exec'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:30:in `run'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:59:in `block in run_initializers'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `call'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
/opt/rh/rh-ruby24/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/initializable.rb:58:in `run_initializers'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:353:in `initialize!'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/rh/rh-ruby24/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'
/opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:329:in `require_environment!'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/application.rb:445:in `block in run_tasks_blocks'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:22:in `<main>'
Tasks: TOP => db:migrate => environment

I work fairly early in the AM, so I can take a maintenance window every morning to try to resolve this. I've tried removing all the plugins I know of (yum packaged ones and manually installed ones). I've tried removing them from all the gem locations where they show up, that I know of:

- /opt/rh/rh-ruby22/root/usr/local/share/gems/gems/
- /opt/theforeman/tfm/root/usr/share/gems/gems/
- /usr/local/share/gems/gems/

I find it a bit odd there are so many gem directories..

Anyhow, thats all I've got. Not really sure where to go from here. I've tried multiple ways to get this upgrade to work and have had no luck.

History

#1 Updated by Ewoud Kohl van Wijngaarden over 2 years ago

I suspect you installed the foreman datacenter plugin manually. That depends on prawn. In 1.17 we moved from ruby 2.2 to ruby 2.4 so you need to reinstall any manually installed dependencies.

#2 Updated by Jeff Sparrow over 2 years ago

Thanks. Does that still pertain to the last error? The deadlock that happens after the plugins were removed?

#3 Updated by Ewoud Kohl van Wijngaarden over 2 years ago

No, that's new to me. I wonder if they all try to open the log file. I'd verify there's no process with that file open before starting, otherwise I'm out of ideas.

#4 Updated by Jeff Sparrow over 2 years ago

I was able to get past the Deadlock error by removing all gems in /opt/rh/rh-ruby22/root/usr/local/share/gems/gems
Im not really familiar with the entire ruby/gems/foreman environment, so not sure how or why this helped. Maybe someone else has an idea?

#5 Updated by Jeff Sparrow over 2 years ago

Well nevermind. I tried to re-run the db migrate and hit the first error listed in this ticket. Went to try and remove the info in usr/share/foreman/bundler.d (which keeps coming back for some reason) and now I hit the deadlock issues again. ugh.

#6 Updated by Jeff Sparrow over 2 years ago

Narrowed it down. I hit the deadlock error as soon as I rename/remove /usr/share/foreman/bundler.d/Gemfile.local.rb which contains:

[*PROD* root@puppet 1 /usr/share/foreman/bundler.d]# cat Gemfile.local.rb
gem 'foreman_datacenter', :git => "https://github.com/theforeman/foreman_datacenter.git" 
gem 'foreman_dhcp_browser', :git => "https://github.com/theforeman/foreman_dhcp_browser.git" 
gem 'uglifier', '2.4.0'
gem 'foreman_host_overview', :git => "https://github.com/sfu-rcg/foreman_host_overview.git" 

Not really sure why, just trying to document this issue.

#7 Updated by Jeff Sparrow over 2 years ago

Narrowed it down more. It seems like its all related to foreman_host_overview - If I comment out the other information in this file, I do not get the deadlock error. If I comment out

gem 'foreman_host_overview', :git => "https://github.com/sfu-rcg/foreman_host_overview.git" 

Then I hit the deadlock error. If I uncomment it I get the standard missing gem error.

#8 Updated by Jeff Sparrow over 2 years ago

kerberos issue fix is to use cr: yum-config-manager --enable cr

#9 Updated by Jeff Sparrow over 2 years ago

Ewoud Kohl van Wijngaarden wrote:

No, that's new to me. I wonder if they all try to open the log file. I'd verify there's no process with that file open before starting, otherwise I'm out of ideas.

Hello Ewoud. When you say "verify theres no process with "that" file open", what file are you referring to? The --trace debug output doesnt really make sense to me and Im having a hard time trying to figure out what file/process to specifically look at. I can now fix all the other issues related to this ticket, but in the end it all comes back to the deadlock issue. Thanks.

#10 Updated by Jeff Sparrow over 2 years ago

Here is an strace of the command if this helps:

https://gist.github.com/tek0011/ea691e3692d651c0e60a8211bf6b3b83

#11 Updated by Jeff Sparrow over 2 years ago

Another failed attempt. No matter what I seem to do I get the deadlock issue during rake/ db migration. I watched mysql at the time of the rake and can see that foreman user is logging in to it and attempting to do some things. Then I see that process go into a sleep, until I get the deadlock. I spoke with Daniel Lobato in IRC about paying for support and he linked me to Satellite, so now I just need to figure out how or where to get support from there. I am sure Dell Technologies (the vendor for this issue) has a subscription.

Hopefully someone can direct me to the right channels and help me expedite getting support for this issue, as it remains a hot topic and high priority with Dell and their use of Foreman.

#12 Updated by Jeff Sparrow over 2 years ago

WOOOOOOOO HOOOOOO after 10 days, 3 hours of a maintenance window each morning, I finally got 1.16 to 1.17 upgrade to work!
I worked backwards and decided to remove every bit of foreman_datacenter and foreman_host_overview that I could find, prior to doing any upgrade work.

Removed every gem of them, in all the gem environments, in each ruby environment including foremans ruby env (scl enable tfm bash)
Removed all its gem dependencies that no other part of foreman relied on.
Cleaned up all the directories and contents that I could find with find / -f foreman_data*
Removed duplicates of gems in every gem env (gem cleanup)
Went in mysql and deleted every possible table I could find that related to it.
Deleted all stuck tasks from foreman_task_task table
Stopped all services possible
Restarted mysql/mariadb (needed to be on for foreman-rake db:migrate)

I am not sure exactly what did it. All I know is that every other time I tried I would get the deadlock issues. I started watching mysql logs, and could see that each time the migrate was attempting to do some work, then would just go into a sleep state. So I focused my efforts on mariadb/mysql. I am assuming some aspect of foreman_datacenter or possible foreman_tasks_tasks was broke.

All is working now. Excited to see 1.17 in production.

#13 Updated by Jeff Sparrow over 2 years ago

  • Difficulty set to hard
  • Legacy Backlogs Release (now unused) set to 296
  • Status changed from New to Resolved

#14 Updated by Ewoud Kohl van Wijngaarden over 2 years ago

Good to hear that you got it resolved. I intend to package it. Currently it doesn't work on nightly so https://github.com/theforeman/foreman-packaging/pull/2336 is somewhat blocked on it.

Also available in: Atom PDF