Bug #18879
closedupgrade fail with error : Upgrade Step: set_virt_who_on_pools (this may take a while) ...
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1430823
Description of problem:
Activation keys keeps loading in satellite6.2.7
So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8 but getting below error while upgrade:
==========================
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions
Upgrade step set_virt_who_on_pools failed. Check logs for more information. ====================================
Version-Release number of selected component (if applicable):
Red hat Satellite6.2.8
How reproducible:
Frequently
Steps to Reproduce:
1. Customer upgraded server from Satellite6.1.x to Satellite6.2.7, but after this activation keys page was kept loading endlessly. So, As per bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1387776 tried to upgrade the satellite server to Satellite6.2.8
2.customer has external virt-who server. Stopped the virt-who server.
3. But, when running upgrade step, it got stuck with error as below.
Actual results:
Upgrade Step: set_virt_who_on_pools (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
rake aborted!
NoMethodError: undefined method `redhat?' for nil:NilClass
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/pool.rb:126:in `import_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:44:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/glue/candlepin/candlepin_object.rb:42:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/3.3/import_subscriptions.rake:8:in `block (4 levels) in <top (required)>'
Tasks: TOP => katello:upgrades:3.3:import_subscriptions
(See full trace by running task with --trace)
Importing Subscriptions
Upgrade step set_virt_who_on_pools failed. Check logs for more information.
Expected results:
Upgrade should work without any error.
Additional info:
Upgrade shows warning as below before throwing the error as below:
Upgrade Step: migrate_foreman...
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
true
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
false
========================
Following things are checked on satellite server when upgrade fail.
1) > rpm -qa tfm-rubygem-katello
tfm-rubygem-katello-3.0.0.105-1.el7sat.noarch
rpm -qa satellite
satellite-6.2.8-4.0.el7sat.noarch
2) > curl -k https://$(hostname -f):9090/features
["discovery","dynflow","openscap","pulp","puppet","puppetca","ssh"]
3) > hammer ping
candlepin:
Status: ok
Server Response: Duration: 51ms
candlepin_auth:
Status: ok
Server Response: Duration: 38ms
pulp:
Status: ok
Server Response: Duration: 87ms
foreman_tasks:
Status: ok
Server Response: Duration: 697ms
4) > su - postgres -c "psql -d foreman -c \"select * from foreman_tasks_tasks where state != 'stopped'\""
/etc/profile: line 88: HISTFILE: readonly variable
id | type | label | started_at
| ended_at | state | result | external_id | parent_task_id | start_at | start_befo
re
--------------------------------------+---------------------------------+---------------------------------------------+---------------
------+----------+---------+---------+--------------------------------------+----------------+----------------------------+-----------
---
e2476ddb-6ba9-435c-b11f-2f3148935e20 | ForemanTasks::Task::DynflowTask | Actions::Katello::EventQueue::Monitor | 2017-03-08 16:
08:31 | | running | pending | aa2a255d-2102-4bfd-8e37-b26464ce6d0e | | 2017-03-08 16:08:31.521042 |
19573cbe-e57b-4503-bbd9-934e9b0a1230 | ForemanTasks::Task::DynflowTask | Actions::Candlepin::ListenOnCandlepinEvents | 2017-03-08 16:
08:31 | | running | pending | 4c1e8ea4-2cdb-46af-823c-8ce296430906 | | 2017-03-08 16:08:31.419712 |
(2 rows)
5) tried reinstalling "tfm-rubygem-katello" but no luck.