Project

General

Profile

Actions

Bug #15121

closed

[3.0.0 RC5] UPG failed due to Candlepin/Pulp Issues (Multiple Content Host Entries linked to Host-Collections inside DB)

Added by Christian Ehart almost 8 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

Dear all,

I think my UPG is failing due to the below which is related to the new "Host Unification" Feature.
Please let me know if the Infos are enough or if you need the logs as well in order to supply a fix...

"foreman-installer --scenario katello --upgrade"

Upgrade Step: Running installer...
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing Done [100%] [...............................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
---
Run `rake db:migrate` to update your database then try again.
Upgrade step db_seed failed. Check logs for more information.
---
Multiple content hosts with the hostname centos-6-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Multiple content hosts with the hostname centos-7-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.
Content Host d3078602-3cb0-43b7-b326-c3534c8e0e13 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Exception when destroying candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"} (DELETE /candlepin/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13): {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"}
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 404 NotFound | application/json 528 bytes |
Pulp consumer not found for consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 proceeding.
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Content Host ece0be43-0e4d-423a-967c-8d53fa4c99d6 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Exception when destroying candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"} (DELETE /candlepin/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6): {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"}
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
[katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 404 NotFound | application/json 528 bytes |
Pulp consumer not found for consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 proceeding.
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.

"GUI => Monitor => Tasks"
Actions::Katello::Host::Destroy {"host"=>{"id"=>251, "name"=>"centos-6-clone1.cesystems.at"},
"pool_ids"=>
["8a2180484c14ecee014c1503fe820033",
"8a2180484ca12e87014ca2277340001a",
"8a2180484c14ecee014c15052fd4003d",
"8a2180484ca12e87014ca21b43930002"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL: Key (id)=(157) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1
---
Actions::Katello::Host::Destroy {"host"=>{"id"=>252, "name"=>"centos-7-clone1.cesystems.at"},
"pool_ids"=>
["8a2180484ca12e87014ca21bdf580009",
"8a2180484ca12e87014ca22908890023",
"8a2180484c14ecee014c1503fe820033",
"8a2180484c14ecee014c15052fd4003d"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL: Key (id)=(158) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1

"hammer ping"
candlepin:
Status: ok
Server Response: Duration: 30ms
candlepin_auth:
Status: ok
Server Response: Duration: 32ms
pulp:
Status: ok
Server Response: Duration: 35ms
foreman_tasks:
Status: ok
Server Response: Duration: 17ms

"foreman-rake db:migrate"
20150930183738 MigrateContentHosts: migrating ============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
and so on..
---
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=d3078602-3cb0-43b7-b326-c3534c8e0e13
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=ece0be43-0e4d-423a-967c-8d53fa4c99d6

Actions #1

Updated by Christian Ehart almost 8 years ago

[root@katello ~]# cat /tmp/pre-upgrade-1463908532.csv
uuid, name, hostname, last_checkin, causes for deletion
d3078602-3cb0-43b7-b326-c3534c8e0e13,centos-6-manual.cesystems.at,centos-6-clone1.cesystems.at,2015-10-03T21:45:22.136+0000,More recently registered Content Host with same hostname centos-6-clone1.cesystems.at found.
ece0be43-0e4d-423a-967c-8d53fa4c99d6,centos-7-manual.cesystems.at,centos-7-clone1.cesystems.at,2015-10-03T22:11:28.702+0000,More recently registered Content Host with same hostname centos-7-clone1.cesystems.at found.

Actions #2

Updated by Justin Sherrill almost 8 years ago

  • Bugzilla link set to 1340122
Actions #3

Updated by The Foreman Bot almost 8 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Justin Sherrill
  • Pull request https://github.com/Katello/katello/pull/6075 added
Actions #4

Updated by Justin Sherrill almost 8 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF