Bug #15121
closed[3.0.0 RC5] UPG failed due to Candlepin/Pulp Issues (Multiple Content Host Entries linked to Host-Collections inside DB)
Description
Dear all,
I think my UPG is failing due to the below which is related to the new "Host Unification" Feature.
Please let me know if the Infos are enough or if you need the logs as well in order to supply a fix...
"foreman-installer --scenario katello --upgrade"
Upgrade Step: Running installer...
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing Done [100%] [...............................]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
---
Run `rake db:migrate` to update your database then try again.
Upgrade step db_seed failed. Check logs for more information.
---
Multiple content hosts with the hostname centos-6-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Multiple content hosts with the hostname centos-7-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.
Content Host d3078602-3cb0-43b7-b326-c3534c8e0e13 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Exception when destroying candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"} (DELETE /candlepin/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13): {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"}
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json"
| \n# => 404 NotFound | application/json 528 bytes
|
Pulp consumer not found for consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 proceeding.
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Content Host ece0be43-0e4d-423a-967c-8d53fa4c99d6 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Exception when destroying candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"} (DELETE /candlepin/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6): {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"}
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
[katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json"
| \n# => 404 NotFound | application/json 528 bytes
|
Pulp consumer not found for consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 proceeding.
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.
"GUI => Monitor => Tasks"
Actions::Katello::Host::Destroy
{"host"=>{"id"=>251, "name"=>"centos-6-clone1.cesystems.at"},
"pool_ids"=>
["8a2180484c14ecee014c1503fe820033",
"8a2180484ca12e87014ca2277340001a",
"8a2180484c14ecee014c15052fd4003d",
"8a2180484ca12e87014ca21b43930002"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL: Key (id)=(157) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1
---
Actions::Katello::Host::Destroy
{"host"=>{"id"=>252, "name"=>"centos-7-clone1.cesystems.at"},
"pool_ids"=>
["8a2180484ca12e87014ca21bdf580009",
"8a2180484ca12e87014ca22908890023",
"8a2180484c14ecee014c1503fe820033",
"8a2180484c14ecee014c15052fd4003d"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL: Key (id)=(158) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1
"hammer ping"
candlepin:
Status: ok
Server Response: Duration: 30ms
candlepin_auth:
Status: ok
Server Response: Duration: 32ms
pulp:
Status: ok
Server Response: Duration: 35ms
foreman_tasks:
Status: ok
Server Response: Duration: 17ms
"foreman-rake db:migrate"
20150930183738 MigrateContentHosts: migrating ============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
and so on..
---
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=d3078602-3cb0-43b7-b326-c3534c8e0e13
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=ece0be43-0e4d-423a-967c-8d53fa4c99d6
Updated by Christian Ehart over 8 years ago
[root@katello ~]# cat /tmp/pre-upgrade-1463908532.csv
uuid, name, hostname, last_checkin, causes for deletion
d3078602-3cb0-43b7-b326-c3534c8e0e13,centos-6-manual.cesystems.at,centos-6-clone1.cesystems.at,2015-10-03T21:45:22.136+0000,More recently registered Content Host with same hostname centos-6-clone1.cesystems.at found.
ece0be43-0e4d-423a-967c-8d53fa4c99d6,centos-7-manual.cesystems.at,centos-7-clone1.cesystems.at,2015-10-03T22:11:28.702+0000,More recently registered Content Host with same hostname centos-7-clone1.cesystems.at found.
Updated by The Foreman Bot over 8 years ago
- Status changed from New to Ready For Testing
- Assignee set to Justin Sherrill
- Pull request https://github.com/Katello/katello/pull/6075 added
Updated by Justin Sherrill over 8 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset katello|3b80307c8f0e2e502e03f219e725480c7fd551af.