Bug #20245
closedUpgrading with an orphaned product/repository results in 404 Error on redhat repos page
Description
A user provided a database recently and it seemed like some products were missing in candlepin but they were still in katello. I suspect there may be some new behavior in candlepin wrt refreshing a manifest where products no longer appear.
Investigation needs to be done.
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release set to 267
Updated by Brad Buckingham over 7 years ago
- Target version changed from 202 to 205
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release changed from 267 to 281
Updated by Klaas D over 7 years ago
I seem to be having similar issues, I think this could be related to products that are no longer supported by a valid license (ie have expired). This also seems to break product sync for products that have a valid license - that happened here after I uploaded a new manifest that does not include the expired licenses anymore.
Updated by Klaas D over 7 years ago
The products on Katello 3.3 (candlepin 0.9.54) still show in candlepin.cp_products whereas they are not showing in candlepin.cp2_products on Katello 3.4 (candlepin 2.0.35). This seems to be causing some confusion within katello.
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 281 to 286
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release changed from 286 to 281
After quite a bit of investigation and discussion with the candlepin team, we've determined that it is a failure of the migration from candlepin 0.9 to 2.0. The candlepin team is resoling this here: https://github.com/candlepin/candlepin/pull/1658
Sadly this means if you've already upgraded to katello 3.4, this will not help you. I'm working on a procedure to correct the problem now and will update the issue with that procedure once its complete.
Updated by Justin Sherrill over 7 years ago
Workaround:
foreman-rake console
and then paste in:
User.current = User.anonymous_admin products = [] Organization.all.each do |org| org.owner_info #catch for candlepin being down org.products.redhat.each do |product| begin product.multiplier rescue RestClient::ResourceNotFound products << product end end end products.each do |product| product.repositories.where.not(:library_instance_id => nil).each do |repo| ForemanTasks.sync_task(::Actions::Pulp::Repository::Destroy, :pulp_id => repo.pulp_id) repo.destroy! end repos = product.repositories.where(:library_instance_id => nil) repos.each do |repo| ForemanTasks.sync_task(::Actions::Pulp::Repository::Destroy, :pulp_id => repo.pulp_id) repo.destroy! end end products.each(&:destroy!)
I think we will still need to ship some sort of script that runs after upgrade.
Updated by Klaas D over 7 years ago
Hi Justin,
the workaround does not seem to work for me
irb(main):001:0> User.current = User.anonymous_admin => #<User id: 1, login: "foreman_admin", firstname: "Anonymous", lastname: "Admin", mail: nil, admin: true, last_login_on: nil, auth_source_id: 2, created_at: "2016-06-22 16:00:30", updated_at: "2016-06-22 16:00:30", password_hash: nil, password_salt: nil, locale: nil, avatar_hash: nil, default_organization_id: nil, default_location_id: nil, lower_login: "foreman_admin", mail_enabled: true, timezone: nil, description: nil> irb(main):002:0> irb(main):003:0* products = [] => [] irb(main):004:0> irb(main):005:0* Organization.all.each do |org| irb(main):006:1* org.owner_info #catch for candlepin being down irb(main):007:1> irb(main):008:1* org.products.redhat.each do |product| irb(main):009:2* begin irb(main):010:3* product.multiplier irb(main):011:3> rescue RestClient::ResourceNotFound irb(main):012:3> products << product irb(main):013:3> end irb(main):014:2> end irb(main):015:1> end => [#<Organization id: 1, name: "ORG", type: "Organization", created_at: "2016-06-22 16:00:30", updated_at: "2016-09-14 14:36:41", ignore_types: ["ProvisioningTemplate"], description: "", label: "ORG", ancestry: nil, title: "ORG">] irb(main):016:0> irb(main):017:0* products.each do |product| irb(main):018:1* product.repositories.where.not(:library_instance_id => nil).each do |repo| irb(main):019:2* ForemanTasks.sync_task(::Actions::Katello::Repository::Destroy, repo, :planned_destroy => true) irb(main):020:2> end irb(main):021:1> irb(main):022:1* repo = product.repositories.where(:library_instance_id => nil).first irb(main):023:1> if repo irb(main):024:2> ForemanTasks.sync_task(::Actions::Katello::Repository::Destroy, :pulp_id => repo.pulp_id) irb(main):025:2> repo.destroy! irb(main):026:2> end irb(main):027:1> end ForemanTasks::TaskError: Task 4a930a93-0474-4eba-9654-0b556f095a44: Katello::Errors::CandlepinError: Unable to find content with the ID "4117". from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:40:in `block in sync_task' from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:39:in `tap' from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:39:in `sync_task' from (irb):19:in `block (2 levels) in irb_binding' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each' from (irb):18:in `block in irb_binding' from (irb):17:in `each' from (irb):17 from /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/commands/console.rb:110:in `start' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/commands/console.rb:9:in `start' from /usr/share/foreman/lib/tasks/console.rake:8:in `block in <top (required)>' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain' from /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run' from /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'irb(main):028:0> irb(main):029:0* products.each(&:destroy!) ActiveRecord::DeleteRestrictionError: Cannot delete record because of dependent repositories from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/has_many_association.rb:14:in `handle_dependency' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/builder/association.rb:146:in `block in add_destroy_callbacks' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:448:in `instance_exec' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:448:in `block in make_lambda' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in `call' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in `block in halting' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `call' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `block in call' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `each' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `call' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_destroy_callbacks' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/callbacks.rb:292:in `destroy' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:281:in `block in destroy' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status' from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction' ... 13 levels... from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain' from /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling' from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run' from /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'irb(main):030:0>
Do you want to remove the products that no longer work? I would like to keep the ones where my license expired, they should become valid soon again when sales fixed the license issue :)
Greetings
Klaas
Updated by Justin Sherrill over 7 years ago
- Subject changed from investigate missing products after manifest import/refresh to Upgrading with an orphaned product/repository results in 404 Error on redhat repos page
Updated by The Foreman Bot over 7 years ago
- Status changed from New to Ready For Testing
- Pull request https://github.com/Katello/katello/pull/6892 added
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 281 to 286
Updated by The Foreman Bot over 7 years ago
- Pull request https://github.com/Katello/katello-installer/pull/524 added
Updated by Brad Buckingham over 7 years ago
- Target version changed from 205 to 208
Updated by Justin Sherrill over 7 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset katello-installer|a3d3a9c4d99bf50b6bbdf2b0f68ecda57acda0e7.