Project

General

Profile

Actions

Bug #20245

closed

Upgrading with an orphaned product/repository results in 404 Error on redhat repos page

Added by Justin Sherrill almost 7 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Fixed in Releases:
Found in Releases:

Description

A user provided a database recently and it seemed like some products were missing in candlepin but they were still in katello. I suspect there may be some new behavior in candlepin wrt refreshing a manifest where products no longer appear.

Investigation needs to be done.

Actions #1

Updated by Justin Sherrill almost 7 years ago

  • translation missing: en.field_release set to 267
Actions #2

Updated by Brad Buckingham almost 7 years ago

  • Target version changed from 202 to 205
Actions #3

Updated by Justin Sherrill over 6 years ago

  • translation missing: en.field_release changed from 267 to 281
Actions #4

Updated by Klaas D over 6 years ago

I seem to be having similar issues, I think this could be related to products that are no longer supported by a valid license (ie have expired). This also seems to break product sync for products that have a valid license - that happened here after I uploaded a new manifest that does not include the expired licenses anymore.

Actions #5

Updated by Klaas D over 6 years ago

The products on Katello 3.3 (candlepin 0.9.54) still show in candlepin.cp_products whereas they are not showing in candlepin.cp2_products on Katello 3.4 (candlepin 2.0.35). This seems to be causing some confusion within katello.

Actions #6

Updated by Eric Helms over 6 years ago

  • translation missing: en.field_release changed from 281 to 286
Actions #7

Updated by Justin Sherrill over 6 years ago

  • translation missing: en.field_release changed from 286 to 281

After quite a bit of investigation and discussion with the candlepin team, we've determined that it is a failure of the migration from candlepin 0.9 to 2.0. The candlepin team is resoling this here: https://github.com/candlepin/candlepin/pull/1658

Sadly this means if you've already upgraded to katello 3.4, this will not help you. I'm working on a procedure to correct the problem now and will update the issue with that procedure once its complete.

Actions #8

Updated by Justin Sherrill over 6 years ago

Workaround:

foreman-rake console

and then paste in:

User.current = User.anonymous_admin

products = []

Organization.all.each do |org|
  org.owner_info #catch for candlepin being down

  org.products.redhat.each do |product|
    begin
      product.multiplier
    rescue RestClient::ResourceNotFound
      products << product
    end
  end
end

products.each do |product|
  product.repositories.where.not(:library_instance_id => nil).each do |repo|
    ForemanTasks.sync_task(::Actions::Pulp::Repository::Destroy, :pulp_id => repo.pulp_id) 
    repo.destroy!
  end

  repos = product.repositories.where(:library_instance_id => nil)
  repos.each do |repo|
    ForemanTasks.sync_task(::Actions::Pulp::Repository::Destroy, :pulp_id => repo.pulp_id) 
    repo.destroy!
  end
end

products.each(&:destroy!)

I think we will still need to ship some sort of script that runs after upgrade.

Actions #9

Updated by Klaas D over 6 years ago

Hi Justin,
the workaround does not seem to work for me

irb(main):001:0> User.current = User.anonymous_admin
=> #<User id: 1, login: "foreman_admin", firstname: "Anonymous", lastname: "Admin", mail: nil, admin: true, last_login_on: nil, auth_source_id: 2, created_at: "2016-06-22 16:00:30", updated_at: "2016-06-22 16:00:30", password_hash: nil, password_salt: nil, locale: nil, avatar_hash: nil, default_organization_id: nil, default_location_id: nil, lower_login: "foreman_admin", mail_enabled: true, timezone: nil, description: nil>
irb(main):002:0> 
irb(main):003:0* products = []
=> []
irb(main):004:0> 
irb(main):005:0* Organization.all.each do |org|
irb(main):006:1*   org.owner_info #catch for candlepin being down
irb(main):007:1> 
irb(main):008:1*   org.products.redhat.each do |product|
irb(main):009:2*     begin
irb(main):010:3*       product.multiplier
irb(main):011:3>     rescue RestClient::ResourceNotFound
irb(main):012:3>       products << product
irb(main):013:3>     end
irb(main):014:2>   end
irb(main):015:1> end
=> [#<Organization id: 1, name: "ORG", type: "Organization", created_at: "2016-06-22 16:00:30", updated_at: "2016-09-14 14:36:41", ignore_types: ["ProvisioningTemplate"], description: "", label: "ORG", ancestry: nil, title: "ORG">]
irb(main):016:0> 
irb(main):017:0* products.each do |product|
irb(main):018:1*   product.repositories.where.not(:library_instance_id => nil).each do |repo|
irb(main):019:2*     ForemanTasks.sync_task(::Actions::Katello::Repository::Destroy, repo, :planned_destroy => true) 
irb(main):020:2>   end
irb(main):021:1> 
irb(main):022:1*   repo = product.repositories.where(:library_instance_id => nil).first
irb(main):023:1>   if repo
irb(main):024:2>     ForemanTasks.sync_task(::Actions::Katello::Repository::Destroy, :pulp_id => repo.pulp_id) 
irb(main):025:2>     repo.destroy!
irb(main):026:2>   end
irb(main):027:1> end
ForemanTasks::TaskError: Task 4a930a93-0474-4eba-9654-0b556f095a44: Katello::Errors::CandlepinError: Unable to find content with the ID "4117".
        from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:40:in `block in sync_task'
        from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:39:in `tap'
        from /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.4/lib/foreman_tasks.rb:39:in `sync_task'
        from (irb):19:in `block (2 levels) in irb_binding'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
        from (irb):18:in `block in irb_binding'
        from (irb):17:in `each'
        from (irb):17
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/commands/console.rb:110:in `start'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/commands/console.rb:9:in `start'
        from /usr/share/foreman/lib/tasks/console.rake:8:in `block in <top (required)>'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
        from /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
        from /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'irb(main):028:0> 
irb(main):029:0* products.each(&:destroy!)
ActiveRecord::DeleteRestrictionError: Cannot delete record because of dependent repositories
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/has_many_association.rb:14:in `handle_dependency'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/associations/builder/association.rb:146:in `block in add_destroy_callbacks'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:448:in `instance_exec'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:448:in `block in make_lambda'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in `call'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:164:in `block in halting'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `call'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `block in call'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `each'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:504:in `call'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_destroy_callbacks'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/callbacks.rb:292:in `destroy'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:281:in `block in destroy'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
        from /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
... 13 levels...
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
        from /opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
        from /opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
        from /opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'irb(main):030:0>

Do you want to remove the products that no longer work? I would like to keep the ones where my license expired, they should become valid soon again when sales fixed the license issue :)

Greetings
Klaas

Actions #10

Updated by Justin Sherrill over 6 years ago

  • Bugzilla link set to 1476386
Actions #11

Updated by Justin Sherrill over 6 years ago

  • Subject changed from investigate missing products after manifest import/refresh to Upgrading with an orphaned product/repository results in 404 Error on redhat repos page
Actions #12

Updated by The Foreman Bot over 6 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/6892 added
Actions #13

Updated by Eric Helms over 6 years ago

  • translation missing: en.field_release changed from 281 to 286
Actions #14

Updated by The Foreman Bot over 6 years ago

  • Pull request https://github.com/Katello/katello-installer/pull/524 added
Actions #15

Updated by Brad Buckingham over 6 years ago

  • Target version changed from 205 to 208
Actions #16

Updated by Justin Sherrill over 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF