Project

General

Profile

Bug #29342

katello:pulp3_content_switchover fails if multiple docker tags point to the same manifest

Added by Ian Ballou 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:

Description

I tried running katello:pulp3_content_switchover on a Katello instance that had many content view versions with the same docker repos. This caused the following error: http://pastebin.test.redhat.com/844411

This is due to the fact that Pulp 3 is merging tags that point to the same manifest during migration.


Related issues

Blocks Katello - Tracker #26987: Pulp3 Main trackerNew

Associated revisions

Revision 89bee135 (diff)
Added by Justin Sherrill about 2 months ago

Fixes #29342 - handle one-many docker tag conversion on switch

Pulp2 duplicates docker tag units across different repos, so that
a docker repo with a tag 'latest' that is copied into multiple cvs
will have multiple units in pulp and in the docker_tags table.

In pulp3, pulp 'unifies' them into one unit, so in this case upon
switchover we 'migrate' these over to a single unit

Revision 4ba11295 (diff)
Added by Justin Sherrill about 1 month ago

Refs #29342 - reset settings in teardown

to avoid test failures

History

#1 Updated by Ian Ballou 2 months ago

#2 Updated by Chris Roberts 2 months ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.15.0

#3 Updated by Justin Sherrill 2 months ago

[vagrant@centos7-katello-nightly katello]$ time sudo foreman-rake katello:pulp3_content_switchover
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
: UPDATE "katello_docker_tags" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

#4 Updated by Ian Ballou 2 months ago

This issue might go further than just docker tags. I hit an error with pulp3_content_switchover with the following traceback:

ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
: UPDATE "katello_files" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

#5 Updated by Ian Ballou 2 months ago

It looks like this was caused by running pulp3_content_switchover previously and then syncing new content. Perhaps there should be a warning if trying to run content_switchover if there are any pulp_ids that match pulp3_migrated_hrefs?

#6 Updated by The Foreman Bot 2 months ago

  • Assignee set to Justin Sherrill
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/8629 added

#7 Updated by Ian Ballou about 2 months ago

We've noticed that the following docker tables are missing foreign keys:

katello_repository_docker_tags
katello_repository_docker_meta_tags

This is likely exacerbating the docker issues we're experiencing.

#8 Updated by James Jeffers about 2 months ago

  • Target version changed from Katello 3.15.0 to Katello 3.15.1

#9 Updated by James Jeffers about 2 months ago

  • Target version changed from Katello 3.15.1 to Katello 3.15.0

#10 Updated by The Foreman Bot about 2 months ago

  • Fixed in Releases Katello 3.16.0 added

#11 Updated by Justin Sherrill about 2 months ago

  • Status changed from Ready For Testing to Closed

#12 Updated by The Foreman Bot about 1 month ago

  • Pull request https://github.com/Katello/katello/pull/8655 added

Also available in: Atom PDF