Project

General

Profile

Actions

Bug #29342

closed

katello:pulp3_content_switchover fails if multiple docker tags point to the same manifest

Added by Ian Ballou about 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Fixed in Releases:
Found in Releases:

Description

I tried running katello:pulp3_content_switchover on a Katello instance that had many content view versions with the same docker repos. This caused the following error: http://pastebin.test.redhat.com/844411

This is due to the fact that Pulp 3 is merging tags that point to the same manifest during migration.


Related issues 1 (1 open0 closed)

Blocks Katello - Tracker #26987: Pulp3 Main trackerNew

Actions
Actions #1

Updated by Ian Ballou about 4 years ago

Actions #2

Updated by Chris Roberts about 4 years ago

  • Target version set to Katello 3.15.0
  • Triaged changed from No to Yes
Actions #3

Updated by Justin Sherrill about 4 years ago

[vagrant@centos7-katello-nightly katello]$ time sudo foreman-rake katello:pulp3_content_switchover
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
: UPDATE "katello_docker_tags" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Actions #4

Updated by Ian Ballou about 4 years ago

This issue might go further than just docker tags. I hit an error with pulp3_content_switchover with the following traceback:

ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
: UPDATE "katello_files" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)
Actions #5

Updated by Ian Ballou about 4 years ago

It looks like this was caused by running pulp3_content_switchover previously and then syncing new content. Perhaps there should be a warning if trying to run content_switchover if there are any pulp_ids that match pulp3_migrated_hrefs?

Actions #6

Updated by The Foreman Bot about 4 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Justin Sherrill
  • Pull request https://github.com/Katello/katello/pull/8629 added
Actions #7

Updated by Ian Ballou almost 4 years ago

We've noticed that the following docker tables are missing foreign keys:

katello_repository_docker_tags
katello_repository_docker_meta_tags

This is likely exacerbating the docker issues we're experiencing.

Actions #8

Updated by James Jeffers almost 4 years ago

  • Target version changed from Katello 3.15.0 to Katello 3.15.1
Actions #9

Updated by James Jeffers almost 4 years ago

  • Target version changed from Katello 3.15.1 to Katello 3.15.0
Actions #10

Updated by The Foreman Bot almost 4 years ago

  • Fixed in Releases Katello 3.16.0 added
Actions #11

Updated by Justin Sherrill almost 4 years ago

  • Status changed from Ready For Testing to Closed
Actions #12

Updated by The Foreman Bot almost 4 years ago

  • Pull request https://github.com/Katello/katello/pull/8655 added
Actions

Also available in: Atom PDF