Bug #29342
closedkatello:pulp3_content_switchover fails if multiple docker tags point to the same manifest
Description
I tried running katello:pulp3_content_switchover on a Katello instance that had many content view versions with the same docker repos. This caused the following error: http://pastebin.test.redhat.com/844411
This is due to the fact that Pulp 3 is merging tags that point to the same manifest during migration.
Updated by Ian Ballou about 4 years ago
- Blocks Tracker #26987: Pulp3 Main tracker added
Updated by Chris Roberts about 4 years ago
- Target version set to Katello 3.15.0
- Triaged changed from No to Yes
Updated by Justin Sherrill about 4 years ago
[vagrant@centos7-katello-nightly katello]$ time sudo foreman-rake katello:pulp3_content_switchover
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
: UPDATE "katello_docker_tags" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Caused by:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)
Updated by Ian Ballou about 4 years ago
This issue might go further than just docker tags. I hit an error with pulp3_content_switchover with the following traceback:
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_files_on_pulp_id" DETAIL: Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists. : UPDATE "katello_files" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href) /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>' Caused by: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_files_on_pulp_id" DETAIL: Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists. /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>' Tasks: TOP => katello:pulp3_content_switchover (See full trace by running task with --trace)
Updated by Ian Ballou about 4 years ago
It looks like this was caused by running pulp3_content_switchover previously and then syncing new content. Perhaps there should be a warning if trying to run content_switchover if there are any pulp_ids that match pulp3_migrated_hrefs?
Updated by The Foreman Bot about 4 years ago
- Status changed from New to Ready For Testing
- Assignee set to Justin Sherrill
- Pull request https://github.com/Katello/katello/pull/8629 added
Updated by Ian Ballou almost 4 years ago
We've noticed that the following docker tables are missing foreign keys:
katello_repository_docker_tags
katello_repository_docker_meta_tags
This is likely exacerbating the docker issues we're experiencing.
Updated by James Jeffers almost 4 years ago
- Target version changed from Katello 3.15.0 to Katello 3.15.1
Updated by James Jeffers almost 4 years ago
- Target version changed from Katello 3.15.1 to Katello 3.15.0
Updated by The Foreman Bot almost 4 years ago
- Fixed in Releases Katello 3.16.0 added
Updated by Justin Sherrill almost 4 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset katello|89bee13590f26703d01e05c05c1b527ce31d8d23.
Updated by The Foreman Bot almost 4 years ago
- Pull request https://github.com/Katello/katello/pull/8655 added