Bug #29342: katello:pulp3_content_switchover fails if multiple docker tags point to the same manifest - Katello - Foreman

Custom queries

All issues closed in the last 3 weeks
Assigned to me
Closed Issues without Release
Easy and Trivial Issues
Errata Management
Good first issues - Foreman core
Issues without Bugzilla
Katello 4.11.0 TODO
Katello 4.12.0 TODO
Katello 4.13.0 TODO
Mine
My issues closed in the last 3 weeks
open docker issues
open docker issues 2.1
Open Features
Open Katello issues by target version
Open Pull Requests
Stale
Team Brad Backlog
Trackers
Untriaged
Untriaged and opened in the past two weeks

Actions

Copy link

Bug #29342

closed

katello:pulp3_content_switchover fails if multiple docker tags point to the same manifest

Added by Ian Ballou over 4 years ago. Updated over 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Justin Sherrill

Category:

Target version:

Katello 3.15.0

Difficulty:

Triaged:

Yes

Bugzilla link:

Pull request:

https://github.com/Katello/katello/pull/8629, https://github.com/Katello/katello/pull/8655

Fixed in Releases:

Katello 3.16.0

Found in Releases:

Red Hat JIRA:

Description

I tried running katello:pulp3_content_switchover on a Katello instance that had many content view versions with the same docker repos. This caused the following error: http://pastebin.test.redhat.com/844411

This is due to the fact that Pulp 3 is merging tags that point to the same manifest during migration.

Related issues 1 (1 open — 0 closed)

Blocks Katello - Tracker #26987: Pulp3 Main tracker

New

Actions

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Ian Ballou over 4 years ago

Blocks Tracker #26987: Pulp3 Main tracker added

Actions

Copy link

Updated by Chris Roberts over 4 years ago

Target version set to Katello 3.15.0
Triaged changed from No to Yes

Actions

Copy link

Updated by Justin Sherrill over 4 years ago

[vagrant@centos7-katello-nightly katello]$ time sudo foreman-rake katello:pulp3_content_switchover
rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
: UPDATE "katello_docker_tags" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_katello_docker_tags_on_pulp_id"
DETAIL: Key (pulp_id)=(/pulp/api/v3/content/container/tags/28f3fe85-844e-432d-a624-cb97cd73e0ca/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Actions

Copy link

Updated by Ian Ballou over 4 years ago

This issue might go further than just docker tags. I hit an error with pulp3_content_switchover with the following traceback:

ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
: UPDATE "katello_files" SET pulp_id = migrated_pulp3_href WHERE NOT (pulp_id=migrated_pulp3_href)
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'

Caused by:
PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_katello_files_on_pulp_id" 
DETAIL:  Key (pulp_id)=(/pulp/api/v3/content/file/files/b4a2f6e8-bd8d-4394-a07b-8364b6d47774/) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `async_exec'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:603:in `block (2 levels) in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:579:in `block (2 levels) in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:578:in `block in log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:569:in `log'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_no_cache'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:588:in `execute_and_clear'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:94:in `exec_delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:157:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/query_cache.rb:21:in `update'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation.rb:337:in `update_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:19:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.pre.master/lib/katello/tasks/pulp3_content_switchover.rake:16:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_content_switchover
(See full trace by running task with --trace)

Actions

Copy link

Updated by Ian Ballou over 4 years ago

It looks like this was caused by running pulp3_content_switchover previously and then syncing new content. Perhaps there should be a warning if trying to run content_switchover if there are any pulp_ids that match pulp3_migrated_hrefs?

Actions

Copy link