Bug #18706

Race condition among capsule sync tasks to destroy/create pulp repos

Added by Brad Buckingham about 1 month ago. Updated 15 days ago.

Status:Closed
Priority:High
Assigned To:Andrew Kofink
Category:Foreman Proxy Content
Target version:Team Brad - Iteration 12
Difficulty: Pull request:https://github.com/Katello/katello/pull/6675
Bugzilla link:1391298
Story points-
Velocity based estimate-
Release3.3.2Release relationshipAuto

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1391298

Description of problem:

Bulk capsule syncs failed when they were trying to remove the same pulp repos ('Actions::Katello::CapsuleContent::RemoveUnneededRepos').

Please see the below task information:

'Actions::Katello::Repository::CapsuleGenerateAndSync'
there are 7 Actions::Katello::CapsuleContent::Sync sub tasks in it matches the number of capsules in pnt-sysops org
each one of the capsulecontent::sync have one 'Actions::Katello::CapsuleContent::ConfigureCapsule'
and a lot of 'Actions::Pulp::Repository::Refresh'
under these 'Actions::Pulp::Repository::Refresh', some of them have skipped 'Actions::Pulp::Repository::RefreshDistributor'
under 'Actions::Katello::CapsuleContent::ConfigureCapsule', there is one 'Actions::Katello::CapsuleContent::RemoveUnneededRepos', where lots and lots of 'Actions::Pulp::Repository::Destroy' is in listed
'Actions::Pulp::Repository::Destroy' <-- lots of these are skipped or failed

The tasks paused after a whole bunch of repo::destroy failed. It can not be resumed until the user literally pressed the skip link next to every single errorred repo:destroy. The parent task can be resumed only when all of those errorred ones are skipped. After resuming, user got the Repo::create errors.

More information:
http://paste-platops.itos.redhat.com/pzpwqg41u/yyulhy#line-8
http://paste-platops.itos.redhat.com/p6cfi7rzn/sqiixo#line-15

Associated revisions

Revision 271f5808
Added by Andrew Kofink 16 days ago

Fixes #18706 - Handle 409 RestClient::Conflict

Handle conflict errors when trying to add repositories to a capsule that
already exist.

History

#1 Updated by Brad Buckingham about 1 month ago

  • Subject changed from Race condition among capsule sync tasks to destroy/create pulp repos to Race condition among capsule sync tasks to destroy/create pulp repos
  • Target version set to Team Brad - Iteration 11
  • Release set to Backlog

#2 Updated by Klaas D 28 days ago

is this the same as my errors in http://projects.theforeman.org/issues/18363 ?

#3 Updated by Andrew Kofink 21 days ago

  • Assigned To set to Andrew Kofink

#4 Updated by Brad Buckingham 17 days ago

  • Target version changed from Team Brad - Iteration 11 to Team Brad - Iteration 12

#5 Updated by The Foreman Bot 16 days ago

  • Status changed from New to Ready For Testing
  • Release deleted (Backlog)
  • Pull request https://github.com/Katello/katello/pull/6675 added

#6 Updated by Andrew Kofink 16 days ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#7 Updated by Justin Sherrill 15 days ago

  • Release set to 3.3.2

Also available in: Atom PDF