Project

General

Profile

Bug #25626

race condition when creating multiple repos in same product

Added by Chris Duryee 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

If you attempt to create multiple repos on the same product, it is possible that multiple `Candlepin::Environment::SetContent` actions will run at the same time. This may result in candlepin being updated with conflicting info, and one request will "undo" or delete another's update.

Repro steps (many thanks to pmoravec):

#!/bin/bash

hmr="hammer -u admin -p changeme" 

org="TestOrg" # create this org ahead of time
product="ZOO-1" # increment this each time you want to test the bug
$hmr product create --name=$product --label=$product --organization=$org

from=${1:-1}
to=${2:-10}
for i in $(seq $from $to); do
        $hmr repository create --product=$product --organization=$org --name=zoo_${i} --content-type=yum --url=https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/ &
done
wait

for i in $(seq $from $to); do
        $hmr repository synchronize --organization=$org --product=$product --name=zoo_${i} --async &
done

while [ $($hmr task list --search "state = running" | grep -ci sync) -gt 0 ]; do
        sleep 5
done

After this script runs, attach a subscription to the "ZOO-1" product to a host, and check if it has all of the repos. If you hit this bug, you'll only be subscribed a few (possibly even just one) of the repos created. You'll also see DELETE statements in candlepin.log like this:

INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=DELETE, uri=/candlepin/environments/ac912e17404a24abc9169113c0051b8/content?content=1543947592886

This bugfix is to keep Katello from inadvertently deleting content off of a product during concurrent repo creates. There's a candlepin bug that needs to be handled as well, but that is outside the scope of this issue.

Associated revisions

Revision e737e2e3 (diff)
Added by Chris Duryee 7 months ago

Fixes #25626 - race condition when creating repos

If you attempt to create multiple repos at the same time for the same
product, you may hit a race condition (see issue for full details and
repro script).

This patch fixes the issue by avoiding
`Candlepin::Environment::SetContent`, and adds a new
`AddContentToEnvironment` task. The new task simply makes an attempt
to add a single content id to a product, and if the content already
exists (409 from candlepin), it logs an info message and no-ops.

History

#1 Updated by The Foreman Bot 7 months ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/7867 added

#2 Updated by Andrew Kofink 7 months ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.11.0

#3 Updated by Brad Buckingham 7 months ago

  • Bugzilla link set to 1609543

#4 Updated by Chris Duryee 7 months ago

  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF