Project

General

Profile

Bug #31254

"Package matching query does not exist." error when syncing two repositories, that contain identical packages

Added by Carsten Menzel over 1 year ago. Updated 12 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

When syncing the Oracle Linux repositories "Oracle Linux 7 (x86_64) Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64) and "Oracle Linux 7 (x86_64) Optional Latest" (http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64), the synchronization of the repo, which is synced after the other one fails with the error "Package matching query does not exist".

When looking at the pulp worker logs, I can see, that the database throws a duplicate key error, because some packages are present in both repositories:

Examples:
c-ares-devel-1.10.0-3.el7.i686.rpm
c-ares-devel-1.10.0-3.el7.x86_64.rpm
cdparanoia-10.2-17.el7.x86_64.rpm

I found this behaviour in both Katello 3.16 and in Katello 3.17RC2.

As I don't have any possibility to change the foreign repositories, it would be nice to have either a possibility to filter out the packages like in Content Views, so that they are not synced for specific repositories, or if the duplicate would be detected by an error handling and the existing package is also linked to the other repository (no duplicate in the file system, but an additional reference in the database).

I attached a PDF with the error in the Foreman interface, the pulp worker log with the error and the corresponding entry from the database.

Kind regards
Carsten

package_sync_error.pdf View package_sync_error.pdf 63.4 KB Carsten Menzel, 11/04/2020 08:30 AM
pulp_error.txt pulp_error.txt 15.3 KB Carsten Menzel, 11/04/2020 08:30 AM
package_in_database.txt package_in_database.txt 7.27 KB Carsten Menzel, 11/04/2020 08:30 AM

History

#1 Updated by Carsten Menzel over 1 year ago

  • Category set to Repositories

#2 Updated by Ian Ballou over 1 year ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.16.2

TODO: We should investigate if this is strictly a Pulp 3 bug.

#3 Updated by Carsten Menzel over 1 year ago

Good morning,
yesterday I did some further tests to see, why the problem occurs.

It seems to me, that the problem occurs, if a product with multiple large repositories is generated and is initially synced in parallel (complete product sync).

##########

Commands to create an example product, which contains the two repositories that contain some identical packages ("OL 7 - Latest" and "OL 7 - Optional Latest"):

hammer product create --organization "<organization name>" --name "Oracle Linux"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 6 - Latest" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 6 - Addons" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 7 - Latest" --content-type "yum" --url "https://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 7 - Optional Latest" --content-type "yum" --url "https://public-yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 7 - Addons" --content-type "yum" --url "https://public-yum.oracle.com/repo/OracleLinux/OL7/addons/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 7 - Software Collection" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL7/SoftwareCollections/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 8 - AppStream Latest" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 8 - Addons" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL8/addons/x86_64/"
hammer repository create --organization "<organization name>" --product "Oracle Linux" --name "OL 8 - BaseOS Latest" --content-type "yum" --url "https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/"

###########

Sync the complete product, so that all repositories are synced at the same time.
This seems to lead to the error situation.

Perhaps this is a timing issue when a proxy is used (which slows down the downloads a bit).

Yesterday I deleted in my system both repositories ("OL 7 - Latest" and "OL 7 - Optional Latest"), removed all orphaned content and recreated the two repositories.
After that, I synced the Repos one after the other (the first one yesterday in the evening, the second one today in the morning)
In this case, the error did not occur anymore.

Kind Regards
Carsten

#4 Updated by Carsten Menzel over 1 year ago

As this issue was a temporary problem and could be caused by a timing issue during the initial sync, I don't think, that this is a general bug anymore.
As I now know how to avoid the problem, you can close the bug (either as resolved or as rejected).

Regards
Carsten

#5 Updated by Justin Sherrill over 1 year ago

I think this is a pulp3 bug still worth investigating. I filed: https://pulp.plan.io/issues/7828

#6 Updated by Ian Ballou over 1 year ago

  • Target version changed from Katello 3.16.2 to 1341

#7 Updated by Ian Ballou over 1 year ago

  • Target version changed from 1341 to Katello 3.17.2

#8 Updated by Chris Roberts over 1 year ago

  • Target version changed from Katello 3.17.2 to Katello 3.17.3

#9 Updated by Jonathon Turel about 1 year ago

  • Target version changed from Katello 3.17.3 to Katello 4.1.0

#10 Updated by Jonathon Turel 12 months ago

  • Target version changed from Katello 4.1.0 to Katello Recycle Bin
  • Status changed from New to Closed

Closing this out since the root of the issue is on the Pulp side. However we will raise this with pulp devs and slot the fix into a future Katello release. Thanks for filing :)

Also available in: Atom PDF