Project

General

Profile

Bug #14807

Capsule sync redundantly generates metadata for all repos

Added by Tomáš Strachota about 3 years ago. Updated about 1 year ago.

Status:
Rejected
Priority:
Normal
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1315326
Description of problem:
Invoking capsule sync:
1) Sat orders capsule to sync all repositories in all content views / lifecycle environments to the capsule
2) for every such repository, pulp on the capsule generates new repo metadata

Assume a use case where Sat having few hundreds of repositories in different content views, and just some repo might need to be synced (as there are doubts if it was synced properly). A capsule sync would do so so much redundant work taking hours.

To see the scope of ridiculous work being done:
- assume a use case where large repos (say rhel5-7 base) are present in many content views as a base - repo metadata takes nontrivial time to be computed, multiply it by # of repos..
- particular example: a customer behind this bug has 1250 repos and capsule sync takes 3-4 hours(!) doing nothing

Please optimize either 1 or 2. While I understand Sat does not know what repo needs and what does not need to be synced to the capsule (i.e. 1 sounds legit), there should be an option to e.g. fetch metadata from Sat to Caps, compare if they are the same and if so, do nothing (and if differ or missing on Caps, then do the sync).

Version-Release number of selected component (if applicable):
Sat 6.1.7

How reproducible:
100%

Steps to Reproduce:
1. Have more pulp repos enabled (e.g. have sat61-tools repo in 10 published content views)
2. do repeatedly capsule sync, without any repo / content view manipulation meantime
3. Check times of execution of 1st sync and other synces
4. ll /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/ContentViewName1/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/repodata/

Actual results:
3. synces takes the same/very similar time, nontrivial
4. repodata recalculated every time

Expected results:
3. sync takes small time, if no work to be done
4. repodata not updated / recalculated every time (if there is no need to)

Additional info:

History

#1 Updated by Tomáš Strachota about 3 years ago

  • Category changed from Web UI to Repositories
  • Status changed from New to Assigned

#2 Updated by The Foreman Bot about 3 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/5996 added

#3 Updated by Tomáš Strachota about 3 years ago

  • Status changed from Ready For Testing to Closed

I'm closing the issue. Per discussion with Justin optimizations have been made on Pulp side and metadata creation process is much faster now. Therefore it's not necessary to skip it.

#4 Updated by Eric Helms about 3 years ago

  • Status changed from Closed to Rejected
  • Legacy Backlogs Release (now unused) set to 114

Also available in: Atom PDF