Project

General

Profile

Actions

Bug #14807

closed

Capsule sync redundantly generates metadata for all repos

Added by Tomáš Strachota almost 8 years ago. Updated almost 4 years ago.

Status:
Rejected
Priority:
Normal
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1315326
Description of problem:
Invoking capsule sync:
1) Sat orders capsule to sync all repositories in all content views / lifecycle environments to the capsule
2) for every such repository, pulp on the capsule generates new repo metadata

Assume a use case where Sat having few hundreds of repositories in different content views, and just some repo might need to be synced (as there are doubts if it was synced properly). A capsule sync would do so so much redundant work taking hours.

To see the scope of ridiculous work being done:
- assume a use case where large repos (say rhel5-7 base) are present in many content views as a base - repo metadata takes nontrivial time to be computed, multiply it by # of repos..
- particular example: a customer behind this bug has 1250 repos and capsule sync takes 3-4 hours(!) doing nothing

Please optimize either 1 or 2. While I understand Sat does not know what repo needs and what does not need to be synced to the capsule (i.e. 1 sounds legit), there should be an option to e.g. fetch metadata from Sat to Caps, compare if they are the same and if so, do nothing (and if differ or missing on Caps, then do the sync).

Version-Release number of selected component (if applicable):
Sat 6.1.7

How reproducible:
100%

Steps to Reproduce:
1. Have more pulp repos enabled (e.g. have sat61-tools repo in 10 published content views)
2. do repeatedly capsule sync, without any repo / content view manipulation meantime
3. Check times of execution of 1st sync and other synces
4. ll /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/ContentViewName1/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.1/os/repodata/

Actual results:
3. synces takes the same/very similar time, nontrivial
4. repodata recalculated every time

Expected results:
3. sync takes small time, if no work to be done
4. repodata not updated / recalculated every time (if there is no need to)

Additional info:

Actions #1

Updated by Tomáš Strachota almost 8 years ago

  • Category changed from Web UI to Repositories
  • Status changed from New to Assigned
Actions #2

Updated by The Foreman Bot almost 8 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/5996 added
Actions #3

Updated by Tomáš Strachota almost 8 years ago

  • Status changed from Ready For Testing to Closed

I'm closing the issue. Per discussion with Justin optimizations have been made on Pulp side and metadata creation process is much faster now. Therefore it's not necessary to skip it.

Actions #4

Updated by Eric Helms almost 8 years ago

  • Status changed from Closed to Rejected
  • translation missing: en.field_release set to 114
Actions #5

Updated by Jonathon Turel almost 4 years ago

  • Target version changed from Katello Backlog to Katello Recycle Bin
Actions

Also available in: Atom PDF