Project

General

Profile

Bug #17691

"hammer repository upload-content --path" redundantly non-performant against big repos

Added by Brad Buckingham over 2 years ago. Updated 9 months ago.

Status:
Closed
Priority:
High
Category:
Repositories
Target version:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1402423

Description of problem:
"hammer repository upload-content --path" command invokes for each and every RPM in the directory foreman task Actions::Katello::Repository::ImportUpload. That task consists of:

Actions::Pulp::Repository::ImportUpload
Actions::Pulp::Repository::DistributorPublish
Actions::Katello::Repository::FilteredIndexContent

dynflow sub-tasks.

Assume one uploads hundreds of RPMs in bulk action to a big repo. Then we call pulp sequence "upload RPM to repo, publish repo" 100times in a row.

If the repo is big in content, every publish can take tens of seconds. Redundantly, since another iteration of the loop will discard the published bits. This can even cause upload package failure per https://bugzilla.redhat.com/show_bug.cgi?id=1402415 .

Please ensure by either way that the DistributorPublish (and I expect FilteredIndexContent as well) is called just once, at the end of the bulk action.

I file the BZ against hammer, though a change in hammer and foreman/katello tasks needs to be done.

Version-Release number of selected component (if applicable):
tfm-rubygem-hammer_cli_katello-0.0.22.26-1.el7sat.noarch
Sat6.2.4

How reproducible:
100%

Steps to Reproduce:
1. Have a big repo (say RHEL6 base one)
2. Bulk upload many RPMs via "hammer repository upload-content --path /tmp/dir_with_many_rpms"
3. Count the time, check for potential failed uploads and for the time spent in individual DistributorPublish sub-tasks.

Actual results:
Huge time, some uploads might fail (if repo publish exceeds hammer timeout, see bz1402415 for reasoning), great deal of the time spent in DistributorPublish steps.

Expected results:
Substantially slower times, no faile uploads, DistributorPublish called just once at the end.

Additional info:
Idea of fix: add option "skip_publish" to Actions::Katello::Repository::ImportUpload and hammer sets it to true for all except the latest RPM upload. skip_publish=true will not trigger the 2 dynflow steps


Related issues

Blocked by Katello - Bug #17941: allow for force generation of repo metadataClosed2017-01-05

Associated revisions

Revision 9e1a95b7 (diff)
Added by Justin Sherrill about 2 years ago

Refs #17691 - Add param to not regen repository on upload

This allows for multiple content uploads before
publishing repository metadata

Revision ab8925e7 (diff)
Added by Justin Sherrill about 2 years ago

Fixes #17691 - publish repo after all content uploads (#472)

History

#1 Updated by Brad Buckingham over 2 years ago

  • Subject changed from "hammer repository upload-content --path" redundantly non-performant against big repos to "hammer repository upload-content --path" redundantly non-performant against big repos
  • Legacy Backlogs Release (now unused) set to 114

#2 Updated by Justin Sherrill about 2 years ago

  • Assignee set to Justin Sherrill

#3 Updated by Justin Sherrill about 2 years ago

  • Blocked by Bug #17941: allow for force generation of repo metadata added

#4 Updated by Justin Sherrill about 2 years ago

  • Target version set to 151
  • Difficulty set to medium

#5 Updated by Brad Buckingham about 2 years ago

  • Target version changed from 151 to 157

#6 Updated by The Foreman Bot about 2 years ago

  • Status changed from New to Ready For Testing
  • Legacy Backlogs Release (now unused) deleted (114)
  • Pull request https://github.com/Katello/hammer-cli-katello/pull/472 added

#7 Updated by The Foreman Bot about 2 years ago

  • Pull request https://github.com/Katello/katello/pull/6544 added

#8 Updated by Justin Sherrill about 2 years ago

  • Legacy Backlogs Release (now unused) set to 211

#9 Updated by Anonymous about 2 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF