Project

General

Profile

Actions

Bug #17691

closed

"hammer repository upload-content --path" redundantly non-performant against big repos

Added by Brad Buckingham over 7 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
High
Category:
Repositories
Target version:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1402423

Description of problem:
"hammer repository upload-content --path" command invokes for each and every RPM in the directory foreman task Actions::Katello::Repository::ImportUpload. That task consists of:

Actions::Pulp::Repository::ImportUpload
Actions::Pulp::Repository::DistributorPublish
Actions::Katello::Repository::FilteredIndexContent

dynflow sub-tasks.

Assume one uploads hundreds of RPMs in bulk action to a big repo. Then we call pulp sequence "upload RPM to repo, publish repo" 100times in a row.

If the repo is big in content, every publish can take tens of seconds. Redundantly, since another iteration of the loop will discard the published bits. This can even cause upload package failure per https://bugzilla.redhat.com/show_bug.cgi?id=1402415 .

Please ensure by either way that the DistributorPublish (and I expect FilteredIndexContent as well) is called just once, at the end of the bulk action.

I file the BZ against hammer, though a change in hammer and foreman/katello tasks needs to be done.

Version-Release number of selected component (if applicable):
tfm-rubygem-hammer_cli_katello-0.0.22.26-1.el7sat.noarch
Sat6.2.4

How reproducible:
100%

Steps to Reproduce:
1. Have a big repo (say RHEL6 base one)
2. Bulk upload many RPMs via "hammer repository upload-content --path /tmp/dir_with_many_rpms"
3. Count the time, check for potential failed uploads and for the time spent in individual DistributorPublish sub-tasks.

Actual results:
Huge time, some uploads might fail (if repo publish exceeds hammer timeout, see bz1402415 for reasoning), great deal of the time spent in DistributorPublish steps.

Expected results:
Substantially slower times, no faile uploads, DistributorPublish called just once at the end.

Additional info:
Idea of fix: add option "skip_publish" to Actions::Katello::Repository::ImportUpload and hammer sets it to true for all except the latest RPM upload. skip_publish=true will not trigger the 2 dynflow steps


Related issues 1 (0 open1 closed)

Blocked by Katello - Bug #17941: allow for force generation of repo metadataClosedJustin Sherrill01/05/2017Actions
Actions #1

Updated by Brad Buckingham about 7 years ago

  • Subject changed from "hammer repository upload-content --path" redundantly non-performant against big repos to "hammer repository upload-content --path" redundantly non-performant against big repos
  • translation missing: en.field_release set to 114
Actions #2

Updated by Justin Sherrill about 7 years ago

  • Assignee set to Justin Sherrill
Actions #3

Updated by Justin Sherrill about 7 years ago

  • Blocked by Bug #17941: allow for force generation of repo metadata added
Actions #4

Updated by Justin Sherrill about 7 years ago

  • Target version set to 151
  • Difficulty set to medium
Actions #5

Updated by Brad Buckingham about 7 years ago

  • Target version changed from 151 to 157
Actions #6

Updated by The Foreman Bot about 7 years ago

  • Status changed from New to Ready For Testing
  • translation missing: en.field_release deleted (114)
  • Pull request https://github.com/Katello/hammer-cli-katello/pull/472 added
Actions #7

Updated by The Foreman Bot about 7 years ago

  • Pull request https://github.com/Katello/katello/pull/6544 added
Actions #8

Updated by Justin Sherrill about 7 years ago

  • translation missing: en.field_release set to 211
Actions #9

Updated by Anonymous about 7 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF