"hammer repository upload-content --path" redundantly non-performant against big repos
Description of problem:
"hammer repository upload-content --path" command invokes for each and every RPM in the directory foreman task Actions::Katello::Repository::ImportUpload. That task consists of:
Assume one uploads hundreds of RPMs in bulk action to a big repo. Then we call pulp sequence "upload RPM to repo, publish repo" 100times in a row.
If the repo is big in content, every publish can take tens of seconds. Redundantly, since another iteration of the loop will discard the published bits. This can even cause upload package failure per https://bugzilla.redhat.com/show_bug.cgi?id=1402415 .
Please ensure by either way that the DistributorPublish (and I expect FilteredIndexContent as well) is called just once, at the end of the bulk action.
I file the BZ against hammer, though a change in hammer and foreman/katello tasks needs to be done.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Have a big repo (say RHEL6 base one)
2. Bulk upload many RPMs via "hammer repository upload-content --path /tmp/dir_with_many_rpms"
3. Count the time, check for potential failed uploads and for the time spent in individual DistributorPublish sub-tasks.
Huge time, some uploads might fail (if repo publish exceeds hammer timeout, see bz1402415 for reasoning), great deal of the time spent in DistributorPublish steps.
Substantially slower times, no faile uploads, DistributorPublish called just once at the end.
Idea of fix: add option "skip_publish" to Actions::Katello::Repository::ImportUpload and hammer sets it to true for all except the latest RPM upload. skip_publish=true will not trigger the 2 dynflow steps
Refs #17691 - Add param to not regen repository on upload
This allows for multiple content uploads before
publishing repository metadata
#9 Updated by Anonymous over 2 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset hammer-cli-katello|ab8925e79445d759be128ff2ae8caf5083395361.