Project

General

Profile

Bug #23322

[Performance improvement] On CV publish/promote, call Pulp::Repository::* dynflow steps concurrently

Added by Justin Sherrill about 4 years ago. Updated almost 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
Content Views
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1554282

Description of problem:
When publishing a CV or promoting it to a LE, a content of a pulp repo (or two repos, sequentially) is copied to another one. That is achieved via Pulp::Repository::* dynflow steps that each copies a different type of content units (rpm, errata, srpm,..).

Currently, these dynflow steps are run sequentially, so the call flow is:
- trigger Pulp::Repository::CopySrpm
- wait until it completes - by periodical checking of the pulp task status
- after it completes, trigger Pulp::Repository::CopyRpm
- wait until it completes
- trigger ..
..

These dynflow steps can be executed concurrently - pulp engineering confirmed me this is safe to trigger those pulp tasks concurrently. Internally in pulp, they will be performed sequentially either way, but simply we dont require the extra "locking" on dynflow steps level. Since the waiting for task completion adds redundant delay too many times.

Preliminary tests showed observable-to-significant improvement in CV publish time, but these will definitely vary on many factors (i.e. # and size of repos in the CV, # of pulp workers, other dynflow tasks etc.).

Version-Release number of selected component (if applicable):
Sat 6.3.0

How reproducible:
100%

Steps to Reproduce:
1. Publish or promote a CV with yum repos.
2. Count the time it took

Actual results:
2. is slower when https://github.com/Katello/katello/blob/master/app/lib/actions/katello/repository/clone_yum_content.rb#L22 is "sequence do"

Expected results:
2. is faster if https://github.com/Katello/katello/blob/master/app/lib/actions/katello/repository/clone_yum_content.rb#L22 is "concurrence do"

Additional info:
(*) Not sure if the above change improves also CV promote (it should, havent tested). It doesnt improve incremental update where my proposal isnt applicable.

(*) I will provide a script to test performance gain

History

#1 Updated by Jonathon Turel about 4 years ago

  • Legacy Backlogs Release (now unused) set to 349
  • Assignee set to Samir Jha

#2 Updated by The Foreman Bot about 4 years ago

  • Pull request https://github.com/Katello/katello/pull/7343 added

#3 Updated by Samir Jha almost 4 years ago

  • Status changed from New to Rejected

#4 Updated by Samir Jha almost 4 years ago

  • Target version changed from Katello 3.8.0 to Katello Recycle Bin

Also available in: Atom PDF