Project

General

Profile

Bug #17564

Unpacking tree in Red Hat Repositories sequentially contacts CDN for each and every CDN path

Added by Justin Sherrill over 2 years ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1398945

Description of problem:
User scenario: I want to enable some CDN repo. Going to WebUI -> Content -> Red Hat Repositories. Once that is loaded, navigating / unwrapping any "tree leaf" (that already shows particular repos, not sub-tree), this step takes tens of seconds.

The reason is evident when enabling debug logs:

Rails.logger.debug "CDN: Requesting path #{used_url}" in /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/katello/resources/cdn.rb is logged for each and every path that katello scans on CDN. Unwrapping " Red Hat Enterprise Linux 6 Server (RPMs) " logs

2016-11-27 14:57:15 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/listing
2016-11-27 14:57:17 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.1/listing
2016-11-27 14:57:18 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.2/listing
2016-11-27 14:57:20 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.3/listing
2016-11-27 14:57:22 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.4/listing
2016-11-27 14:57:24 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.5/listing
2016-11-27 14:57:26 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.6/listing
2016-11-27 14:57:28 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.7/listing
2016-11-27 14:57:30 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.8/listing
2016-11-27 14:57:32 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6Server/listing

See the time spent between individual log entries. And also total time spent to process the user action.

Version-Release number of selected component (if applicable):
Sat6.2.4

How reproducible:
100%

Steps to Reproduce:
1. Have some manifest granting access to RHEL6 base repos.
2. In WebUI, click to Content -> Red Hat Repositories
3. Click to "Red Hat Enterprise Linux"
4. Measure how much time it takes to unwrap " Red Hat Enterprise Linux 6 Server (RPMs) "
5. Optionally enable debugs to see the "CDN: Requesting path .." logs.

Actual results:
30 seconds or so

Expected results:
below 10 seconds, if possible

Additional info:

Associated revisions

Revision 1897a6bb (diff)
Added by Justin Sherrill about 2 years ago

Fixes #17564 - use threading to improve RH repos page

This commit spawns threads in order to speed up the
Red Hat Repositories page for fetching listing files.

Given a directory structure such as:

/path/$RELEASEVER/$ARCH

we have to make 1 + $RELEASEVER number of requests. On
a high latency connection this can take some time per request.
With this change we put each request in a thread and so are able
speed up the total time to be closer to the same amount of time
as two requests.

In addition, this refactors the code removing features of the
cdn_var_substitutor that are no longer needed and making the code
much more readable and better tested.

History

#1 Updated by The Foreman Bot over 2 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/6481 added

#2 Updated by Justin Sherrill over 2 years ago

  • Target version set to 147

#3 Updated by Justin Sherrill over 2 years ago

  • Legacy Backlogs Release (now unused) set to 188

#4 Updated by Justin Sherrill over 2 years ago

  • Target version changed from 147 to 157

#5 Updated by Justin Sherrill over 2 years ago

  • Legacy Backlogs Release (now unused) changed from 188 to 114

#6 Updated by The Foreman Bot over 2 years ago

  • Legacy Backlogs Release (now unused) deleted (114)

#7 Updated by Brad Buckingham over 2 years ago

  • Target version changed from 157 to 163

#8 Updated by Justin Sherrill over 2 years ago

  • Legacy Backlogs Release (now unused) set to 114

#9 Updated by The Foreman Bot over 2 years ago

  • Legacy Backlogs Release (now unused) deleted (114)

#10 Updated by Eric Helms over 2 years ago

  • Legacy Backlogs Release (now unused) set to 211

#11 Updated by Justin Sherrill about 2 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF