Bug #17564
closedUnpacking tree in Red Hat Repositories sequentially contacts CDN for each and every CDN path
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1398945
Description of problem:
User scenario: I want to enable some CDN repo. Going to WebUI -> Content -> Red Hat Repositories. Once that is loaded, navigating / unwrapping any "tree leaf" (that already shows particular repos, not sub-tree), this step takes tens of seconds.
The reason is evident when enabling debug logs:
Rails.logger.debug "CDN: Requesting path #{used_url}" in /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.82/app/lib/katello/resources/cdn.rb is logged for each and every path that katello scans on CDN. Unwrapping " Red Hat Enterprise Linux 6 Server (RPMs) " logs
2016-11-27 14:57:15 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/listing
2016-11-27 14:57:17 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.1/listing
2016-11-27 14:57:18 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.2/listing
2016-11-27 14:57:20 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.3/listing
2016-11-27 14:57:22 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.4/listing
2016-11-27 14:57:24 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.5/listing
2016-11-27 14:57:26 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.6/listing
2016-11-27 14:57:28 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.7/listing
2016-11-27 14:57:30 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6.8/listing
2016-11-27 14:57:32 [app] [D] CDN: Requesting path https://cdn.redhat.com:443/content/dist/rhel/server/6/6Server/listing
See the time spent between individual log entries. And also total time spent to process the user action.
Version-Release number of selected component (if applicable):
Sat6.2.4
How reproducible:
100%
Steps to Reproduce:
1. Have some manifest granting access to RHEL6 base repos.
2. In WebUI, click to Content -> Red Hat Repositories
3. Click to "Red Hat Enterprise Linux"
4. Measure how much time it takes to unwrap " Red Hat Enterprise Linux 6 Server (RPMs) "
5. Optionally enable debugs to see the "CDN: Requesting path .." logs.
Actual results:
30 seconds or so
Expected results:
below 10 seconds, if possible
Additional info: