Project

General

Profile

Bug #30694

Redhat repos are no longer listed completely

Added by Richard Stempfl 2 months ago. Updated 12 days ago.

Status:
Closed
Priority:
High
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

On foreman 1.24.3 it works as usual.
But if I load the same manifest on a foreman 2.1 I only get to see a few versions of repos, sometimes only some versions are missing, sometimes every version.
I also noticed that the time until the results appear is much longer. Before it was 5 seconds now it is 60 seconds until the contents are displayed.

No log information found.

One Pic is from Foreman 1.24 the other with less versions is from Foreman 2.1

Foreman 1.24

Foreman 2.1

Screenshot_20200820_165514.png View Screenshot_20200820_165514.png 131 KB Foreman 2.1 Richard Stempfl, 08/24/2020 12:44 PM
Screenshot_20200820_165449.png View Screenshot_20200820_165449.png 57.7 KB Foreman 1.24 Richard Stempfl, 08/24/2020 12:45 PM
Screenshot 20200820 165514 Screenshot 20200820 165449

Associated revisions

Revision e433f4a4 (diff)
Added by Justin Sherrill 28 days ago

Fixes #30694 - limit threading on cdn scan

History

#1 Updated by Tomer Brisker 2 months ago

  • Project changed from Foreman to Katello

#2 Updated by Bernhard Suttner 2 months ago

  • Found in Releases Katello 3.16.0 added
  • Found in Releases deleted (2.1.1)

#3 Updated by Bernhard Suttner 2 months ago

  • Priority changed from Normal to High

#4 Updated by Jonathon Turel 2 months ago

Can you try refreshing your subscription manifest and see if that has any effect on the results in the list? Thanks for the report!

#5 Updated by Jonathon Turel 2 months ago

  • Status changed from New to Need more information

#6 Updated by Brad Glascock 2 months ago

I have the same issue and have refreshed the manifest a couple times. Tried both with simple content access enabled and without.

#7 Updated by Richard Stempfl 2 months ago

refresh of the manifest unfortunately brings no improvement. From time to time other release versions are shown but never completely

#8 Updated by Chris Roberts about 2 months ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.17.0
  • Status changed from Need more information to New
  • Category set to Repositories

#9 Updated by Bernhard Suttner about 2 months ago

This is broken for the Katello 3.16.0. I think it should be at least cherry-picked back to 3.16 as its a very important issue.

#10 Updated by Richard Stempfl about 2 months ago

In the meantime I have noticed that only redhat7 repositories are affected. redhat8 works.

#11 Updated by Justin Sherrill about 2 months ago

  • Target version changed from Katello 3.17.0 to Katello Recycle Bin
  • Status changed from New to Rejected

This is caused by an issue in dynflow, and fixed here: https://github.com/Dynflow/dynflow/pull/362

the fix isn't released yet though. You can try to apply the patch locally and restart all services, or enable the repo using hammer cli.

#12 Updated by Marcel Kühlhorn about 1 month ago

Justin Sherrill wrote:

This is caused by an issue in dynflow, and fixed here: https://github.com/Dynflow/dynflow/pull/362

the fix isn't released yet though. You can try to apply the patch locally and restart all services, or enable the repo using hammer cli.

Applying the patch slightly improved things:
Now, on a foreman server that accesses the Internet through an HTTP proxy, for the first product selected all versions are shown, but trying to list versions for another product after enabling one of them leads to "No repositories available"
After reloading the page the versions for that product can be listed completely, so you now have to reload the page every time you extended a products version list and want to list another

#13 Updated by Tomer Brisker about 1 month ago

  • Triaged changed from Yes to No
  • Target version deleted (Katello Recycle Bin)
  • Status changed from Rejected to New

reopening per comment #12

#14 Updated by Justin Sherrill about 1 month ago

  • Status changed from New to Need more information

I've never seen this new behavior you describe. A few questions:

1) do you have to 'enable' a repo and then expand a new repo set to see it come up empty? or does just expanding two repos sets in a row also reproduce the behavior?

2) does it happen every time?

3) is it possible to try without an http proxy? (not that i know why that make any difference assuming there is not any sort of connection limit)

#15 Updated by Justin Sherrill about 1 month ago

Also, after reproducing the issue, can you go to Monitor > tasks and look for the most recent 'scan CDN' task. Are there any errors in it?

#16 Updated by Marcel Kühlhorn about 1 month ago

1) Enabling a repository seems to increase the frequency at which it happens, but after expanding multiple repos it eventually happens without too

2) When I wrote the first comment it happened every time, now its more sporadic

3) I'll have to try that still

4) the scan CDN tasks do not show any error

As Richard noted further up, this seems to only affect RHEL 7 repositories

#17 Updated by Marcel Kühlhorn about 1 month ago

I can now say it only happens with HTTP Proxy.

For completeness sake I also tried disabling Auth in squid, with the same result:
after a random number of expanded repos I get No repositories available.
The squid access log shows no connection errors either.

My reproducer is simply putting "rhel-7" in the search box and then expand a number of repos until the failure to load more occurs

#18 Updated by Justin Sherrill about 1 month ago

  • Category deleted (Repositories)

Thanks Marcel!!!

Does the version you are using have this change: https://github.com/Katello/katello/pull/8728/files

(I don't think it does, as that issue is marked for katello 3.17). Would you be able to apply that patch, restart services (including dynflow*) and re-test. Then look in the logs for 'Failed at scanning for repository'

#19 Updated by Justin Sherrill about 1 month ago

Specifically /var/log/foreman/production.log

#20 Updated by Marcel Kühlhorn about 1 month ago

With the patch I get this in production.log:

2020-09-17T15:48:49 [I|app|] Completed 500 Internal Server Error in 7526ms (Views: 0.6ms | ActiveRecord: 15.5ms | Allocations: 41764)
2020-09-17T15:48:49 [E|app|] Failed at scanning for repository: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use

I also now see the scan CDN tasks ending up in warning state with

Exception:
ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use

#21 Updated by The Foreman Bot about 1 month ago

  • Assignee set to Justin Sherrill
  • Status changed from Need more information to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/8963 added

#22 Updated by Justin Sherrill about 1 month ago

Would you be able to try this patch? https://github.com/Katello/katello/pull/8963

this limits the number of concurrent fetches to 5. If you still hit the issue, you may want to try lowering it. Putting it at 1 would basically make it sequential.

You'll need to restart the dynflow workers between each try.

#23 Updated by Marcel Kühlhorn about 1 month ago

After some testing I found that to reliably not run into the error even when extending multiple repos at the same time I had to decrease the group size down to 2.

#24 Updated by Ian Ballou about 1 month ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.17.0
  • Category set to Repositories

#25 Updated by Justin Sherrill about 1 month ago

Hello!, I updated the pr with another change. This change should cause the http proxy information being loaded prior to spawning threads. This should reduce the amount of db access that is needed within the threads. I also bumped up the thread count to 8 by default, but i'm curious again with this change what seems to work for you.

#26 Updated by Marcel Kühlhorn about 1 month ago

Thank you Justin, with the latest version of the patch it works reliable with your set group size

#27 Updated by Justin Sherrill about 1 month ago

Awesome to hear! Thank you for the testing!

#28 Updated by Justin Sherrill about 1 month ago

  • Target version changed from Katello 3.17.0 to Katello 3.16.1

#29 Updated by Justin Sherrill about 1 month ago

  • Bugzilla link set to 1882690

#30 Updated by The Foreman Bot 28 days ago

  • Fixed in Releases Katello 4.0.0 added

#31 Updated by Justin Sherrill 28 days ago

  • Status changed from Ready For Testing to Closed

#32 Updated by Marcel Kühlhorn 18 days ago

  • Found in Releases Katello 3.16.1.2 added

#33 Updated by Marcel Kühlhorn 12 days ago

  • Found in Releases deleted (Katello 3.16.1.2)

Also available in: Atom PDF