Bug #29370: Syncing a large product leaves Pulp tasks "waiting". - Katello - Foreman

Actions

Copy link

Bug #29370

open

Syncing a large product leaves Pulp tasks "waiting".

Added by Ian Ballou over 4 years ago. Updated about 1 month ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Repositories

Target version:

Difficulty:

Triaged:

Yes

Bugzilla link:

Pull request:

Fixed in Releases:

Found in Releases:

Red Hat JIRA:

Description

I've tried a few times to sync a product with ~200 repositories and it never completes successfully. There are always sync tasks left over that are stuck waiting on "pulp:action:publish". The tasks waiting are a mix of Docker and File repos. I've tried restarting Pulp services but that doesn't help. It seems this could be a Katello bug or a Pulp bug.

Detailed information about the server:

Repositories: 212
~~> 154 Docker repos with a mix of BusyBox, Alpine, and Bash. These are Docker Hub official images.~~
> 58 file repos. All except 6 have the upstream url http://quartet.usersys.redhat.com/pub/fake-repos/large_file/. 1 has https://repos.fedorapeople.org/repos/pulp/pulp/fixtures/file-many/ and the other 5 have no upstream url.

CPUs: 8
RAM: 32 GB
Pulp workers: 8

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Chris Roberts over 4 years ago

Category set to Repositories
Target version set to Katello 3.16.0
Triaged changed from No to Yes

Actions

Copy link

Updated by Justin Sherrill over 4 years ago

Has duplicate Bug #29458: Syncing of repos got stuck in foreman (katello nightly version ) added

Actions

Copy link

Updated by Suresh Thirugnanasambandan over 4 years ago

Any update on this?

Actions

Copy link

Updated by Imaan please_edit_me over 4 years ago

Hello,

This issue is blocking us to test the performance of the katello nightly with the puma web server. Is there any work-around for this issue?

Any suggestions/recommendations??

Actions

Copy link

Updated by Justin Sherrill over 4 years ago

Would you be able to try a couple things:

1. in /etc/foreman/database.yml, change the 'pool' setting from '5' to '30':

production:
  adapter: postgresql
  database: foreman
  username: foreman
  password: "SMA42qaA3FFFm2ioiam8RrMJMxpBZ7CB" 
  pool: 30

2. Create a new file:
/usr/share/foreman/config/initializers/foreman-tasks-db-pool-size.rb

with this one line:

ForemanTasks.dynflow.config.db_pool_size = 30

3. run foreman-maintain service restart

can you see if that helps at all? I have seen some benefit (with ~40 syncs).

Actions

Copy link

Updated by Ewoud Kohl van Wijngaarden over 4 years ago

There is an installer option --foreman-db-pool which can be used to change the pool size in /etc/foreman/database.yml. If we need to tune the DB pool size as well then we need a proper way to configure this.

Actions

Copy link