Bug #29979
closed
Synchronize repository task hanging
Description
It's happening after a day or two that during a sync plan run one or more repository sync tasks are hanging and never finish. It’s different repositories and at different times. It’s working fine before that for multiple times. A reboot or restart of katello and a manual sync work fine.
Tasks are pending, delayed.
Running action is Actions::Pulp::Repository::Sync in state suspended.
Dynflow console says “waiting for Pulp to finish the task”.
It will stay like this until a restart of katello/foreman.
It's a new installation of katello 13.5 which I have updated to 3.15.1.1 recently. But the problem persists.
Updated by Jonathon Turel almost 4 years ago
- Status changed from New to Need more information
Next time this happens can you try restarting just the pulp_resource_manager and pulp_workers services? If that fixes the problem, that'll help us narrow down the cause.
Updated by Gerald Vogt almost 4 years ago
- Found in Releases Katello 3.15.1 added
- Found in Releases deleted (
Katello 3.15.1.1)
O.K. Tonight it happened again. The sync task for CentOS 8 PowerTools is hanging for a couple of hours now.
As requested I restarted the services:
# systemctl restart pulp_resource_manager.service pulp_worker*.service
However, the hanging task still shows in foreman. So no change here.
What I have noticed, though I can't tell for sure: I have two sync plans (one each hour, one each four hours). They are actually start at the same minute 53. I think, the hanging task only happens at times when both sync plans start. Tonight it was at 12:53 which was start time for both plans. But I am not 100% sure about this. I know a couple of hangs on EPEL syncs which are on my 4 hour plan. The CentOS 8 PowerTools is on the hourly but than hang was again at the time when the four plan was also running.
So maybe it's an issue when sync task runs overlap?
Updated by Gerald Vogt almost 4 years ago
- Found in Releases Katello 3.15.1.1 added
Updated by Chris Roberts almost 4 years ago
- Status changed from Need more information to New
Updated by Gerald Vogt almost 4 years ago
Tonight it happened again during the hourly sync plan. The four hour sync plan was not running at that time. Thus, it also happens when there is only a single sync plan running.
Do you need more information?
Updated by James Jeffers almost 4 years ago
- Status changed from New to Closed
- Triaged changed from No to Yes
Closed in favor of discussion at https://community.theforeman.org/t/synchronize-repository-task-hanging/18970/12
Updated by Jonathon Turel almost 4 years ago
- Target version set to Katello Recycle Bin