Project

General

Profile

Bug #29979

Synchronize repository task hanging

Added by Gerald Vogt almost 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Red Hat JIRA:

Description

It's happening after a day or two that during a sync plan run one or more repository sync tasks are hanging and never finish. It’s different repositories and at different times. It’s working fine before that for multiple times. A reboot or restart of katello and a manual sync work fine.

Tasks are pending, delayed.

Running action is Actions::Pulp::Repository::Sync in state suspended.

Dynflow console says “waiting for Pulp to finish the task”.

It will stay like this until a restart of katello/foreman.

It's a new installation of katello 13.5 which I have updated to 3.15.1.1 recently. But the problem persists.

History

#1 Updated by Jonathon Turel almost 2 years ago

  • Status changed from New to Need more information

Next time this happens can you try restarting just the pulp_resource_manager and pulp_workers services? If that fixes the problem, that'll help us narrow down the cause.

#2 Updated by Gerald Vogt almost 2 years ago

  • Found in Releases Katello 3.15.1 added
  • Found in Releases deleted (Katello 3.15.1.1)

O.K. Tonight it happened again. The sync task for CentOS 8 PowerTools is hanging for a couple of hours now.

As requested I restarted the services:

# systemctl restart pulp_resource_manager.service pulp_worker*.service

However, the hanging task still shows in foreman. So no change here.

What I have noticed, though I can't tell for sure: I have two sync plans (one each hour, one each four hours). They are actually start at the same minute 53. I think, the hanging task only happens at times when both sync plans start. Tonight it was at 12:53 which was start time for both plans. But I am not 100% sure about this. I know a couple of hangs on EPEL syncs which are on my 4 hour plan. The CentOS 8 PowerTools is on the hourly but than hang was again at the time when the four plan was also running.

So maybe it's an issue when sync task runs overlap?

#3 Updated by Gerald Vogt almost 2 years ago

  • Found in Releases Katello 3.15.1.1 added

#4 Updated by Chris Roberts almost 2 years ago

  • Status changed from Need more information to New

#5 Updated by Chris Roberts almost 2 years ago

  • Assignee set to Partha Aji

#6 Updated by Gerald Vogt almost 2 years ago

Tonight it happened again during the hourly sync plan. The four hour sync plan was not running at that time. Thus, it also happens when there is only a single sync plan running.

Do you need more information?

#7 Updated by James Jeffers almost 2 years ago

  • Triaged changed from No to Yes
  • Status changed from New to Closed

#8 Updated by Jonathon Turel almost 2 years ago

  • Target version set to Katello Recycle Bin

Also available in: Atom PDF