Project

General

Profile

Bug #26821

PG::TRDeadlockDetected: ERROR: deadlock detected when deleting an Organization

Added by Evgeni Golov 5 months ago. Updated 3 months ago.

Status:
Closed
Priority:
High
Category:
Organizations and Locations
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

From time to time, I see the following error in our nightly pipelines:

not ok 1 Delete an Organization
# (in test file fb-destroy-organization.bats, line 15)
#   `hammer organization delete --name="${ORGANIZATION}"' failed with status 70
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 planned: 0.0/1, 0%, elapsed: 00:00:00
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.021739130434782608/1, 2%, 0.0/s, elapsed: 00:00:02, ETA: 00:01:55
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.16304347826086957/1, 16%, 0.0/s, elapsed: 00:00:05, ETA: 00:00:28
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.2391304347826087/1, 23%, 0.0/s, elapsed: 00:00:07, ETA: 00:00:25
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.32608695652173914/1, 32%, 0.0/s, elapsed: 00:00:09, ETA: 00:00:20
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.4782608695652174/1, 47%, 0.0/s, elapsed: 00:00:12, ETA: 00:00:12
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 running: 0.6739130434782609/1, 67%, 0.0/s, elapsed: 00:00:14, ETA: 00:00:06
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 error: 0.75/1, 75%, 0.1/s, elapsed: 00:00:16, ETA: 00:00:04
# Task fdb90a4d-89fc-4509-9bee-b87f7fbb5e89 error: 0.75/1, 75%, 0.1/s, elapsed: 00:00:16, ETA: 00:00:04
# Error: PG::TRDeadlockDetected: ERROR:  deadlock detected
# DETAIL:  Process 20568 waits for ShareLock on transaction 18434; blocked by process 23406.
# Process 23406 waits for ShareLock on transaction 18406; blocked by process 20568.
# HINT:  See server log for query details.
# : DELETE FROM "katello_pool_products" WHERE "katello_pool_products"."id" = $1

Related issues

Related to Katello - Bug #25711: org delete sometimes fails on product/destroy.rbClosed

Associated revisions

Revision b20f298d (diff)
Added by Justin Sherrill 3 months ago

Fixes #26821 - move cp owner delete to finalize phase

We are currently hitting a deadlock sometimes while
deleting an organization. The current understanding is that
the following is occuring:

1. in run phase: delete org in candlepin
2. in finalize: delete product AR models and all their associations
3. in finalize: delete the org AR model and all their associations

As part of 1), messages are sent to the candlepin event queue that are
picked up and handled asyncronously. This asyncronous task will
try to delete a specific pool and if that pools product is being deleted
in 2) there is the possibility of a deadlock.

To work around this, lets reorder things a little to be like this:

1. in finalize phase: delete the org in katello
2. in finalize phase delete the org in candlepin

this delays all the messages about pool deletion until after
the AR relations are destroyed

In addition i hit an error trying to delete RH content in candlepin
'content ID is locked'. This happens when trying to delete rh content
from manifests. With candlepin 2.0 we don't need to remove the content
since the owner delete will handle it

History

#1 Updated by Evgeni Golov 5 months ago

  • Related to Bug #25711: org delete sometimes fails on product/destroy.rb added

#2 Updated by Evgeni Golov 5 months ago

Yes, this also happens after #25711 :(

#3 Updated by John Mitsch 5 months ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.13.0
  • Category set to Organizations and Locations

#4 Updated by The Foreman Bot 4 months ago

  • Assignee set to Justin Sherrill
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/8185 added

#5 Updated by Justin Sherrill 3 months ago

  • Status changed from Ready For Testing to Closed

#6 Updated by Evgeni Golov 3 months ago

  • Bugzilla link set to 1726683

Also available in: Atom PDF