Project

General

Profile

Bug #16170

Unregistering a Content Host can pause ListenOnCandlepinEvents with Candlepin::Consumer: 410 Gone error

Added by Tomáš Strachota almost 3 years ago. Updated 11 months ago.

Status:
Closed
Priority:
High
Category:
Hosts
Target version:
Difficulty:
easy
Triaged:
Yes
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

When unregistering a content host (either by subscription-manager unregister, or deleting Content Host, or deleting Host), Actions::Katello::System::Destroy task is being processed.

This task has a concurrency bug that with some probability causes Actions::Candlepin::ListenOnCandlepinEvents task to be paused with error:

Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit b3012a90-41b6-4788-98a9-5b41839b6dca has been deleted","requestUuid":"04dbfeda-75cb-4ffa-ad9f-9f4c818fc868","deletedId":"b3012a90-41b6-4788-98a9-5b41839b6dca"} (GET /candlepin/consumers/b3012a90-41b6-4788-98a9-5b41839b6dca)

(UUID matches the UUID of just being deleted Content Host)

sequence of steps leading to the bug:

- Actions::Candlepin::Consumer::Destroy sub-task executed
  - it deletes the consumer from candlepin and recalculates compliance for it
  - it announces the compliance.create event to ListenOnCandlepinEvents task
  - katello finds the katello_system is present so it runs "reindex_consumer" 
- the parent task Actions::Katello::System::Destroy even _now_ enters Actions::Katello::System::Destroy
  - this subtask deletes the system from katello_systems _after_ the check before "re-indexing content host .." is made, so the re-index is _not_ skipped
- the re-index of the content host calls GET consumer/<uuid> on candlepin, what triggers the 410 Gone in ListenOnCandlepinEvents task

Steps to Reproduce:
1. tail -f /var/log/foreman/production.log | grep -e "in phase Finalize Actions::Katello::System::Destroy" -e "skip re-indexing of non-existent content host" -e "re-indexing content host"

2. Have opened Actions::Candlepin::ListenOnCandlepinEvents task in WebUI

3. On some Content Host, register and unregister it in a loop (here RHEL7 used, update if using RHEL5 or 6 accordingly):

while true; do
  subscription-manager register --force --org="Default_Organization" --environment="Library" --username=admin --password=faYakexMm5XN543x
  subscription-manager subscribe --pool=8aa2d415526494380152732fc8d20dd7
  subscription-manager repos --enable rhel-7-server-rpms --enable rhel-7-server-satellite-tools-6.1-rpms
  date
  subscription-manager unregister
  date
  sleep 5
done

4. monitor the tail -f output and ListenOnCandlepinEvents task

Associated revisions

Revision 883c8066 (diff)
Added by Tomáš Strachota almost 3 years ago

Fixes #16170 - consumer gone error in ListenOnCandlepinEvents (#6258)

History

#1 Updated by Tomáš Strachota almost 3 years ago

  • Subject changed from unregistering a Content Host can pause ListenOnCandlepinEvents with Candlepin::Consumer: 410 Gone error to Unregistering a Content Host can pause ListenOnCandlepinEvents with Candlepin::Consumer: 410 Gone error
  • Description updated (diff)

#2 Updated by Tomáš Strachota almost 3 years ago

  • Description updated (diff)
  • Status changed from New to Assigned

#3 Updated by Tomáš Strachota almost 3 years ago

  • Description updated (diff)
  • Status changed from Assigned to Needs design

#4 Updated by Tomáš Strachota almost 3 years ago

Note that the issue was reported against older version of Katello that used elasticsearch. Elasticsearch is gone now and the code has been moved elsewhere but the root of the problem persists. It is still possible that an event form Candlepin comes when the host is already unregistered.

app/lib/actions/candlepin/import_pool_handler.rb:

def reindex_consumer(message)
  if message.content['newEntity']
    uuid = JSON.parse(message.content['newEntity'])['consumer']['uuid']
    subscription_facet = ::Katello::Host::SubscriptionFacet.find_by_uuid(uuid)
    if subscription_facet.nil?
      @logger.debug "skip re-indexing of non-existent registered host #{uuid}" 
    else
      @logger.debug "re-indexing content host #{subscription_facet.host.name}" 
      subscription_facet.update_subscription_status
    end
  end
end

#5 Updated by The Foreman Bot almost 3 years ago

  • Status changed from Needs design to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/6258 added

#6 Updated by Tomáš Strachota almost 3 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#7 Updated by Justin Sherrill almost 3 years ago

  • Category set to Hosts
  • Legacy Backlogs Release (now unused) set to 162
  • Difficulty set to easy

#8 Updated by Marek Hulán over 2 years ago

  • Target version set to 1.6.0

Also available in: Atom PDF