Bug #16170
Updated by Tomáš Strachota over 7 years ago
When unregistering a content host (either by subscription-manager unregister, or deleting Content Host, or deleting Host), Actions::Katello::System::Destroy task is being processed.
This task has a concurrency bug that with some probability causes Actions::Candlepin::ListenOnCandlepinEvents task to be paused with error:
Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit b3012a90-41b6-4788-98a9-5b41839b6dca has been deleted","requestUuid":"04dbfeda-75cb-4ffa-ad9f-9f4c818fc868","deletedId":"b3012a90-41b6-4788-98a9-5b41839b6dca"} (GET /candlepin/consumers/b3012a90-41b6-4788-98a9-5b41839b6dca)
(UUID matches the UUID of just being deleted Content Host)
sequence of steps leading to the bug:
- Actions::Candlepin::Consumer::Destroy sub-task executed
- it deletes the consumer from candlepin and recalculates compliance for it
- it announces the compliance.create event to ListenOnCandlepinEvents task
- katello finds the katello_system is present so it runs "reindex_consumer"
- the parent task Actions::Katello::System::Destroy even _now_ enters Actions::Katello::System::Destroy
- this subtask deletes the system from katello_systems _after_ the check before "re-indexing content host .." is made, so the re-index is _not_ skipped
- the re-index of the content host calls GET consumer/<uuid> on candlepin, what triggers the 410 Gone in ListenOnCandlepinEvents task
Steps to Reproduce:
1. tail -f /var/log/foreman/production.log | grep -e "in phase Finalize Actions::Katello::System::Destroy" -e "skip re-indexing of non-existent content host" -e "re-indexing content host"
2. Have opened Actions::Candlepin::ListenOnCandlepinEvents task in WebUI
3. On some Content Host, register and unregister it in a loop (here RHEL7 used, update if using RHEL5 or 6 accordingly):
<pre>
while true; do
subscription-manager register --force --org="Default_Organization" --environment="Library" --username=admin --password=faYakexMm5XN543x
subscription-manager subscribe --pool=8aa2d415526494380152732fc8d20dd7
subscription-manager repos --enable rhel-7-server-rpms --enable rhel-7-server-satellite-tools-6.1-rpms
date
subscription-manager unregister
date
sleep 5
done
</pre>
4. monitor the tail -f output and ListenOnCandlepinEvents task