Plugins » Katello

Hi there,

Pulp somewhat recently stopped users from deleting repository versions that are distributed via Pulp distributions. Katello is supposed to know all repository versions that are being distributed, and so this bug technically should not be hit unless a task fails or if Pulp is being used outside of Katello (like for hosting your own separate repos). Katello assumes it has complete control of Pulp.

With that said, to help you out, we can assist in finding the extra distribution that you need to delete. Then orphan cleanup should work just fine.

1. Open up the foreman console via `foreman-rake console`.
2. Run

PulpcoreClient::DistributionsApi.new(PulpcoreClient::ApiClient.new(SmartProxy.pulp_primary.pulp3_configuration(PulpcoreClient::Configuration))).list(per_page: 1000).results.map(&:pulp_href) - ::Katello::Pulp3::DistributionReference.select(:href).pluck(:href)

-> The output should be all of the Pulp distributions that Katello does not know. There is likely one that looks like `/pulp/api/v3/distributions/core/artifacts/.../` that shouldn't be deleted since it's for contentguard, which should not be causing your error here.

4. Inspect each href using `sudo pulp show --href <href>` to make sure it doesn't look like anything you need.
5. If it looks extra, delete it:

sudo pulp --force <rpm/container/file/etc> destroy --href <href>

6. Repeat step (5) for any remaining hrefs that Katello does not know about.

If you need more help, I might suggest we take this conversation to https://community.theforeman.org/tags/c/support/10/katello

We decided to close this Redmine out due to two weeks of needinfo. If you still need assistance with this, please feel free to open a support request.

Opening back up -- we realized that more users might hit this issue for a variety of reasons, especially on smart proxies since only truly orphaned distributions get deleted (which theoretically, with the latest Pulp changes, won't exist any more).

We need to look into how these extra distributions can be cleaned up on both the Katello server and on smart proxies. On the Katello server, we can theoretically delete all distributions that we don't see in our database. On smart proxies, since we don't track the distributions, we'll need to look into getting the list of distributed paths, looking up distributions from those paths, and deleting the rest.

It looks like this bug may be caused by smart proxies distributing out of date repository versions. In one reproducer analysis, the smart proxy was found to have a repo with 3 repository versions. However, version 2 was the one being distributed. So, orphan cleanup found version 3 and assumed it was being distributed since it was the latest, but this was not the case.

Therefore, anyone who hits this bug likely had some previous sync that failed to refresh the distributions for some reason. A complete sync of the affected repositories might fix the issue.