Plugins » Katello

Description of problem:
Trying to fix corrupted metadata of a repository on Capsule, an attempt to run Full Capsule Sync does not help. The reason is Pulp sync task returns "Skipping Sync (no change from previous sync)" - and this is caused by the fact katello always call `RepositoriesRpmApi.sync` endpoint with `"optimize":true` - any Capsule sync calls optimised pulp repos sync, in fact. That is caused by:

https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/capsule_content/sync.rb#L16-L17

That needs to be:

{noblock}
sync_options[:optimize] = false if input[:options][:skip_metadata_check]
output[:pulp_tasks] = repo.backend_service(smart_proxy).with_mirror_adapter.sync(sync_options) {noblock}
 

How reproducible:
100%
 

{*}Is this issue a regression from an earlier version:{*}
Yes, a regression from 6.9
 

Steps to Reproduce:
1. Have one CV promoted to LE that is associated to a Capsule and synced there. Ideally have just one pulp repository on the Capsule, to simplify the next step.
2. Manually corrupt some metadata on filesystem on the Capsule - e.g. via `file $(find /var/lib/pulp/media/artifact/ -type f)`, identify `primary.xml` file and remove it.
3. Have a Content Host registered via the Capsule and enabled the newly-broken repo. Run there e.g. `dnf update` to confirm metadata are missing. Check `/var/log/messages` on Capsule for the details.
4. Run Full Capsule Sync
5. Once it completes, try `dnf update` again.

Actual behavior:
3. `dnf update` fails with something like:

{noformat}
zoo_repo 304 B/s | 55 B 00:00
Errors during downloading metadata for repository 'RedHat_zoo_product_zoo_repo':
- Status code: 500 for https://pmoravec-caps615.example.com/pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml (IP: 1.2.3.4)
Error: Failed to download metadata for repo 'RedHat_zoo_product_zoo_repo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried {noformat}

`/var/log/messages` on Capsule shows:

{noformat}
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: [2024-10-04 08:58:55 +0000] [2043] [ERROR] Error handling request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: Traceback (most recent call last):
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib64/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in handle_request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: resp = await request_handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: resp = await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib64/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: response = await self.make_entry(
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: response = await handler(*args, **kwargs)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 249, in stream_content
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: return await self._match_and_stream(path, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 610, in _match_and_stream
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: return await self._serve_content_artifact(ca, headers, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: ^^{^^}^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 918, in _serve_content_artifact
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: raise Exception(("Expected path '{}' is not found").format(path))
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: Exception: Expected path '/var/lib/pulp/media/artifact/3b/bd4af9a5cac6195a2b138e9754df0c3d8acf7cdf34ed736bc313342409aa45' is not found
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content²⁰⁴³: [04/Oct/2024:08:58:55 +0000] "GET /pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml HTTP/1.1" 500 245 "-" "libdnf (Red Hat Enterprise Linux 8.0; generic; Linux.x86_64)" {noformat}

So far, so good.

BUT the same persists also after Full Capsule Sync - that will have "Skipping Sync (no change from previous sync)" message in the pulp task details (see e.g. in Actions::Pulp3::CapsuleContent::Sync dynflow step of the task). Step 5 will fail the same way..

{*}Expected behavior:{*}
4. Full Caps Sync should not skip syncing the repo in Actions::Pulp3::CapsuleContent::Sync step.
5. `dnf update` should not fail after the Full Caps Sync

{*}Business Impact / Additional info:{*}
This prolongs resolution of the problems with corrupted /var/lib/pulp content where we had to manually remove or fix individual repos instead of simply clicking to Full Caps Sync.

I will prepare a PR for this later today.