Bug #37879
closedFull Capsule Sync always trigger pulp optimised sync
Description
Description of problem:
Trying to fix corrupted metadata of a repository on Capsule, an attempt to run Full Capsule Sync does not help. The reason is Pulp sync task returns "Skipping Sync (no change from previous sync)" - and this is caused by the fact katello always call `RepositoriesRpmApi.sync` endpoint with `"optimize":true` - any Capsule sync calls optimised pulp repos sync, in fact. That is caused by:
https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/capsule_content/sync.rb#L16-L17
That needs to be:
{noblock}
sync_options[:optimize] = false if input[:options][:skip_metadata_check]
output[:pulp_tasks] = repo.backend_service(smart_proxy).with_mirror_adapter.sync(sync_options)
{noblock}
How reproducible:
100%
{*}Is this issue a regression from an earlier version:{*}
Yes, a regression from 6.9
Steps to Reproduce:
1. Have one CV promoted to LE that is associated to a Capsule and synced there. Ideally have just one pulp repository on the Capsule, to simplify the next step.
2. Manually corrupt some metadata on filesystem on the Capsule - e.g. via `file $(find /var/lib/pulp/media/artifact/ -type f)`, identify `primary.xml` file and remove it.
3. Have a Content Host registered via the Capsule and enabled the newly-broken repo. Run there e.g. `dnf update` to confirm metadata are missing. Check `/var/log/messages` on Capsule for the details.
4. Run Full Capsule Sync
5. Once it completes, try `dnf update` again.
Actual behavior:
3. `dnf update` fails with something like:
{noformat}
zoo_repo 304 B/s | 55 B 00:00
Errors during downloading metadata for repository 'RedHat_zoo_product_zoo_repo':
- Status code: 500 for https://pmoravec-caps615.example.com/pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml (IP: 1.2.3.4)
Error: Failed to download metadata for repo 'RedHat_zoo_product_zoo_repo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
{noformat}
`/var/log/messages` on Capsule shows:
{noformat}
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: [2024-10-04 08:58:55 +0000] [2043] [ERROR] Error handling request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: Traceback (most recent call last):
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib64/python3.11/site-packages/aiohttp/web_protocol.py", line 452, in handle_request
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: resp = await request_handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib64/python3.11/site-packages/aiohttp/web_app.py", line 543, in _handle
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: resp = await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib64/python3.11/site-packages/aiohttp/web_middlewares.py", line 114, in impl
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: return await handler(request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: response = await self.make_entry(
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: response = await handler(*args, **kwargs)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 249, in stream_content
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: return await self._match_and_stream(path, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 610, in _match_and_stream
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: return await self._serve_content_artifact(ca, headers, request)
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: File "/usr/lib/python3.11/site-packages/pulpcore/content/handler.py", line 918, in _serve_content_artifact
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: raise Exception(("Expected path '{}' is not found").format(path))
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: Exception: Expected path '/var/lib/pulp/media/artifact/3b/bd4af9a5cac6195a2b138e9754df0c3d8acf7cdf34ed736bc313342409aa45' is not found
Oct 4 10:58:55 pmoravec-caps615 pulpcore-content2043: [04/Oct/2024:08:58:55 +0000] "GET /pulp/content/RedHat/RECYCLE/cv_zoo/custom/zoo_product/zoo_repo/repodata/repomd.xml HTTP/1.1" 500 245 "-" "libdnf (Red Hat Enterprise Linux 8.0; generic; Linux.x86_64)"
{noformat}
So far, so good.
BUT the same persists also after Full Capsule Sync - that will have "Skipping Sync (no change from previous sync)" message in the pulp task details (see e.g. in Actions::Pulp3::CapsuleContent::Sync dynflow step of the task). Step 5 will fail the same way..
{*}Expected behavior:{*}
4. Full Caps Sync should not skip syncing the repo in Actions::Pulp3::CapsuleContent::Sync step.
5. `dnf update` should not fail after the Full Caps Sync
{*}Business Impact / Additional info:{*}
This prolongs resolution of the problems with corrupted /var/lib/pulp content where we had to manually remove or fix individual repos instead of simply clicking to Full Caps Sync.
I will prepare a PR for this later today.
Updated by The Foreman Bot about 1 month ago
- Status changed from New to Ready For Testing
- Assignee set to Pavel Moravec
- Pull request https://github.com/Katello/katello/pull/11168 added
Updated by Jeremy Lenz about 1 month ago
- Target version changed from Katello 4.14.1 to Katello 4.13.2
- Triaged changed from No to Yes
Updated by Pavel Moravec 14 days ago
- Status changed from Ready For Testing to Closed
Applied in changeset katello|27741e86740daf2c729f65a429cf569f52a8d5ae.