Plugins » Katello

Description of problem:

After replacing custom certificates on Satellite by new ones, signed by a new CA (not previously trusted) capsule is unable to fetch on-demand content from Satellite and complains about self signed certificate in certificate chain.

Version-Release number of selected component (if applicable):

Satellite 6.11

How reproducible:

Steps to Reproduce:
1. Have a Satellite + Capsule deployed ( using default self-signed certificates).
2. Ensure capsule is configured with on-demand download policy and some CV are synced to it
3. Deploy custom certs on Satellite + Capsule
4. Try to install, in a client registered to the capsule, any content not yet locally downloaded. You should see errors like this on the capsule logs:

~_{~
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')])
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')])
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: [2022-07-28 15:36:37 +0000] [2666] [ERROR] Error handling request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Traceback (most recent call last):
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 1050, in create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: transport, protocol = await self._create_connection_transport(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: await waiter
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/sslproto.py", line 529, in data_received
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ssldata, appdata = self._sslpipe.feed_ssldata(data)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._sslobj.do_handshake()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 944, in do_handshake
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._sslobj.do_handshake()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: The above exception was the direct cause of the following exception:
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Traceback (most recent call last):
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: resp = await request_handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: resp = await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 173, in cached_function
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await self.make_entry(key, bk, func, args, self.default_expires_ttl)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 202, in make_entry
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await handler(*args)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 203, in stream_content
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._match_and_stream(path, request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 514, in _match_and_stream
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._stream_content_artifact(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 656, in _stream_content_artifact
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await self._stream_remote_artifact(request, response, remote_artifact)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 868, in _stream_remote_artifact
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: download_result = await downloader.run()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 271, in run
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await download_wrapper()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ret = await target(*args, **kwargs)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._run(extra_data=extra_data)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 114, in _run
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: async with self.session.get(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client.py", line 1138, in aenter
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._resp = await self._coro
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client.py", line 535, in _request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: conn = await self._connector.connect(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 542, in connect
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: proto = await self._create_connection(req, traces, timeout)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 907, in _create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: _, proto = await self._create_direct_connection(req, traces, timeout)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: raise last_exc
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: transp, proto = await self._wrap_create_connection(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: raise ClientConnectorCertificateError(req.connection_key, exc) from exc
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')]
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: [28/Jul/2022:15:36:27 +0000] "GET /pulp/content/Default_Organization/Dev/Test1/content/dist/rhel/server/7/7Server/x86_64/ansible/2.9/os/Packages/s/sshpass-1.06-2.el7.x86_64.rpm HTTP/1.1" 500 244 "-" "urlgrabber/3.10 yum/3.4.3"}
~~

Actual results:

Fail to download content. Return HTTP 500 to client.

Expected results:

Successfully download the content.

Additional info:

The problem is that the ca_cert on the remote (inside pulp3) didn't get updated after changing the certificate.

Running an optimized sync of the capsule fix it and content can be downloaded again.