Project

General

Profile

Bug #16620

custom certificates do not work out-of-the box on katello 3.1

Added by Daniel Kimsey almost 3 years ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
Category:
Installer
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Using custom certs reveal a few issues in the katello/foreman stack. Out of the box it is not possible to use custom certs without workarounds.

This ticket summaries the issues laid out in a foreman-users thread

katello-default-ca not trusted on master box, breaking reverse proxy behavior

The katello-server-ca certificate is added to the system store (expected). However when a custom server cert is used and therefore not equal to the self-signed, the self-signed cert is not included in the system trust. This breaks candlepin setup as well as the httpd proxy for rhsm. Please see the IRC thread
The foreman-installer can't register with itself since it's using custom certs. Since it can't pre-load the certs into the CA bundle because the internal-ca doesn't exist until you run foreman-installer.

RestClient::SSLCertificateNotVerified
Katello::Resources::Candlepin::OwnerInfo: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (GET /candlepin/owners/acme/info)

Resolution

Copy /root/ssl-build/katello-default-ca.crt to /etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean install using custom certs. After performing these steps, re-run the installer. It should complete correctly the second time through.

foreman-installer smart proxy/capsule fails, the foreman master has the incorrect CA in proxy_ca.pem.

The proxy_ca.pem should be the self-signed CA, but it is incorrectly the custom cert chain. Resolution found in an IRC conversation .

Error from installer:

[ERROR 2016-09-19 10:54:36 verbose]  Proxy capsule.tw-test.net cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy https://capsule.acme.com:9090/features Please check the proxy is configured and running on the host.
[ INFO 2016-09-19 10:54:36 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'

Error in capsule's proxy:

E, [2016-09-19T11:33:26.811258 #9849] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
        /usr/share/ruby/openssl/ssl.rb:226:in `accept'

Resolution

Swap the incorrect custom cert chain in /etc/foreman/proxy_ca.pem with the contents of /root/ssl-build/katello-default-ca.crt. Restart proxy, this resolves the issue.

Additional Observations

These are some thing I personally noticed and wanted to mention, they were not discussed or mentioned in the referenced foreman-users discussion thread.

1. A number of the master's components (for instance the candlepin and foreman-proxy) are using the self-signed CA as their server certificate. Server certs should be cut from the CA for these components. This also reduces the surface or number of components that have access to the CA key, as the only time it is being read is during the capsule-cert-generate stage. I feel having these components have an identical structure to the capsules makes this more clear. But also feeds into the next bullet point.
2. The custom cert chain option requires inclusion of the CA and is attached to httpd as the chain file. However it is common practice to /not/ include the CA as the client already has a copy of the CA. (As there is no trust w/o the client having the CA through out-of-bands means.) Only the intermediates should be included in the chain. The current design means httpd is passing along the CA (meh, whatever, bytes are 'free') but also that when distributed to the clients the intermediate is included. The chain file is being added to the trust store but it contains N certificates, it is not obvious on the client side that multiple certs are being included. Furthermore this complicates certificate renewal as the chain may change but the root CA will likely not. Summary: intermediates are only for servers, CA is for clients.

History

#1 Updated by Eric Helms almost 3 years ago

  • Assignee set to Eric Helms
  • Legacy Backlogs Release (now unused) set to 114

#2 Updated by Daniel Kimsey almost 3 years ago

I'm still debugging a related issue regarding custom certs on the capsule. But it appears the installer is setting the SSLCACertificatePath in 05-pulp-https.conf to the system's certs directory. I believe this should be pointed at the katello-default-ca.crt.

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt" 
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key" 
  SSLCACertificatePath    "/etc/pki/tls/certs" 
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt" 

I noticed this as I was trying to browse to the /pub path in my browser and kept getting prompted to provide a client certificate even though I have none installed from the katello ca.

#3 Updated by Daniel Kimsey almost 3 years ago

Follow up on invalid proxy-ca.pem

This appears to be a common issue in the installer (to both katello and capsule scenarios). The capsule also fails to install with a similar error message. But it lacks the proxy-ca.pem.

[ INFO 2016-09-23 11:26:55 verbose]  Class[Foreman_proxy::Register]: Scheduling refresh of Foreman_smartproxy[capsule.acme.com]
[ERROR 2016-09-23 11:26:55 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[capsule.acme.com]: Failed to call refresh: Proxy capsule.acme.com cannot be refreshed: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy https://capsule.acme.com:9090/features Please check the proxy is configured and running on the host.

When I connect to this port with @openssl s_client@ I see the custom certificate being returned.

Instead swapping out foreman_ssl_ca.pem which contains my custom CA with the katello-default-ca seems to have addressed the issue.

#4 Updated by Daniel Kimsey almost 3 years ago

To correctly debug all of this, the server cert in the default situation (self-signed) should be created from a completely separate ca than katello-default-ca. This would reveal the issues with custom certs in the non-custom situation since they'd be following the same logical setup.

In other words, in the completely default install katello-server-ca == katello-default-ca. I suggest the default install make these !=, that'd make errors like these regarding custom certs very obvious.

Daniel Kimsey wrote:

the installer is setting the SSLCACertificatePath in 05-pulp-https.conf to the system's certs directory

Quick update, nor is it setting the SSLCertificateChainFile. So it is not returning a valid certificate chain to clients.

#5 Updated by Daniel Kimsey over 2 years ago

Daniel Kimsey wrote:

Follow up on invalid proxy-ca.pem
....
When I connect to this port with openssl s_client I see the custom certificate being returned.

Instead swapping out foreman_ssl_ca.pem which contains my custom CA with the katello-default-ca seems to have addressed the issue.

However this breaks the katello's self-proxy which also breaks puppet registration. Do not recommend this solution.

#6 Updated by Daniel Kimsey over 2 years ago

The certificates setup on the capsules in /etc/httpd/conf.d/03-crane.conf are also incorrect. They contain the custom certificate with the default chain. This breaks any docker client having the correct CAs installed.

  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt" 
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key" 
  SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt" 
  SSLCACertificatePath    "/etc/pki/tls/certs" 
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt" 

Setting the SSLCertificateChainFile on the capsules to the .../katello-server-ca.crt seems to address it. But I'm certain the installer will overwrite this at some point.

The main server configuration for port 5000 returns the cert chain correctly.

#7 Updated by Daniel Kimsey over 2 years ago

I'd also like to note the SSLCertificateCAPath setting in many of the httpd configurations mean any web client that acutally has a client cert will get a pop-up from their browser asking which cert to submit.

I've had a few users complain about this. The client auth configuration should probably be restricted to just the katello-default-ca and corresponding capsules.

Also available in: Atom PDF