Bug #16620
closed
custom certificates do not work out-of-the box on katello 3.1
Description
Using custom certs reveal a few issues in the katello/foreman stack. Out of the box it is not possible to use custom certs without workarounds.
This ticket summaries the issues laid out in a foreman-users thread
katello-default-ca not trusted on master box, breaking reverse proxy behavior¶
The katello-server-ca certificate is added to the system store (expected). However when a custom server cert is used and therefore not equal to the self-signed, the self-signed cert is not included in the system trust. This breaks candlepin setup as well as the httpd proxy for rhsm. Please see the IRC thread
The foreman-installer can't register with itself since it's using custom certs. Since it can't pre-load the certs into the CA bundle because the internal-ca doesn't exist until you run foreman-installer.
RestClient::SSLCertificateNotVerified Katello::Resources::Candlepin::OwnerInfo: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (GET /candlepin/owners/acme/info)
Resolution¶
Copy /root/ssl-build/katello-default-ca.crt to /etc/pki/ca-trust/source/anchors/ and rebuild the openssl ca certs with update-ca-trust. Due to chicken-and-egg issue, this may prevent a clean install using custom certs. After performing these steps, re-run the installer. It should complete correctly the second time through.
foreman-installer smart proxy/capsule fails, the foreman master has the incorrect CA in proxy_ca.pem.¶
The proxy_ca.pem should be the self-signed CA, but it is incorrectly the custom cert chain. Resolution found in an IRC conversation .
Error from installer:
[ERROR 2016-09-19 10:54:36 verbose] Proxy capsule.tw-test.net cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy https://capsule.acme.com:9090/features Please check the proxy is configured and running on the host. [ INFO 2016-09-19 10:54:36 verbose] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
Error in capsule's proxy:
E, [2016-09-19T11:33:26.811258 #9849] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
/usr/share/ruby/openssl/ssl.rb:226:in `accept'
Resolution¶
Swap the incorrect custom cert chain in /etc/foreman/proxy_ca.pem with the contents of /root/ssl-build/katello-default-ca.crt. Restart proxy, this resolves the issue.
Additional Observations¶
These are some thing I personally noticed and wanted to mention, they were not discussed or mentioned in the referenced foreman-users discussion thread.
1. A number of the master's components (for instance the candlepin and foreman-proxy) are using the self-signed CA as their server certificate. Server certs should be cut from the CA for these components. This also reduces the surface or number of components that have access to the CA key, as the only time it is being read is during the capsule-cert-generate stage. I feel having these components have an identical structure to the capsules makes this more clear. But also feeds into the next bullet point.
2. The custom cert chain option requires inclusion of the CA and is attached to httpd as the chain file. However it is common practice to /not/ include the CA as the client already has a copy of the CA. (As there is no trust w/o the client having the CA through out-of-bands means.) Only the intermediates should be included in the chain. The current design means httpd is passing along the CA (meh, whatever, bytes are 'free') but also that when distributed to the clients the intermediate is included. The chain file is being added to the trust store but it contains N certificates, it is not obvious on the client side that multiple certs are being included. Furthermore this complicates certificate renewal as the chain may change but the root CA will likely not. Summary: intermediates are only for servers, CA is for clients.
Updated by 
Updated by 
Updated by 
Updated by