Bug #23055
closed
Dynflow SSL Issue when using custom SSL Certificate
Description
This would be my first ticket, so please let me know if I'm not doing something correctly.
Some background first:
We were using Foreman version 1.15.6 with Remote SSH Execution enabled. The SSL for the Web UI was set with our domain's Wildcard SSL Certificate and all was working just perfectly.
When we tried setting up a client that had Puppet version 5 we couldn't get the agent to run because of the puppet version conflict.
So we did the following,
1. Upgraded puppet from 3 to 4
2. Upgraded foreman from 1.15.6 to 1.16
3. Upgraded puppet from 4 to 5
(followed the instructions from the foreman site for each step)
Everything seemed to be working well; the smart proxy has a green checkmark, no warning logs and the clients are in-sync.
However when trying to issue a remote shell script, the script waits for a minute or two and then fails.
But when I look at the sub-task, the command does get executed. (I tested an scp command and it was successfully transferred)
The sub-task result displays as follows:
Failed to initialize: Dynflow::ExecutionPlan::Steps::Error - ERF42-3325 [Foreman::Exception]: The smart proxy task fa31b59e-0aac-484e-a8d8-c8078aee9a7e failed.
I then looked at foreman-proxy/proxy.log and found the following ERROR lines:
ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)
Before this line, the SSL certificate that was displayed was the Puppet Cert.
To check, I've changed the apache SSL setting to use Puppet CA and executed the Remote Shell script again, and it worked.
So something must have changed. I tested the following to see if it works, but no luck.
1. Changed back Websockets SSL to Puppet Cert (Since I've changed this to Custom SSL based on the instructions for setting custom certs)
2. Uncommented and entered the :foreman_ssl_ca: lines in foreman-proxy/settings.d since it states that this is "Mainly useful when Foreman uses different certificates for its web UI and for smart-proxy requests."
3. Did a mix of Puppet/Custom SSL certs for :foreman_ssl_ca: lines and :ssl_ca_file: combination since it didn't exactly state which is for the web UI and which is for the smart-proxy.
4. Also tried only changing the :foreman_ssl_cert: and :foreman_ssl_key:, leaving :foreman_ssl_ca: as the Puppet CA since the ca file is the same in apache settings also.
I restarted the foreman-proxy service each time I tried, but the only change I got from doing the above was that the task immediately failed and showed that "There are no foreman proxies available". (Or something similar, I forgot to make a note of the exact message for this one)
So it looks to me that I can't change the Web UI's SSL and also use Remote SSH Execution at the same time.
If you need more information, please let me know.
Thank you.
Updated by TJ Hamamoto over 6 years ago
Just to update, still seems to be the case for 1.16.1.
Also wanted to add that I've applied the workaround to downgrade sequel gem to 4.15.0 and pg gem to 0.17.1 as indicated here: https://projects.theforeman.org/issues/20050
And yes I'm running foreman on Ubuntu 16.04; sorry I should have pointed this out in the beginning.
Updated by Tatsuya Hamamoto over 6 years ago
*Sorry I got a different account
Still seems to be the case for 1.17.0, however I did get the 'sequel' and 'pg' gem downgrade fixed by following the instructions from Ivan here: https://projects.theforeman.org/issues/20050
If someone has any suggestions on how to fix the SSL error, that will be great.
Best regards,
TJ
Updated by Iain Walmsley over 6 years ago
Will depend on your setup and like you say the apache settings you have. For a typical setep you'll need the root/chain certs for the web UI. Depending on whether
Hence it'll look something like this;
:foreman_ssl_cert: PUPPET_CERT
:foreman_ssl_key: PUPPET_KEY
:foreman_ssl_ca: WEB_UI_CHAIN
Updated by Tatsuya Hamamoto over 6 years ago
Hi Iain,
Thank you very much for your comment; I was beginning to feel lonely here. :)
I went ahead and tried un-commenting and changing the three parameters in foreman-proxy/settings.yml (cert and key for puppet cert, and ca bundle for web UI), and restarted the foreman-proxy service.
The proxy server has a green checkmark and I do not get any SSL errors when logging in, but the remote execution task still fails with the following showing up in the logs.
ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)
If you have anything that you would like me to test, please let me know.
Thank you again.
Updated by Tatsuya Hamamoto over 6 years ago
- Translation missing: en.field_release set to 297
Just to update, still seems to be the case for 1.17.1.
Updated by Tomer Brisker over 6 years ago
- Translation missing: en.field_release changed from 297 to 360
Updated by Tomer Brisker over 6 years ago
- Project changed from Foreman to Foreman Remote Execution
- Category changed from Smart Proxy to Smart Proxy Dynflow
Updated by Tatsuya Hamamoto over 6 years ago
- Triaged set to No
Just to update, still seems to be the case for 1.18.1.
Updated by Peter J almost 6 years ago
Issue still present in 1.20.2
Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pem
to
:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.key
based on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.
Updated by Elias Pereira over 5 years ago
Peter J wrote:
Issue still present in 1.20.2
Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pemto
:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.keybased on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.
Still present in version 1.21.3.
@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml
Maybe the configuration file changed from version 1.20.2 to 1.21.3?
Updated by Peter J over 5 years ago
Elias Pereira wrote:
Peter J wrote:
Issue still present in 1.20.2
Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pemto
:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.keybased on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.
Still present in version 1.21.3.
@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml
Maybe the configuration file changed from version 1.20.2 to 1.21.3?
The file should be on your Smart Proxy host.
Updated by Elias Pereira over 5 years ago
Peter J wrote:
Elias Pereira wrote:
Peter J wrote:
Issue still present in 1.20.2
Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pemto
:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.keybased on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.
Still present in version 1.21.3.
@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml
Maybe the configuration file changed from version 1.20.2 to 1.21.3?
The file should be on your Smart Proxy host.
The only file that my server has is:
- find / -iname settings.yml
/etc/foreman-proxy/settings.yml
It would be the same file?
Updated by Adam Ruzicka over 3 years ago
- Status changed from New to Rejected
Running smart proxy dynflow core as a standalone service is being deprecated starting with Foreman 2.5. No point fixing something that will get dropped.