Project

General

Profile

Actions

Bug #23055

closed

Dynflow SSL Issue when using custom SSL Certificate

Added by TJ Hamamoto almost 6 years ago. Updated almost 3 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Smart Proxy Dynflow
Target version:
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

This would be my first ticket, so please let me know if I'm not doing something correctly.

Some background first:
We were using Foreman version 1.15.6 with Remote SSH Execution enabled. The SSL for the Web UI was set with our domain's Wildcard SSL Certificate and all was working just perfectly.
When we tried setting up a client that had Puppet version 5 we couldn't get the agent to run because of the puppet version conflict.
So we did the following,

1. Upgraded puppet from 3 to 4
2. Upgraded foreman from 1.15.6 to 1.16
3. Upgraded puppet from 4 to 5
(followed the instructions from the foreman site for each step)

Everything seemed to be working well; the smart proxy has a green checkmark, no warning logs and the clients are in-sync.
However when trying to issue a remote shell script, the script waits for a minute or two and then fails.
But when I look at the sub-task, the command does get executed. (I tested an scp command and it was successfully transferred)
The sub-task result displays as follows:

Failed to initialize: Dynflow::ExecutionPlan::Steps::Error - ERF42-3325 [Foreman::Exception]: The smart proxy task fa31b59e-0aac-484e-a8d8-c8078aee9a7e failed.

I then looked at foreman-proxy/proxy.log and found the following ERROR lines:

ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)

Before this line, the SSL certificate that was displayed was the Puppet Cert.
To check, I've changed the apache SSL setting to use Puppet CA and executed the Remote Shell script again, and it worked.
So something must have changed. I tested the following to see if it works, but no luck.

1. Changed back Websockets SSL to Puppet Cert (Since I've changed this to Custom SSL based on the instructions for setting custom certs)
2. Uncommented and entered the :foreman_ssl_ca: lines in foreman-proxy/settings.d since it states that this is "Mainly useful when Foreman uses different certificates for its web UI and for smart-proxy requests."
3. Did a mix of Puppet/Custom SSL certs for :foreman_ssl_ca: lines and :ssl_ca_file: combination since it didn't exactly state which is for the web UI and which is for the smart-proxy.
4. Also tried only changing the :foreman_ssl_cert: and :foreman_ssl_key:, leaving :foreman_ssl_ca: as the Puppet CA since the ca file is the same in apache settings also.

I restarted the foreman-proxy service each time I tried, but the only change I got from doing the above was that the task immediately failed and showed that "There are no foreman proxies available". (Or something similar, I forgot to make a note of the exact message for this one)
So it looks to me that I can't change the Web UI's SSL and also use Remote SSH Execution at the same time.

If you need more information, please let me know.
Thank you.

Actions #1

Updated by TJ Hamamoto almost 6 years ago

Just to update, still seems to be the case for 1.16.1.

Also wanted to add that I've applied the workaround to downgrade sequel gem to 4.15.0 and pg gem to 0.17.1 as indicated here: https://projects.theforeman.org/issues/20050
And yes I'm running foreman on Ubuntu 16.04; sorry I should have pointed this out in the beginning.

Actions #2

Updated by Tatsuya Hamamoto almost 6 years ago

*Sorry I got a different account

Still seems to be the case for 1.17.0, however I did get the 'sequel' and 'pg' gem downgrade fixed by following the instructions from Ivan here: https://projects.theforeman.org/issues/20050
If someone has any suggestions on how to fix the SSL error, that will be great.
Best regards,

TJ

Actions #3

Updated by Iain Walmsley almost 6 years ago

Will depend on your setup and like you say the apache settings you have. For a typical setep you'll need the root/chain certs for the web UI. Depending on whether

Hence it'll look something like this;

:foreman_ssl_cert: PUPPET_CERT
:foreman_ssl_key: PUPPET_KEY
:foreman_ssl_ca: WEB_UI_CHAIN

Actions #4

Updated by Tatsuya Hamamoto almost 6 years ago

Hi Iain,

Thank you very much for your comment; I was beginning to feel lonely here. :)

I went ahead and tried un-commenting and changing the three parameters in foreman-proxy/settings.yml (cert and key for puppet cert, and ca bundle for web UI), and restarted the foreman-proxy service.
The proxy server has a green checkmark and I do not get any SSL errors when logging in, but the remote execution task still fails with the following showing up in the logs.

ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)

If you have anything that you would like me to test, please let me know.
Thank you again.

Actions #5

Updated by Tatsuya Hamamoto almost 6 years ago

  • translation missing: en.field_release set to 297

Just to update, still seems to be the case for 1.17.1.

Actions #6

Updated by Tomer Brisker almost 6 years ago

  • translation missing: en.field_release changed from 297 to 360
Actions #7

Updated by Tomer Brisker over 5 years ago

  • Project changed from Foreman to Foreman Remote Execution
  • Category changed from Smart Proxy to Smart Proxy Dynflow
Actions #8

Updated by Tatsuya Hamamoto over 5 years ago

  • Triaged set to No

Just to update, still seems to be the case for 1.18.1.

Actions #9

Updated by Peter J about 5 years ago

Issue still present in 1.20.2

Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pem

to

:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.key

based on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.

Actions #10

Updated by Elias Pereira almost 5 years ago

Peter J wrote:

Issue still present in 1.20.2

Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pem

to

:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.key

based on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.

Still present in version 1.21.3.

@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml

Maybe the configuration file changed from version 1.20.2 to 1.21.3?

Actions #11

Updated by Peter J almost 5 years ago

Elias Pereira wrote:

Peter J wrote:

Issue still present in 1.20.2

Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pem

to

:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.key

based on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.

Still present in version 1.21.3.

@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml

Maybe the configuration file changed from version 1.20.2 to 1.21.3?

The file should be on your Smart Proxy host.

Actions #12

Updated by Elias Pereira almost 5 years ago

Peter J wrote:

Elias Pereira wrote:

Peter J wrote:

Issue still present in 1.20.2

Workaround is to manually edit '/etc/smart_proxy_dynflow_core/settings.yml' and change
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/<hostname>.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/<hostname>.pem

to

:ssl_ca_file: /etc/httpd/ssl/<foreman-ui ca>.crt
:ssl_certificate: /etc/httpd/ssl/<foreman-ui>.crt
:ssl_private_key: /etc/httpd/ssl/<foreman-ui>.key

based on the workaround it seems that foreman-proxy is using 'foreman_ssl_*' parameters to connect to smart_proxy_dynflow_core service.

Still present in version 1.21.3.

@Peter J, on my server there is no file /etc/smart_proxy_dynflow_core/settings.yml

Maybe the configuration file changed from version 1.20.2 to 1.21.3?

The file should be on your Smart Proxy host.

The only file that my server has is:

  1. find / -iname settings.yml
    /etc/foreman-proxy/settings.yml

It would be the same file?

Actions #13

Updated by Adam Ruzicka almost 3 years ago

  • Status changed from New to Rejected

Running smart proxy dynflow core as a standalone service is being deprecated starting with Foreman 2.5. No point fixing something that will get dropped.

Actions

Also available in: Atom PDF