Project

General

Profile

Bug #23055

Dynflow SSL Issue when using custom SSL Certificate

Added by TJ Hamamoto 4 months ago. Updated 9 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Smart Proxy Dynflow
Target version:
Difficulty:
Triaged:
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

This would be my first ticket, so please let me know if I'm not doing something correctly.

Some background first:
We were using Foreman version 1.15.6 with Remote SSH Execution enabled. The SSL for the Web UI was set with our domain's Wildcard SSL Certificate and all was working just perfectly.
When we tried setting up a client that had Puppet version 5 we couldn't get the agent to run because of the puppet version conflict.
So we did the following,

1. Upgraded puppet from 3 to 4
2. Upgraded foreman from 1.15.6 to 1.16
3. Upgraded puppet from 4 to 5
(followed the instructions from the foreman site for each step)

Everything seemed to be working well; the smart proxy has a green checkmark, no warning logs and the clients are in-sync.
However when trying to issue a remote shell script, the script waits for a minute or two and then fails.
But when I look at the sub-task, the command does get executed. (I tested an scp command and it was successfully transferred)
The sub-task result displays as follows:

Failed to initialize: Dynflow::ExecutionPlan::Steps::Error - ERF42-3325 [Foreman::Exception]: The smart proxy task fa31b59e-0aac-484e-a8d8-c8078aee9a7e failed.

I then looked at foreman-proxy/proxy.log and found the following ERROR lines:

ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)

Before this line, the SSL certificate that was displayed was the Puppet Cert.
To check, I've changed the apache SSL setting to use Puppet CA and executed the Remote Shell script again, and it worked.
So something must have changed. I tested the following to see if it works, but no luck.

1. Changed back Websockets SSL to Puppet Cert (Since I've changed this to Custom SSL based on the instructions for setting custom certs)
2. Uncommented and entered the :foreman_ssl_ca: lines in foreman-proxy/settings.d since it states that this is "Mainly useful when Foreman uses different certificates for its web UI and for smart-proxy requests."
3. Did a mix of Puppet/Custom SSL certs for :foreman_ssl_ca: lines and :ssl_ca_file: combination since it didn't exactly state which is for the web UI and which is for the smart-proxy.
4. Also tried only changing the :foreman_ssl_cert: and :foreman_ssl_key:, leaving :foreman_ssl_ca: as the Puppet CA since the ca file is the same in apache settings also.

I restarted the foreman-proxy service each time I tried, but the only change I got from doing the above was that the task immediately failed and showed that "There are no foreman proxies available". (Or something similar, I forgot to make a note of the exact message for this one)
So it looks to me that I can't change the Web UI's SSL and also use Remote SSH Execution at the same time.

If you need more information, please let me know.
Thank you.

History

#1 Updated by TJ Hamamoto 3 months ago

Just to update, still seems to be the case for 1.16.1.

Also wanted to add that I've applied the workaround to downgrade sequel gem to 4.15.0 and pg gem to 0.17.1 as indicated here: https://projects.theforeman.org/issues/20050
And yes I'm running foreman on Ubuntu 16.04; sorry I should have pointed this out in the beginning.

#2 Updated by Tatsuya Hamamoto 3 months ago

*Sorry I got a different account

Still seems to be the case for 1.17.0, however I did get the 'sequel' and 'pg' gem downgrade fixed by following the instructions from Ivan here: https://projects.theforeman.org/issues/20050
If someone has any suggestions on how to fix the SSL error, that will be great.
Best regards,

TJ

#3 Updated by Iain Walmsley 2 months ago

Will depend on your setup and like you say the apache settings you have. For a typical setep you'll need the root/chain certs for the web UI. Depending on whether

Hence it'll look something like this;

:foreman_ssl_cert: PUPPET_CERT
:foreman_ssl_key: PUPPET_KEY
:foreman_ssl_ca: WEB_UI_CHAIN

#4 Updated by Tatsuya Hamamoto 2 months ago

Hi Iain,

Thank you very much for your comment; I was beginning to feel lonely here. :)

I went ahead and tried un-commenting and changing the three parameters in foreman-proxy/settings.yml (cert and key for puppet cert, and ca bundle for web UI), and restarted the foreman-proxy service.
The proxy server has a green checkmark and I do not get any SSL errors when logging in, but the remote execution task still fails with the following showing up in the logs.

ERROR -- SSL_connect returned=1 errno=0 state=error: certificate verify failed (RestClient::SSLCertificateNotVerified)

If you have anything that you would like me to test, please let me know.
Thank you again.

#5 Updated by Tatsuya Hamamoto about 2 months ago

  • Legacy Backlogs Release (now unused) set to 297

Just to update, still seems to be the case for 1.17.1.

#6 Updated by Tomer Brisker about 2 months ago

  • Legacy Backlogs Release (now unused) changed from 297 to 360

#7 Updated by Tomer Brisker 18 days ago

  • Category changed from Smart proxies to Smart Proxy Dynflow
  • Project changed from Foreman to Foreman Remote Execution

Also available in: Atom PDF