Foreman » Smart Proxy

while installing a new installation for experimental purposes before our upgrade to Puppet 5 & Foreman 1.16, I'm found a problem.

Here's a quick overview of my setup since I split my services on different hosts

1 x foreman host (foreman 1.16-RC2)
1 x puppet master (puppet server 5) + registered foreman-proxy 1.16-RC2
1 x puppet ca (puppet server 5) + registered foreman-proxy 1.16-RC2
1 x apache loadbalancer (offloads SSL, sends traffic to puppetmaster & puppetca with the needed http headers)

The problem is that I am unable to call the Puppet api through the foreman-proxy, for example to import the classes in Foreman.
When doing that, I get the following error message:

2017-11-28 21:06:39,914 ERROR [qtp133375752-56] [p.t.a.rules] Forbidden request: 127.0.0.1 access to /puppet/v3/environments (method :get) (authenticated: false) denied by rule 'puppetlabs environments'.

The 'puppetlabs environments' rule is defined in /etc/puppetlabs/puppetserver/conf.d/auth.conf and looks like this:

{
match-request: {
path: "/puppet/v3/environments"
type: path
method: get
}
allow: "*"
sort-order: 500
name: "puppetlabs environments"
}

This looks good and is also what's specified in the documentation.

I was able to fix the problem, by in the same file (/etc/puppetlabs/puppetserver/conf.d/auth.conf) changing allow-header-cert-info to false.
This is however not something I want since it's very unsafe (it would allow un-authorized requests to my Puppet master).

I kept debugging (both http:8139 and https:8140) and found out that with the allow-header-cert-info the following does not work:

curl localhost:8139/puppet/v3/environments
Forbidden request: /puppet/v3/environments (method :get). Please see the server logs for details.

... and this does work (adding correct headers):

curl -H "X-Client-DN: CN=authorized-host-by-ca" -H "X-Client-Verify: SUCCESS" localhost:8139/puppet/v3/environments
"search_paths":["file:///etc/puppetlabs/code/environments","data:text/plain,internal"],...

I added the logging of the request headers in the logback.xml to see what data the foreman-proxy sends:

<pattern>%h %l %u [%t] "%r" %s %b "%i{Referer}" "%i{User-Agent}" %D %i{X-Client-DN} %i{X-Client-Verify}</pattern>

With the changed access logging, this is the failed foreman-proxy request:

127.0.0.1 - - [28/Nov/2017:21:06:44 +0000] "GET /puppet/v3/environments HTTP/1.1" 403 97 "-" "Ruby" 7 - -

... and this was the previous succeeding curl request:

127.0.0.1 - - [28/Nov/2017:21:26:34 +0000] "GET /puppet/v3/environments HTTP/1.1" 200 1047 "-" "curl/7.52.1" 438 CN=authorized-host-by-ca SUCCESS

So as far as I understand, it looks like foreman-proxy is not sending the X-Client-DN and X-Client-Verify header data and because of Puppet server only authenticating with headers, the proxy is unable to import classes and environments.

As a reference, here's the foreman-installer command used for installation of the puppetmaster host:

foreman-installer \
--no-enable-foreman \
--no-enable-foreman-cli \
--no-enable-foreman-plugin-setup \
--enable-foreman-proxy \
--enable-puppet \
--foreman-proxy-trusted-hosts=foreman.example.lan \
--foreman-proxy-tftp=false \
--foreman-proxy-dhcp=false \
--foreman-proxy-dhcp-range="false" \
--foreman-proxy-dns=false \
--foreman-proxy-puppetca=false \
--foreman-proxy-foreman-base-url=https://foreman.example.lan \
--foreman-proxy-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/puppetmaster.example.lan.pem \
--foreman-proxy-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/puppetmaster.example.lan.pem \
--foreman-proxy-foreman-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/foreman.example.lan.pem \
--foreman-proxy-foreman-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/foreman.example.lan.pem \
--foreman-proxy-puppet-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/foreman.example.lan.pem \
--foreman-proxy-puppet-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/foreman.example.lan.pem \
--foreman-proxy-oauth-consumer-key=masked-key \
--foreman-proxy-oauth-consumer-secret=masked-secret \
--puppet-server-ca=false \
--puppet-server-certname=puppetmaster.example.lan \
--puppet-server-foreman-url=https://foreman.example.lan \
--puppet-server-http=true \
--puppet-server-http-port=8139 \
--puppet-client-certname=puppetmaster.example.lan \
--foreman-proxy-registered-name=puppetmaster.example.lan