Support #21876

foreman-proxy unable to access puppet api when allow-header-cert-info = true

Added by Joost Polley about 3 years ago. Updated about 3 years ago.

Target version:
Fixed in Releases:
Found in Releases:


while installing a new installation for experimental purposes before our upgrade to Puppet 5 & Foreman 1.16, I'm found a problem.

Here's a quick overview of my setup since I split my services on different hosts

1 x foreman host (foreman 1.16-RC2)
1 x puppet master (puppet server 5) + registered foreman-proxy 1.16-RC2
1 x puppet ca (puppet server 5) + registered foreman-proxy 1.16-RC2
1 x apache loadbalancer (offloads SSL, sends traffic to puppetmaster & puppetca with the needed http headers)

The problem is that I am unable to call the Puppet api through the foreman-proxy, for example to import the classes in Foreman.
When doing that, I get the following error message:

2017-11-28 21:06:39,914 ERROR [qtp133375752-56] [p.t.a.rules] Forbidden request: access to /puppet/v3/environments (method :get) (authenticated: false) denied by rule 'puppetlabs environments'.

The 'puppetlabs environments' rule is defined in /etc/puppetlabs/puppetserver/conf.d/auth.conf and looks like this:

match-request: {
path: "/puppet/v3/environments"
type: path
method: get
allow: "*"
sort-order: 500
name: "puppetlabs environments"

This looks good and is also what's specified in the documentation.

I was able to fix the problem, by in the same file (/etc/puppetlabs/puppetserver/conf.d/auth.conf) changing allow-header-cert-info to false.
This is however not something I want since it's very unsafe (it would allow un-authorized requests to my Puppet master).

I kept debugging (both http:8139 and https:8140) and found out that with the allow-header-cert-info the following does not work:

curl localhost:8139/puppet/v3/environments
Forbidden request: /puppet/v3/environments (method :get). Please see the server logs for details.

... and this does work (adding correct headers):

curl -H "X-Client-DN: CN=authorized-host-by-ca" -H "X-Client-Verify: SUCCESS" localhost:8139/puppet/v3/environments

I added the logging of the request headers in the logback.xml to see what data the foreman-proxy sends:

<pattern>%h %l %u [%t] "%r" %s %b "%i{Referer}" "%i{User-Agent}" %D %i{X-Client-DN} %i{X-Client-Verify}</pattern>

With the changed access logging, this is the failed foreman-proxy request: - - [28/Nov/2017:21:06:44 +0000] "GET /puppet/v3/environments HTTP/1.1" 403 97 "-" "Ruby" 7 - -

... and this was the previous succeeding curl request: - - [28/Nov/2017:21:26:34 +0000] "GET /puppet/v3/environments HTTP/1.1" 200 1047 "-" "curl/7.52.1" 438 CN=authorized-host-by-ca SUCCESS

So as far as I understand, it looks like foreman-proxy is not sending the X-Client-DN and X-Client-Verify header data and because of Puppet server only authenticating with headers, the proxy is unable to import classes and environments.

As a reference, here's the foreman-installer command used for installation of the puppetmaster host:

foreman-installer \
--no-enable-foreman \
--no-enable-foreman-cli \
--no-enable-foreman-plugin-setup \
--enable-foreman-proxy \
--enable-puppet \
--foreman-proxy-trusted-hosts=foreman.example.lan \
--foreman-proxy-tftp=false \
--foreman-proxy-dhcp=false \
--foreman-proxy-dhcp-range="false" \
--foreman-proxy-dns=false \
--foreman-proxy-puppetca=false \
--foreman-proxy-foreman-base-url=https://foreman.example.lan \
--foreman-proxy-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/puppetmaster.example.lan.pem \
--foreman-proxy-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/puppetmaster.example.lan.pem \
--foreman-proxy-foreman-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/foreman.example.lan.pem \
--foreman-proxy-foreman-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/foreman.example.lan.pem \
--foreman-proxy-puppet-ssl-cert=/etc/puppetlabs/puppet/ssl/certs/foreman.example.lan.pem \
--foreman-proxy-puppet-ssl-key=/etc/puppetlabs/puppet/ssl/private_keys/foreman.example.lan.pem \
--foreman-proxy-oauth-consumer-key=masked-key \
--foreman-proxy-oauth-consumer-secret=masked-secret \
--puppet-server-ca=false \
--puppet-server-certname=puppetmaster.example.lan \
--puppet-server-foreman-url=https://foreman.example.lan \
--puppet-server-http=true \
--puppet-server-http-port=8139 \
--puppet-client-certname=puppetmaster.example.lan \


#1 Updated by Daniel Lobato Garcia about 3 years ago

  • Legacy Backlogs Release (now unused) set to 332

Marking as 1.16.1 to study including a fix for this for that version

#2 Updated by Anonymous about 3 years ago

I think this is actually fine, see

AFAICT, -proxy should use the Puppet API via HTTPS, authenticating with its certs, so the loadbalancer should set the headers accordingly.

#3 Updated by Joost Polley about 3 years ago

Maybe this is something I do not understand. Suppose I have one apache load balancer where my agents connect to. This load balancer does the SSL termination and adds the required headers. It:

  • forwards the certificate traffic to the puppetca pool (which contains 1 host)
  • forwards all the rest to the puppetmaster pool (which contains 2 hosts)

In this case, the puppet masters and the puppet ca only use their HTTP listener (in my case port 8139). The foreman-proxies are installed on the 3 hosts. So I'll get 3 registered smart proxies in Foreman, being master01, master02 and ca01.

In this setup, how would the smart proxy be able to get to the puppet master? Keep in mind that in this setup all the hosts have the allow-header-cert-info setting set to true, which means the puppetserver will never look to the certificates, only to the HTTP headers. That means the HTTPS listener is active but does not work.

#4 Updated by Dmitri Dolguikh about 3 years ago

I think you can drop authentication for smart-proxy addresses on puppet master -- http headers over http aren't providing any useful authentication anyway.

#5 Updated by Daniel Lobato Garcia about 3 years ago

  • Legacy Backlogs Release (now unused) deleted (332)

#6 Updated by Joost Polley about 3 years ago

Thank you Dmitri. Dropping authentication, could you please be a bit more specific? Are we then talking about 'client-whitelist' in

This however still means I have to support the HTTPS listener since this whitelist is based on the CN in the certificate? I'm still stuck with the problem of my HTTP offloading, no?


#8 Updated by Joost Polley about 3 years ago

If I use allow-unauthenticated I don't authenticate the rest of my hosts anymore. This setting is a boolean, not a whitelist. I don't think this is a good idea?

#9 Updated by Dmitri Dolguikh about 3 years ago

I don't think this is a good idea?

I don't see why not, tbh. Authentication over http isn't useful (I'd argue it's harmful, but it's another topic altogether).

#10 Updated by Joost Polley about 3 years ago

Okay. So here's what I did then in conf.d/auth.conf for the rules 'puppetlabs environments' and 'puppetlabs environment classes':

  • removed setting 'allow'
  • added setting 'allow-unauthenticated: true'

This works as expected which makes me very happy! Now my next question would be, but this might be a little bit off topic for this issue. How do I configure this setting with the foreman-installer? It looks like the theforeman/puppet-puppet module manages these settings but they're not to be manipulated (reference:

#11 Updated by Dmitri Dolguikh about 3 years ago

  • Status changed from New to Rejected

Closing the ticket as the issue has been resolved.

#12 Updated by Joost Polley about 3 years ago

I didn't go with the solution proposed above. I've set up an apache vhost that listens on port 8140 (make puppetserver https listen on 8141 since I don't need that listener). This vhost listens to traffic on the local interface, offloads ssl, adds the correct headers and sends it to the puppetserver.

So with this solution I have two listeners on the server:
  • 8139: puppetserver http listener
    • authentication through http headers
    • this is the traffic that comes from my puppet agents, with ssl offloaded at the loadbalancer
  • 8140: apache2 vhost https listener
    • offloads ssl and forwards traffic to with the needed headers

#13 Updated by Dmitri Dolguikh about 3 years ago

  • Status changed from Rejected to Resolved

Thanks for documenting your solution here!

#14 Updated by Dmitri Dolguikh about 3 years ago

  • Tracker changed from Bug to Support

Also available in: Atom PDF