Foreman » Smart Proxy

Error: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for admybr ([RestClient::RequestTimeout]: Request Timeout) for proxy https://vmctldeploy10.rsvgnw.local:8443/puppet

proxy.log shows very slow iteration over manifests across multiple environments, measured in minutes per manifest. It's unclear from the log if this is during a request (which would explain timeouts) or from the startup process.

D, [2017-01-05T03:42:44.276364 #113893] DEBUG -- : Scanning oradb classes in /etc/puppet/environments/production/modules/oradb/manifests/listener.pp
D, [2017-01-05T03:42:44.276805 #113893] DEBUG -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
D, [2017-01-05T03:46:23.462033 #113893] DEBUG -- : Scanning oradb classes in /etc/puppet/environments/production/modules/oradb/manifests/installem_agent.pp
D, [2017-01-05T03:46:23.462328 #113893] DEBUG -- : Initializing from Puppet config file: /etc/puppet/puppet.conf

Superficially similar to #14266, which I've not confirmed has been fully resolved in the smart proxy.

Note that this only affects Puppet 3, which is now EOL. With Puppet 4 the API should make this obsolete, and since 4.4+ caching is also supported.

Hi.

I get the same issues in my Puppet 4 environment. The Webinterface says:

Error: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for puppet4 ([RestClient::ServiceUnavailable]: 503 Service Unavailable) for proxy https://*.de:8443/puppet

The foreman-proxy.log:

D, [2017-01-20T18:28:25.102324 ] DEBUG -- : accept: 10.111.2.251:55644 
D, [2017-01-20T18:28:25.103942 ] DEBUG -- : Rack::Handler::WEBrick is invoked. 
D, [2017-01-20T18:28:25.104827 ] DEBUG -- : verifying remote client 10.111.2.251 against trusted_hosts ["*.de"] 
E, [2017-01-20T18:28:40.106095 ] ERROR -- : Puppet is taking too long to respond, please try again later. 
D, [2017-01-20T18:28:40.106214 ] DEBUG -- : Puppet is taking too long to respond, please try again later. 
I, [2017-01-20T18:28:40.106599 ]  INFO -- : 10.111.2.251 - - [20/Jan/2017:18:28:40 +0100] "GET /puppet/environments/puppet4/classes HTTP/1.1" 503 61 15.0020 

99% of the requests fail, just a few work. I digged around with curl and time, working and nonworking requests run around 15s. I'm running foreman 1.14.0

When puppet 4 and puppet environment-classes api are used, smart-proxy will attempt to initialize puppet classes cache on startup. Should that fail, it will skip cache initialization and attempt to fetch classes on next request. It will return a time-out error to the client after waiting for puppet's response for 15 seconds. After that, smart-proxy will continue to await the response for another 285 seconds in the background. Could you try repeating your query again after a minute or two? Could you also check smart-proxy logs to see if there are any errors there?

I would also check how fast puppet server responds and if there are any errors in puppet logs.

The cache is created during startup:

I, [2017-01-20T17:58:58.281767 ]  INFO -- : Started puppet class cache initialization
D, [2017-01-20T17:59:00.408362 ] DEBUG -- : Initializing puppet class cache for 'puppet4' environment
D, [2017-01-20T17:59:00.408857 ] DEBUG -- : Initializing puppet class cache for 'production' environment
D, [2017-01-20T17:59:00.409331 ] DEBUG -- : Initializing puppet class cache for 'puppet4fabricdev' environment
D, [2017-01-20T17:59:00.409610 ] DEBUG -- : Initializing puppet class cache for 'common' environment
I, [2017-01-20T17:59:26.025801 ]  INFO -- : Finished puppet class cache initialization

I've a lot of stdlib deprecation warnings in my puppetserver logs, but I can't spot anything related to this. The foreman-proxy only throws the timeouts I pasted in my first comment. I'm able to get the list via curl from https://localhost:8443/puppet/environments/puppet4/classes after a bit of time, but unable via the webinterface. Tried it for 10 minutes - no luck. Is it possible to increase the timeouts somewhere?

This is quite odd -- it looks like your puppet server responses are very slow: it should return classes pretty quickly once its classes cache is initialized (which, I think, happens on first request). The fact that even repeated requests that should just return 304s time-out (they should come back pretty much instantaneously) also makes me think that there might be a problem with puppet setup. Even more strange is that it only takes 26 seconds to retrieve classes for all environments on startup.

Could you try to use curl to retrieve classes from puppet directly and note the response times?

yeah very strange. I would expect that the puppetserver caches it and further calls are fast and working if one ever suceeded, but thats not the case. I've no idea how Puppet implemented the API. I will see if I can increase the debug output for the puppetserver, maybe I will find some errors there.

Could you try using curl with GET method to puppet/v3/environment_classes?environment=<environment_name> to retrieve classes from puppet server directly and note the response times?

Did 4 calls, all needed 21s for the same env

You should be getting an ETAG with each response. What are the response times if you add "If-None-Match" header with a value appropriate for the environment you are querying?

Caching must be enabled (via environment-class-cache-enabled in puppetserver.conf) for ETAG and caching support in Puppet Server, it isn't the default.

I don't think this is related to the original bug report which was on Puppet 3, please file a new issue or open a thread on foreman-users.

Thanks Dominic, I enabled the caching and everything is working now \o/

#18290 covers a similar problem (possibly as described in comments above) when using Puppet 4. The two are otherwise not related.