Project

General

Profile

Bug #18290

"Puppet is taking too long to respond" during class import with Puppet 4.x

Added by Paul Seymour over 2 years ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Category:
Puppet
Target version:
Difficulty:
Triaged:
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

On the proxy status page I get a tick, I can read the PuppetCA information I can see the reports but I cannot read the environments/classes,or import them.

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for dev ([Rest...)

think I might have tracked this down to a limit of around 1024 classes in the import. If I remove a bunch of modules (any it seems) and bring it to under 1024 then the import (via rake task or UI) it works. If I move some more modules in past 1024 it throws the:-

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for dev ([RestClient::ServiceUnavailable]: 503 Service Unavailable) for proxy <proxy>

Related issues

Related to Smart Proxy - Feature #15095: Add Puppet server environment_classes API supportClosed
Related to Smart Proxy - Bug #17942: Timeout during Puppet class import, slow re-initialization of Puppet 3Rejected2017-01-05

Associated revisions

Revision b1020e19 (diff)
Added by Dmitri Dolguikh over 2 years ago

Fixes #18290 - increased puppet classes api timeout

But only when puppet server class cache is disabled.
Additionally, a warning is logged to notify users about
potentially degraded performance.

History

#1 Updated by Dominic Cleal over 2 years ago

  • Project changed from Foreman to Smart Proxy
  • Subject changed from Smart Proxy Import Issue to 503 error when importing around 1024 classes
  • Description updated (diff)
  • Category changed from Smart proxies to Puppet

Could you please show the corresponding smart proxy logs behind the 503 error received? Debug logs from the startup of the smart proxy may also be useful to show the active configuration.

#2 Updated by Paul Seymour over 2 years ago

Dominic Cleal wrote:

Could you please show the corresponding smart proxy logs behind the 503 error received? Debug logs from the startup of the smart proxy may also be useful to show the active configuration.

Hi,

Sure

D, [2017-01-26T09:24:47.271886 ] DEBUG -- : Rack::Handler::WEBrick is mounted on /.
I, [2017-01-26T09:24:47.272065 ] INFO -- : WEBrick::HTTPServer#start: pid=57900 port=8443
D, [2017-01-26T09:24:47.293966 ] DEBUG -- : Initializing puppet class cache for 'production' environment
D, [2017-01-26T09:24:47.294306 ] DEBUG -- : Initializing puppet class cache for 'dev' environment
I, [2017-01-26T09:25:09.410181 ] INFO -- : Finished puppet class cache initialization
D, [2017-01-26T09:25:26.436903 ] DEBUG -- : accept:<IP>:45794
D, [2017-01-26T09:25:26.454296 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-26T09:25:26.476200 ] DEBUG -- : verifying remote client <IP> against trusted_hosts ["<HOST>l"]
I, [2017-01-26T09:25:26.560670 ] INFO -- : <IP> - - [26/Jan/2017:09:25:26 +0000] "GET /puppet/environments HTTP/1.1" 200 20 0.0847

D, [2017-01-26T09:25:26.602187 ] DEBUG -- : close: <IP>:45794
D, [2017-01-26T09:25:26.662192 ] DEBUG -- : accept: <IP>:45796
D, [2017-01-26T09:25:26.663510 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-26T09:25:26.664222 ] DEBUG -- : verifying remote client <IP> against trusted_hosts ["<HOST>"]
I, [2017-01-26T09:25:26.742639 ] INFO -- : <IP> - - [26/Jan/2017:09:25:26 +0000] "GET /puppet/environments/production/classes HTTP/1.1" 200 2 0.0786

D, [2017-01-26T09:25:26.783797 ] DEBUG -- : close: <IP>:45796
D, [2017-01-26T09:25:28.177582 ] DEBUG -- : accept: <IP>:45806
D, [2017-01-26T09:25:28.179288 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2017-01-26T09:25:28.180054 ] DEBUG -- : verifying remote client 10.34.180.152 against trusted_hosts ["<HOST>"]
E, [2017-01-26T09:25:43.181061 ] ERROR -- : Puppet is taking too long to respond, please try again later.
D, [2017-01-26T09:25:43.181207 ] DEBUG -- : Puppet is taking too long to respond, please try again later.
I, [2017-01-26T09:25:43.181566 ] INFO -- : <IP> - - [26/Jan/2017:09:25:43 +0000] "GET /puppet/environments/dev/classes HTTP/1.1" 503 61 15.0017

#3 Updated by Dominic Cleal over 2 years ago

The environment classes impl added in #15095 is likely too aggressive with the default 15 second timeout, and it should probably have a higher default.

It makes more sense to have a short client timeout when the Puppet Server environment classes API has caching enabled (as then a subsequent query will have cached a value, and the P.S. API will quickly respond with a 304), but not so much in a default configuration where caching is disabled. In the non-caching case, the timeout should be higher to wait for a full response from Puppet Server.

#4 Updated by Dominic Cleal over 2 years ago

  • Related to Feature #15095: Add Puppet server environment_classes API support added

#5 Updated by Dmitri Dolguikh over 2 years ago

One approach is to increase the timeout. It's trivial, but there still will be a puppet environment with sufficient number of classes to trigger this issue again. I would suggest an alternative: leave the timeout as is (maybe increase it slightly). Poll puppet server every couple of minutes in background and update local cache (+ etag if available). On an api request for classes attempt to pull classes from the puppet server, return locally cached classes on a timeout. This way we know we are always getting data that's reasonably recent (no more than a couple of minutes old).

Thoughts?

#6 Updated by Dominic Cleal over 2 years ago

If the data is old by a couple of minutes then it's likely to cause staleness issues for all users automating Foreman's environment imports, or even for users changing an environment manually and importing. It could also create significant background load for large environments.

#7 Updated by The Foreman Bot over 2 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Dmitri Dolguikh
  • Pull request https://github.com/theforeman/smart-proxy/pull/504 added

#8 Updated by Dominic Cleal over 2 years ago

  • Related to Bug #17942: Timeout during Puppet class import, slow re-initialization of Puppet 3 added

#9 Updated by Dominic Cleal over 2 years ago

  • Subject changed from 503 error when importing around 1024 classes to "Puppet is taking too long to respond" during class import with Puppet 4.x

#10 Updated by Dmitri Dolguikh over 2 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#11 Updated by Dominic Cleal over 2 years ago

  • Legacy Backlogs Release (now unused) set to 221

Also available in: Atom PDF