Project

General

Profile

Bug #10941

puppet classes cache not working right

Added by Alejandro Falcon over 5 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Category:
Puppet
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:

-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:

          if lcache[puppetmodule].has_key?(:timestamp) && lcache[puppetmodule][:timestamp] >= mtime


Related issues

Related to Smart Proxy - Feature #8210: Implemented caching for smart-proxy puppet classesClosed2014-10-30
Related to Smart Proxy - Bug #11131: Intermittent Puppet class cache test with timestampsClosed2015-07-16
Related to Smart Proxy - Bug #11229: Import classes does not find any new classesClosed2015-07-28

Associated revisions

Revision 4d5fc803 (diff)
Added by Dmitri Dolguikh over 5 years ago

fixes #10941: fixed puppet class caching

History

#1 Updated by Dominic Cleal over 5 years ago

  • Related to Feature #8210: Implemented caching for smart-proxy puppet classes added

#2 Updated by Jason Smith over 5 years ago

Alejandro Falcon wrote:

Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:

-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:
[...]

I also noticed this problem and tried to debug it a little, but I know very little ruby so I didn't get far. I assume that this worked for the person who originally added this feature, so I think it might be a problem with older versions of ruby. Our puppet master is running on an old RHEL6 server with ruby v1.8.7. Just like what Alejandro found, one problem is the cache timestamp comparison fails, but if I change that line to this:

if lcache[puppetmodule].has_key?(:timestamp) && Time.parse(lcache[puppetmodule][:timestamp]) >= mtime

then this part works. But then the caching still does not work properly because of symbolized names, so I had to change the JSON.parse on line 61 to:

JSON.parse(File.read(cachefile), {:symbolize_names => true})

Then I got to the problem that the json data in the cache file seems to be missing a lot of information, for example, it only contains the module path & name, timestamp, and list of classes in the module, but all of the parameters and their default values are missing. This information is not written to the cache file, possibly due to the same symbolize_names problem in the data serialization.

Since I am not a ruby expert, this is as far as I was able to get, hopefully someone who knows ruby better can figure this out and get the caching to work. Operating a puppet server with dozens of environments, where each can take 30+ seconds to parse, is very difficult with Foreman that does not have a working cache.

#3 Updated by Dominic Cleal over 5 years ago

  • Legacy Backlogs Release (now unused) set to 62

Yeah, I agree with everything you said Jason, you're spot on. Here's a unit test that demonstrates it: https://github.com/theforeman/smart-proxy/compare/develop...domcleal:10941-puppet-cache-serialisation

PuppetCacheTest#test_read_write_cache_idempotency [/home/dcleal/code/foreman/smart-proxy/test/puppet/puppet_cache_test.rb:81]:
<{"testinclude"=>
  {:timestamp=>2015-07-09 09:55:22 +0100,
   :manifest=>
    [[#<Proxy::Puppet::PuppetClass:0x0000000407ab30
       @klass="test",
       @params={}>],
     [#<Proxy::Puppet::PuppetClass:0x0000000407aa18
       @klass="test::sub::foo",
       @params={}>]]}}> expected but was
<{"testinclude"=>
  {"timestamp"=>"2015-07-09 09:55:22 +0100",
   "manifest"=>[["test"], ["test::sub::foo"]]}}>.
  1. timestamp is changing class to a String on deserialisation, so the comparison to mtime later fails unless it's re-parsed
  2. symbol/string differences in Hash keys
  3. Proxy::Puppet::PuppetClass contents aren't serialised so the parameter data is lost

#4 Updated by Dmitri Dolguikh over 5 years ago

  • Status changed from New to Assigned
  • Assignee set to Dmitri Dolguikh

#5 Updated by The Foreman Bot over 5 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/smart-proxy/pull/297 added
  • Pull request deleted ()

#6 Updated by Dominic Cleal over 5 years ago

  • Related to Bug #11131: Intermittent Puppet class cache test with timestamps added

#7 Updated by Anonymous over 5 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#8 Updated by Dominic Cleal over 5 years ago

  • Related to Bug #11229: Import classes does not find any new classes added

#9 Updated by Dominic Cleal over 5 years ago

  • Legacy Backlogs Release (now unused) changed from 62 to 35

Bumping to 1.9 due to regression found.

Also available in: Atom PDF