Project

General

Profile

Actions

Bug #10941

closed

puppet classes cache not working right

Added by Alejandro Falcon almost 9 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Puppet
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:

-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:

          if lcache[puppetmodule].has_key?(:timestamp) && lcache[puppetmodule][:timestamp] >= mtime


Related issues 3 (0 open3 closed)

Related to Smart Proxy - Feature #8210: Implemented caching for smart-proxy puppet classesClosed10/30/2014Actions
Related to Smart Proxy - Bug #11131: Intermittent Puppet class cache test with timestampsClosedDominic Cleal07/16/2015Actions
Related to Smart Proxy - Bug #11229: Import classes does not find any new classesClosed07/28/2015Actions
Actions #1

Updated by Dominic Cleal almost 9 years ago

  • Related to Feature #8210: Implemented caching for smart-proxy puppet classes added
Actions #2

Updated by Jason Smith over 8 years ago

Alejandro Falcon wrote:

Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:

-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:
[...]

I also noticed this problem and tried to debug it a little, but I know very little ruby so I didn't get far. I assume that this worked for the person who originally added this feature, so I think it might be a problem with older versions of ruby. Our puppet master is running on an old RHEL6 server with ruby v1.8.7. Just like what Alejandro found, one problem is the cache timestamp comparison fails, but if I change that line to this:

if lcache[puppetmodule].has_key?(:timestamp) && Time.parse(lcache[puppetmodule][:timestamp]) >= mtime

then this part works. But then the caching still does not work properly because of symbolized names, so I had to change the JSON.parse on line 61 to:

JSON.parse(File.read(cachefile), {:symbolize_names => true})

Then I got to the problem that the json data in the cache file seems to be missing a lot of information, for example, it only contains the module path & name, timestamp, and list of classes in the module, but all of the parameters and their default values are missing. This information is not written to the cache file, possibly due to the same symbolize_names problem in the data serialization.

Since I am not a ruby expert, this is as far as I was able to get, hopefully someone who knows ruby better can figure this out and get the caching to work. Operating a puppet server with dozens of environments, where each can take 30+ seconds to parse, is very difficult with Foreman that does not have a working cache.

Actions #3

Updated by Dominic Cleal over 8 years ago

  • translation missing: en.field_release set to 62

Yeah, I agree with everything you said Jason, you're spot on. Here's a unit test that demonstrates it: https://github.com/theforeman/smart-proxy/compare/develop...domcleal:10941-puppet-cache-serialisation

PuppetCacheTest#test_read_write_cache_idempotency [/home/dcleal/code/foreman/smart-proxy/test/puppet/puppet_cache_test.rb:81]:
<{"testinclude"=>
  {:timestamp=>2015-07-09 09:55:22 +0100,
   :manifest=>
    [[#<Proxy::Puppet::PuppetClass:0x0000000407ab30
       @klass="test",
       @params={}>],
     [#<Proxy::Puppet::PuppetClass:0x0000000407aa18
       @klass="test::sub::foo",
       @params={}>]]}}> expected but was
<{"testinclude"=>
  {"timestamp"=>"2015-07-09 09:55:22 +0100",
   "manifest"=>[["test"], ["test::sub::foo"]]}}>.
  1. timestamp is changing class to a String on deserialisation, so the comparison to mtime later fails unless it's re-parsed
  2. symbol/string differences in Hash keys
  3. Proxy::Puppet::PuppetClass contents aren't serialised so the parameter data is lost
Actions #4

Updated by Anonymous over 8 years ago

  • Status changed from New to Assigned
  • Assignee set to Anonymous
Actions #5

Updated by The Foreman Bot over 8 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/smart-proxy/pull/297 added
  • Pull request deleted ()
Actions #6

Updated by Dominic Cleal over 8 years ago

  • Related to Bug #11131: Intermittent Puppet class cache test with timestamps added
Actions #7

Updated by Anonymous over 8 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #8

Updated by Dominic Cleal over 8 years ago

  • Related to Bug #11229: Import classes does not find any new classes added
Actions #9

Updated by Dominic Cleal over 8 years ago

  • translation missing: en.field_release changed from 62 to 35

Bumping to 1.9 due to regression found.

Actions

Also available in: Atom PDF