Bug #10941
closedpuppet classes cache not working right
Description
Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:
-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:
if lcache[puppetmodule].has_key?(:timestamp) && lcache[puppetmodule][:timestamp] >= mtime
Updated by Dominic Cleal over 9 years ago
- Related to Feature #8210: Implemented caching for smart-proxy puppet classes added
Updated by Jason Smith over 9 years ago
Alejandro Falcon wrote:
Every time I try to import the classes the smart-proxy scans all modules and wont use the cache. I have troubleshoot it and found:
-Cache option is enabled by default
-Cache files are created under /usr/foreman-proxy/cache and the content looks right.
-Cache files are loaded fine by the proxy but this condition fails to match for some reason:
[...]
I also noticed this problem and tried to debug it a little, but I know very little ruby so I didn't get far. I assume that this worked for the person who originally added this feature, so I think it might be a problem with older versions of ruby. Our puppet master is running on an old RHEL6 server with ruby v1.8.7. Just like what Alejandro found, one problem is the cache timestamp comparison fails, but if I change that line to this:
if lcache[puppetmodule].has_key?(:timestamp) && Time.parse(lcache[puppetmodule][:timestamp]) >= mtime
then this part works. But then the caching still does not work properly because of symbolized names, so I had to change the JSON.parse on line 61 to:
JSON.parse(File.read(cachefile), {:symbolize_names => true})
Then I got to the problem that the json data in the cache file seems to be missing a lot of information, for example, it only contains the module path & name, timestamp, and list of classes in the module, but all of the parameters and their default values are missing. This information is not written to the cache file, possibly due to the same symbolize_names problem in the data serialization.
Since I am not a ruby expert, this is as far as I was able to get, hopefully someone who knows ruby better can figure this out and get the caching to work. Operating a puppet server with dozens of environments, where each can take 30+ seconds to parse, is very difficult with Foreman that does not have a working cache.
Updated by Dominic Cleal over 9 years ago
- Translation missing: en.field_release set to 62
Yeah, I agree with everything you said Jason, you're spot on. Here's a unit test that demonstrates it: https://github.com/theforeman/smart-proxy/compare/develop...domcleal:10941-puppet-cache-serialisation
PuppetCacheTest#test_read_write_cache_idempotency [/home/dcleal/code/foreman/smart-proxy/test/puppet/puppet_cache_test.rb:81]: <{"testinclude"=> {:timestamp=>2015-07-09 09:55:22 +0100, :manifest=> [[#<Proxy::Puppet::PuppetClass:0x0000000407ab30 @klass="test", @params={}>], [#<Proxy::Puppet::PuppetClass:0x0000000407aa18 @klass="test::sub::foo", @params={}>]]}}> expected but was <{"testinclude"=> {"timestamp"=>"2015-07-09 09:55:22 +0100", "manifest"=>[["test"], ["test::sub::foo"]]}}>.
- timestamp is changing class to a String on deserialisation, so the comparison to mtime later fails unless it's re-parsed
- symbol/string differences in Hash keys
- Proxy::Puppet::PuppetClass contents aren't serialised so the parameter data is lost
Updated by Anonymous over 9 years ago
- Status changed from New to Assigned
- Assignee set to Anonymous
Updated by The Foreman Bot over 9 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/smart-proxy/pull/297 added
- Pull request deleted (
)
Updated by Dominic Cleal over 9 years ago
- Related to Bug #11131: Intermittent Puppet class cache test with timestamps added
Updated by Anonymous over 9 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 4d5fc80349ac7c0eb4908a2493fd2d243d3352e4.
Updated by Dominic Cleal over 9 years ago
- Related to Bug #11229: Import classes does not find any new classes added
Updated by Dominic Cleal over 9 years ago
- Translation missing: en.field_release changed from 62 to 35
Bumping to 1.9 due to regression found.