Bug #4693
closed
Race condition during provisioning with vagrant
Description
I'm using vagrant with foreman for provisioning - this used to work when I lightly tested a while ago but I've hit a few issues, one I've sorted (I hope!) one I've got close.
What I see is that "puppet agent" runs on the node, a certificate is generated, signed, plugins pulled down. however I get the dreaded "Unable to fetch my node definition" message. If I try again it works.
During debugging, I've been running foreman's ENC script by hand whilst watching the puppet run and what happens is interesting - basically the ENC works only once a report has come in (looking at the debug as soon as it finishes a transaction and a report is generated back - which seems to happen just before plugins are even downloaded):
Started GET "/node/hq-abridgett-d03.xxx.com?format=yml" for 10.100.16.14 at 2014-03-18 21:33:21 +0000
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"hq-abridgett-d03.xxx.com"}
Completed 404 Not Found in 22.1ms (ActiveRecord: 15.8ms)
Started POST "/api/reports" for 10.100.16.14 at 2014-03-18 21:33:21 +0000
Processing by Api::V2::ReportsController#create as JSON
Parameters: {"report"=>"[FILTERED]"}
processing report for hq-abridgett-d03.xxx.com
Completed 201 Created in 369.1ms (Views: 1.6ms | ActiveRecord: 0.0ms)
Started GET "/node/hq-abridgett-d03.xxx.com?format=yml" for 10.100.16.14 at 2014-03-18 21:33:22 +0000
Processing by HostsController#externalNodes as YML
Parameters: {"name"=>"hq-abridgett-d03.xxx.com"}
Rendered text template (0.0ms)
Completed 200 OK in 330.3ms (Views: 0.9ms | ActiveRecord: 47.2ms)
I also wrapped this around generate_fact_request as there doesn't need to be a facts file to exist for the ENC to work:
if File.exists?("#{puppetdir}/yaml/facts/#{certname}.yaml")
On a not-unrelated noted, I wondered why when running as an ENC the default node isn't returned if there's no match?
Updated by Dominic Cleal almost 11 years ago
- Status changed from New to Feedback
The missing fact file is definitely unusual, since this is the mechanism that creates the host between it uploading facts and fetching the catalog (and ENC definition from Foreman.)
Puppet creates these files by default, but there are two places I've seen it go wrong before which I'd suggest checking:
1. If you're using Puppet 3.x under Passenger or similar, check that /etc/puppet/rack/config.ru (location may vary, that's our default install path) specifies the vardir. If not, the Puppet master will write these files to the wrong location. See http://docs.puppetlabs.com/puppet/3/reference/release_notes.html#break-puppet-master-rack-configuration-is-changed
2. If you have /etc/puppet/routes.yaml, it's possible this has the YAML fact cache disabled, see http://docs.puppetlabs.com/puppet/latest/reference/config_file_routes.html
Not sure about the default node, that would be possible I suppose. Perhaps for environments that deliberately don't upload facts.
Updated by Adrian Bridgett almost 11 years ago
Hi Dominic,
I've checked those and we've got them both setup as per the documentation.
Currently I've put a little wrapper around the ENC which just returns nothing if the ENC fails (it then uses just what it finds in /etc/puppet which works for us as it can find the default node there).
Updated by Dominic Cleal almost 11 years ago
Hi Adrian,
Hmm. Assuming Puppet 3, what does puppet master --configprint yamldir return? It should point to the 'yaml' subdirectory (usually /var/lib/puppet/yaml) below the :puppetdir defined in the SETTINGS block of node.rb.
Also check puppet master --configprint facts_terminus which should print "yaml".
Updated by Anonymous