Bug #19081
open
node.rb failing with servers with large amount of facts (ips?)
Description
Hey,
I recently had some boxes come under my management that have a super large amount of IPs on them, like, 500-600 individual ip addresses bound to them.
Foreman seems to ingest these facts really really slowly when /etc/puppet/node.rb is run. I end up getting a timeout after waiting around for 60 seconds:
[root@foreman ~]# /etc/puppet/node.rb host1.hostname.com Could not send facts to Foreman: Net::ReadTimeout
My solution was to raise this like so:
--- /etc/puppet/node.rb 2017-03-29 16:53:12.755852699 -0500
+++ /etc/puppet/node.rb.new 2017-03-29 16:47:06.660634862 -0500
@@ -107,6 +107,7 @@
def initialize_http(uri)
res = Net::HTTP.new(uri.host, uri.port)
res.use_ssl = uri.scheme == 'https'
+ res.read_timeout = 180
if res.use_ssl?
if SETTINGS[:ssl_ca] && !SETTINGS[:ssl_ca].empty?
res.ca_file = SETTINGS[:ssl_ca]
I know this is inelegant, first of all it should probably use SETTINGS[:enc_read_timeout] or something easier to configure, but closer to the issue, I'm not sure why it takes so long for Foreman to ingest these facts. I tried tracing through the code but wasn't really successful in figuring it out.
Updated by Dominic Cleal about 7 years ago
- Project changed from Foreman to Installer
- Category changed from Facts to Foreman modules
The script already has a :timeout setting, which is set to read_timeout (and open_timeout in the latest version) when performing fact uploads. Shouldn't that change the same thing?
Updated by David Ibarra about 7 years ago
Dominic Cleal wrote:
The script already has a
:timeoutsetting, which is set to read_timeout (and open_timeout in the latest version) when performing fact uploads. Shouldn't that change the same thing?
Sort of. I don't claim to have advanced knowledge of the foreman code base, or even Ruby, but it seems that :timeout for node.rb controls a timeout for the block of code that gets the classification of the server, not the read_timeout for the Net::HTTP client that pushes the facts into the foreman API.
Updated by David Ibarra about 7 years ago
I think I found the cause of the slowness, it seems like it's ActiveRecord:
app/models/host/base.rb:
if iface.new_record? || iface.changed?
logger.debug "Saving #{name} NIC for host #{self.name}"
result = iface.save
unless result
logger.warn "Saving #{name} NIC for host #{self.name} failed, skipping because:"
iface.errors.full_messages.each { |e| logger.warn " #{e}" }
end
result
end
If I disable activerecord callbacks (iface.save to iface.save(:validate => false)) this thing goes way way faster. I'm talking like, we're going from 5 minutes to import these facts, to 14 seconds.
Updated by Ewoud Kohl van Wijngaarden over 6 years ago
- Related to Bug #20929: Huge table "fact_names" makes Foreman crawl added