Project

General

Profile

Bug #19081

node.rb failing with servers with large amount of facts (ips?)

Added by David Ibarra over 4 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Foreman modules
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Hey,

I recently had some boxes come under my management that have a super large amount of IPs on them, like, 500-600 individual ip addresses bound to them.

Foreman seems to ingest these facts really really slowly when /etc/puppet/node.rb is run. I end up getting a timeout after waiting around for 60 seconds:

[root@foreman ~]# /etc/puppet/node.rb host1.hostname.com
Could not send facts to Foreman:  Net::ReadTimeout

My solution was to raise this like so:

--- /etc/puppet/node.rb    2017-03-29 16:53:12.755852699 -0500
+++ /etc/puppet/node.rb.new    2017-03-29 16:47:06.660634862 -0500
@@ -107,6 +107,7 @@
 def initialize_http(uri)
   res              = Net::HTTP.new(uri.host, uri.port)
   res.use_ssl      = uri.scheme == 'https'
+  res.read_timeout = 180
   if res.use_ssl?
     if SETTINGS[:ssl_ca] && !SETTINGS[:ssl_ca].empty?
       res.ca_file = SETTINGS[:ssl_ca]

I know this is inelegant, first of all it should probably use SETTINGS[:enc_read_timeout] or something easier to configure, but closer to the issue, I'm not sure why it takes so long for Foreman to ingest these facts. I tried tracing through the code but wasn't really successful in figuring it out.


Related issues

Related to Foreman - Bug #20929: Huge table "fact_names" makes Foreman crawlClosed2017-09-14

History

#1 Updated by Dominic Cleal over 4 years ago

  • Project changed from Foreman to Installer
  • Category changed from Facts to Foreman modules

The script already has a :timeout setting, which is set to read_timeout (and open_timeout in the latest version) when performing fact uploads. Shouldn't that change the same thing?

#2 Updated by David Ibarra over 4 years ago

Dominic Cleal wrote:

The script already has a :timeout setting, which is set to read_timeout (and open_timeout in the latest version) when performing fact uploads. Shouldn't that change the same thing?

Sort of. I don't claim to have advanced knowledge of the foreman code base, or even Ruby, but it seems that :timeout for node.rb controls a timeout for the block of code that gets the classification of the server, not the read_timeout for the Net::HTTP client that pushes the facts into the foreman API.

#3 Updated by David Ibarra over 4 years ago

I think I found the cause of the slowness, it seems like it's ActiveRecord:

app/models/host/base.rb:

      if iface.new_record? || iface.changed?
        logger.debug "Saving #{name} NIC for host #{self.name}" 
        result = iface.save

        unless result
          logger.warn "Saving #{name} NIC for host #{self.name} failed, skipping because:" 
          iface.errors.full_messages.each { |e| logger.warn " #{e}" }
        end

        result
      end

If I disable activerecord callbacks (iface.save to iface.save(:validate => false)) this thing goes way way faster. I'm talking like, we're going from 5 minutes to import these facts, to 14 seconds.

#4 Updated by Ewoud Kohl van Wijngaarden over 3 years ago

  • Related to Bug #20929: Huge table "fact_names" makes Foreman crawl added

Also available in: Atom PDF