Bug #336

Facts not imported when running the task puppet:import:hosts_and_facts

Added by Gustavo Soares over 7 years ago. Updated almost 3 years ago.

Status:Closed
Priority:Normal
Assigned To:Ohad Levy
Category:Puppet integration
Target version:0.1-6
Difficulty: Bugzilla link:
Found in release: Pull request:
Story points-
Velocity based estimate-

Description

When running the task puppet:import:hosts_and_facts the facts related to each host found are not imported into the system and therefore no information is avaiable in the statistics page. Other problem related to this, is that the inventory functionality becomes unavaiable.

The following messages appears in the log

Processing StatisticsController#index (for x.x.x.x at 2010-06-16 16:10:10) [GET]
Parameters: {"action"=>"index", "controller"=>"statistics"}
Couldn't find Operatingsystem without an ID
Rendering template within layouts/standard
Completed in 17ms (View: 13, DB: 5) | 200 OK [http://abc.def.com/statistics]

Commenting the line self.respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values) inside the src/app/models/host.rb file, seems to solve the problem. However, I have not done any futher investigation of the collateral effects of this.

sample.yaml Magnifier (1.56 KB) Mohit Chawla, 07/15/2010 12:54 PM

sample_yaml.yaml Magnifier (2.9 KB) Gustavo Soares, 07/15/2010 06:10 PM

tvvvgslerp100.some-company.ads.yaml Magnifier - YAML File (2.92 KB) Henning Henkel, 08/12/2010 11:55 AM


Related issues

Duplicated by Foreman - Bug #391: No Inventory data within UI Duplicate 09/30/2010

Associated revisions

Revision 49f31762
Added by Ohad Levy over 7 years ago

Partial fix for refs #336, a host needs to be saved after new facts are
imported

History

#1 Updated by Mohit Chawla over 7 years ago

I am getting the same error when trying to view Statistics. But information under hosts, reports, audit & facts otherwise are rendered perfectly with the correct information being displayed.

production.log:

Parameters: {"action"=>"index", "controller"=>"statistics"}
Couldn't find Operatingsystem without an ID
Rendering template within layouts/standard
Completed in 79ms (View: 42, DB: 3) | 200 OK [https://localhost/statistics]

...and an error "No Inventory data has been found - add some hosts and facts and try again" is displayed on the web interface.

Haven't tried the fix suggested above.

#2 Updated by Ohad Levy over 7 years ago

  • Status changed from New to Need more information
  • Priority changed from High to Normal

I'm a bit puzzled how commenting this line solved the problem, that is the line which actually save the facts to the db....

do you have any error trace?

#3 Updated by Gustavo Soares over 7 years ago

The error trace that I have is the one that I use when I've created the issue.

When I were debuging this, I saw that the merge_fact were messing with the facts object. I've put some "puts" in the code and saw that the values of the fact became empty, after the execution of the line that I've commented. So, no facts were being actually imported to the db.

Sorry, I didn't included this information in the original post.

Ohad Levy wrote:

I'm a bit puzzled how commenting this line solved the problem, that is the line which actually save the facts to the db....

do you have any error trace?

#4 Updated by Ohad Levy over 7 years ago

hmm... does running the rake task produce any error? does it give any other input with --trace?

#5 Updated by Ohad Levy over 7 years ago

hmm...
can you try changing that line to:

    respond_to?("merge_facts") ? self.merge_facts(facts.values) : self.setfacts(facts.values)

#6 Updated by Gustavo Soares over 7 years ago

no error is produced when running the task.

Ohad Levy wrote:

hmm... does running the rake task produce any error? does it give any other input with --trace?

#7 Updated by Gustavo Soares over 7 years ago

ok, I'll do that and place here the output.

Ohad Levy wrote:

hmm...
can you try changing that line to:

[...]

#8 Updated by Gustavo Soares over 7 years ago

It hasn't worked. Here is the output.

[puppet@localhost models]$ rake puppet:import:hosts_and_facts dir=/mnt/puppet/yaml/facts RAILS_ENV="production"
(in /opt/puppet/puppet-foreman)
Importing from /mnt/puppet/yaml/facts
Importing abc.globoi.com
running importFact
os_name: CentOS
--------------
os_name:

I've put some "puts" in the code. As follow:

 def importFacts facts
    raise "invalid Fact" unless facts.is_a?(Puppet::Node::Facts)
    puts "running importFact" 
    os_name = fv(:operatingsystem)
    puts "os_name: #{os_name}" 
    puts "--------------" 

    # we are not importing facts for hosts in build state (e.g. waiting for a re-installation)
    raise "Host is pending for Build" if build
    time = facts.values[:_timestamp]
    time = time.to_time if time.is_a?(String)

    # we are not doing anything we already processed this fact (or a newer one)
    return true unless last_compile.nil? or (last_compile + 1.minute < time)

    self.last_compile = time
    # save all other facts - pre 0.25 it was called setfacts

    # Commented due to bug when importing facts
    #self.respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values)
    respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values)

    os_name = fv(:operatingsystem)
    puts "os_name: #{os_name}" 

    # we want to import other information only if this host was never installed via Foreman
    populateFieldsFromFacts if installed_at.nil?

Notice that in the output of the rake import command the fact for the os_name (as all the others) is empty after execution of "respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values)"

Cheers

Gustavo Soares wrote:

ok, I'll do that and place here the output.

Ohad Levy wrote:

hmm...
can you try changing that line to:

[...]

#9 Updated by Mohit Chawla over 7 years ago

I tried both of the fixes suggested above. And in my case they give the same output. Statistics do get displayed, but the OS is detected only for one machine. The other two are virtual machines on a separate host, and their OS is not displayed.

production.log

Processing StatisticsController#index (for ::1 at 2010-07-15 09:11:15) [GET]
  Parameters: {"action"=>"index", "controller"=>"statistics"}
skipped  as it has has no label
Rendering template within layouts/standard
Rendering statistics/index
Completed in 111ms (View: 53, DB: 17) | 200 OK [https://localhost/statistics]

#10 Updated by Mohit Chawla over 7 years ago

Ok, this will probably help you guys diagnose the exact issue. I tried 0.1-4 and by default, statistics are displayed (although os is not displayed for the virtual machines, on the brighter side though everything else is and I can create the OS for them easily!).

On the other hand, the nightly build/source/deb of 0.1-5 show same behavior as suggested by the bug.

#11 Updated by Ohad Levy over 7 years ago

Mohit Chawla wrote:

I tried both of the fixes suggested above. And in my case they give the same output. Statistics do get displayed, but the OS is detected only for one machine. The other two are virtual machines on a separate host, and their OS is not displayed.

production.log
[...]

did you try with this exact line: (note the "self")

    respond_to?("merge_facts") ? self.merge_facts(facts.values) : self.setfacts(facts.values)

if it doesnt work, please upload an example fact file (from /varl/lib/puppet/yaml/facts dir)

what is your puppet version?

#12 Updated by Mohit Chawla over 7 years ago

Yes, I am using the exact same line as you suggested.

I am attaching a fact file.

Puppet version is 0.25.5

#13 Updated by Gustavo Soares over 7 years ago

I am using puppet version 0.25.1.

I also have some virtual machines (XEN) and I am getting the Operating System value correct. What I don't get is the Model, in the Hosts page.

Mohit Chawla wrote:

Yes, I am using the exact same line as you suggested.

I am attaching a fact file.

Puppet version is 0.25.5

#14 Updated by Mohit Chawla over 7 years ago

Gustavo Soares wrote:

I am using puppet version 0.25.1.

I also have some virtual machines (XEN) and I am getting the Operating System value correct. What I don't get is the Model, in the Hosts page.

Mohit Chawla wrote:

Yes, I am using the exact same line as you suggested.

I am attaching a fact file.

Puppet version is 0.25.5

Hi Gustavo, can you check the value of the lsbdistrelease fact in one of your fact files ? If its something like *id001 or the like, then the host.rb needs to be slightly modified. You can check the file on github or in your directory, line number 278, as told by Ohad.

#15 Updated by Gustavo Soares over 7 years ago

Mohit Chawla wrote:

Gustavo Soares wrote:

I am using puppet version 0.25.1.

I also have some virtual machines (XEN) and I am getting the Operating System value correct. What I don't get is the Model, in the Hosts page.

Mohit Chawla wrote:

Hi Mohit! The fact that you asked me to check doesn't exist in my fact file.

I have attached an example here. I have manually edited the IP address and the mac address within the file.

Yes, I am using the exact same line as you suggested.

I am attaching a fact file.

Puppet version is 0.25.5

Hi Gustavo, can you check the value of the lsbdistrelease fact in one of your fact files ? If its something like *id001 or the like, then the host.rb needs to be slightly modified. You can check the file on github or in your directory, line number 278, as told by Ohad.

#16 Updated by Mohit Chawla over 7 years ago

Gustavo, that's probably where the problem lies as the logic in host.rb depends on the lsb related facts in some way. See here:

 os_name = fv(:operatingsystem)
    if orel = fv(:lsbdistrelease) || fv(:operatingsystemrelease)
      major, minor = orel.split(".")
      minor ||= "" 
      self.os = Operatingsystem.find_or_create_by_name_and_major_and_minor os_name, major, minor
    end

    modelname = fv(:productname) || fv(:model)
    self.model = Model.find_or_create_by_name(modelname.strip) if model.nil? and not modelname.empty?

from http://github.com/ohadlevy/foreman/blob/develop/app/models/host.rb

But then you have problem with the model not the os, either way the solution lies somewhere in there. :)

#17 Updated by Gustavo Soares over 7 years ago

I'm a bit puzzled... since the if with the lsbdistrelease.. fact has an "or" operator.. so I if it will get the value from operatingsystemrelease, which I do have in my yaml file.

Cheers,

Mohit Chawla wrote:

Gustavo, that's probably where the problem lies as the logic in host.rb depends on the lsb related facts in some way. See here:

[...]

from http://github.com/ohadlevy/foreman/blob/develop/app/models/host.rb

But then you have problem with the model not the os, either way the solution lies somewhere in there. :)

#18 Updated by Ohad Levy over 7 years ago

Gustavo Soares wrote:

I'm a bit puzzled... since the if with the lsbdistrelease.. fact has an "or" operator.. so I if it will get the value from operatingsystemrelease, which I do have in my yaml file.

Operatingsystem will not save if the value of the major is not numerical, if lsbrelease is present, but not a number, it will not save the os.

I'm guessing we need to check if lsbdistrelease is present and is a number, otherwise use the operatingsystemrelease.

#19 Updated by Mohit Chawla over 7 years ago

Here's the chain of actions I have followed:

1) Uncomment the following line in app/models/host.rb

self.respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values)

and then, do

rake puppet:import:hosts_and_facts RAILS_ENV=production 

I think this forces the PopulateFieldsFromFacts method to well, populate the fields.

Adding the following line and importing again, doesn't have any further effect:

    respond_to?("merge_facts") ? self.merge_facts(facts.values) : self.setfacts(facts.values)

2) Replace the following block:

 os_name = fv(:operatingsystem)
    if orel = fv(:lsbdistrelease) || fv(:operatingsystemrelease)
      major, minor = orel.split(".")
      minor ||= "" 
      self.os = Operatingsystem.find_or_create_by_name_and_major_and_minor os_name, major, minor
    end

with

 os_name = fv(:operatingsystem)
      orel = fv(:operatingsystemrelease)
      major = orel
      minor ||= "" 
      self.os = Operatingsystem.find_or_create_by_name_and_major_and_minor os_name, major, minor
    end

and in the operatingsystem.rb, comment the following lines as:

#validates_numericality_of :major
#validates_numericality_of :minor, :allow_nil => true, :allow_blank => true

And then importing facts. No effect.

3) Comment these lines in host.rb:

 return true unless last_compile.nil? or (last_compile + 1.minute < time)

    self.last_compile = time

and repeat the import process. Unpredictable behavior (that's because as you can see I am not able to debug the code properly).

Clearly I will do good if I just wait for the next release or learn at least some code! :)

#20 Updated by Gustavo Soares over 7 years ago

So, I've spent the whole day investigating this problem and learning some gotchas of the ruby and rails :)

1) New Server

When a fresh new server is shown in foreman interface, I "DO" need the line self.respond_to?("merge_facts") ? merge_facts(facts.values) : self.setfacts(facts.values) to
be able to import the facts to the database. Ohad, you were right. However, no information about the OS and Model are saved in the db.

After, runing the rake task over and over again.. I've noticed (I don't know why) that after the method that I mentioned above is called something happens to the fv method. Every call to this method (let's say os_name = fv(:operatingsystem)) returns a nil object. I've watched the rails log (debug=true) and I have seen the query that is made to the db. I've issued the query on the mysql and It works.

mysql> SELECT fact_values.value FROM `fact_values` INNER JOIN `fact_names` ON `fact_names`.id = `fact_values`.fact_name_id WHERE (`fact_values`.host_id = 504 AND (fact_names.name = 'operatingsystem')) LIMIT 1;
------ | value |
------ | CentOS |
------
1 row in set (0.00 sec)

I have absolutely know idea why this happens.. and I also lack some more ruby/rails experience to give a better answer to this. Since I need this problem solved, I've done this:

changed the method below from:

def self.importHostAndFacts yaml
facts = YAML::load yaml
return false unless facts.is_a?(Puppet::Node::Facts)
h=find_or_create_by_name(facts.name)
h.save(false) if h.new_record?
h.importFacts(facts)
end

to:

def self.importHostAndFacts yaml
facts = YAML::load yaml
return false unless facts.is_a?(Puppet::Node::Facts)
h=find_or_create_by_name(facts.name)
h.save(false) if h.new_record?
h.importFacts(facts)
h.populateFieldsFromFacts if h.installed_at.nil?
end

I know, I am calling the method populateFieldsFromFacts twice.. and this is ulgy.. but as I said before, this need further investigation.

With this ugly patch, I was able to save de OS on db. However, the model of the virtual machine were still not being saved.

2) Model for virtual machines

Some machines that I have are actually XEN virtual machines. Facter, doesn't return a model in this case. But, there is a fact called is_virtual, that is true when the machine is virtual.. and false otherwise. So, I have changed the method populateFieldsFromFacts to this http://pastie.org/1048020

I also need to learn some code :) I am more used to phython and django, but I do need to learn some ruby and rails!! :)

Cheers,
Gus

Mohit Chawla wrote:

Here's the chain of actions I have followed:

1) Uncomment the following line in app/models/host.rb

[...]

and then, do

[...]

I think this forces the PopulateFieldsFromFacts method to well, populate the fields.

Adding the following line and importing again, doesn't have any further effect:

[...]

2) Replace the following block:

[...]

with

[...]

and in the operatingsystem.rb, comment the following lines as:
[...]

And then importing facts. No effect.

3) Comment these lines in host.rb:

[...]

and repeat the import process. Unpredictable behavior (that's because as you can see I am not able to debug the code properly).

Clearly I will do good if I just wait for the next release or learn at least some code! :)

#21 Updated by Henning Henkel over 7 years ago

Hi Ohad,

as agreed on irc, here is the fact file from the rhel 5 machine.

Cheers

Henning

#22 Updated by Ohad Levy over 7 years ago

  • Category set to Puppet integration
  • Assigned To set to Ohad Levy
  • Target version set to 0.1-6

#23 Updated by Ohad Levy over 7 years ago

can anyone confirm that the attached fixed the issue of facts not being imported?

the lsb and operating system issue is still standing.

#24 Updated by Ohad Levy over 7 years ago

  • Status changed from Need more information to Closed

the patch above seems to resolve the issue, please reopen if any problem still occurs.

#25 Updated by apoeleh apoeleh over 5 years ago

Spam deleted by administrator.

#26 Updated by unisiisis unisiisis over 5 years ago

Spam deleted by administrator.

#27 Updated by apoeleh apoeleh over 5 years ago

Spam deleted by administrator.

#28 Updated by jaffa wify over 5 years ago

Spam deleted by administrator.

#29 Updated by nitin khare about 5 years ago

Spam deleted by administrator.

#30 Updated by nitin khare about 5 years ago

Spam deleted by administrator.

#31 Updated by nitin khare almost 5 years ago

Spam deleted by administrator.

#32 Updated by nitin khare almost 5 years ago

Spam deleted by administrator.

#33 Updated by nitin khare over 4 years ago

Spam deleted by administrator.

#34 Updated by kenvic99 kenvic99 over 4 years ago

Spam deleted by administrator.

#35 Updated by nitin khare over 4 years ago

Spam deleted by administrator.

#36 Updated by Liz Anderson over 3 years ago

Thanks for everything guys

#37 Updated by Anonymous almost 3 years ago

Spam deleted by administrator.

#38 Updated by Anonymous almost 3 years ago

Spam deleted by administrator.

#39 Updated by Anonymous almost 3 years ago

Spam deleted by administrator.

#40 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#41 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#42 Updated by abby john almost 3 years ago

Removed by a Redmine administrator.

#43 Updated by david lee almost 3 years ago

Removed by a Redmine administrator.

#44 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#45 Updated by david lee almost 3 years ago

Removed by a Redmine administrator.

#46 Updated by abby john almost 3 years ago

Removed by a Redmine administrator.

#47 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#48 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#49 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#50 Updated by david lee almost 3 years ago

Removed by a Redmine administrator.

#51 Updated by abby john almost 3 years ago

Removed by a Redmine administrator.

#52 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#53 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#54 Updated by abby john almost 3 years ago

Removed by a Redmine administrator.

#55 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#56 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#57 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#58 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#59 Updated by abby john almost 3 years ago

Removed by a Redmine administrator.

#60 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#61 Updated by max well almost 3 years ago

Removed by a Redmine administrator.

#62 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#63 Updated by zaib tyba almost 3 years ago

Removed by a Redmine administrator.

#64 Updated by under tacker almost 3 years ago

Removed by a Redmine administrator.

#65 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#66 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#67 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#68 Updated by under tacker almost 3 years ago

Removed by a Redmine administrator.

#69 Updated by boxer rulz almost 3 years ago

Removed by a Redmine administrator.

#70 Updated by zaib tyba almost 3 years ago

Removed by a Redmine administrator.

#71 Updated by zaib tyba almost 3 years ago

Removed by a Redmine administrator.

#72 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#73 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#74 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#75 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#76 Updated by Adam Daved almost 3 years ago

Removed by a Redmine administrator.

#77 Updated by albetjacko albetjacko almost 3 years ago

Removed by a Redmine administrator.

#78 Updated by jamesboded jamesboded almost 3 years ago

Removed by a Redmine administrator.

#79 Updated by deliciasjame deliciasjame almost 3 years ago

Removed by a Redmine administrator.

Also available in: Atom PDF