Feature #4528

Support Facter 2 structured facts

Added by Greg Sutcliffe over 3 years ago. Updated over 1 year ago.

Status:Closed
Priority:Normal
Assigned To:Dominic Cleal
Category:Facts
Target version:-
Difficulty: Bugzilla link:1132402
Found in release: Pull request:https://github.com/theforeman/foreman/pull/3613
Story points-
Velocity based estimate-
Release1.13.0Release relationshipAuto

Description

Facter 2 is at RC1, and while it has no structered facts by default, it will have in the future, and people will start writing their own probably even sooner.

I quickly looked into this using the foreman_chef plugin. I came up with this:

1. Add 'plainify' and 'get_key' to node.rb from chef_foreman_handler, and change 'build_body' to call:

{'facts' => plainify(puppet_facts['values'].merge({:_type => 'foreman_chef'})).flatten.inject(&:merge), 'name' => hostname, 'certname' => certname}

2. Add foreman_chef plugin to Foreman
3. Fix https://github.com/theforeman/foreman/blob/develop/app/services/fact_importer.rb#L79 to not drop Integers
4. (optional) Handle changes of fact type. Currently if you upload "foo: bar" and then upload "foo: { "bar" => "quux" }" it won't display properly due to the old string-type fact in the db.

We'll probably want to move the hash-handling code out of foreman_chef and into core. I'd also consider investigating the way we compress and uncompress the facts hash with '::' vs just sending/parsing the whole hash directly.


Related issues

Related to Foreman - Feature #2293: Support structured facts Duplicate 03/08/2013
Related to Foreman - Feature #3339: Support for nested facts and specialised plugins Closed 10/17/2013
Related to Foreman - Feature #7525: Report volume sizes via facter Closed 09/18/2014
Related to Discovery - Feature #3860: Implement fully automated provisioning Closed 12/12/2013
Related to Foreman - Tracker #8447: Puppet 4 support Closed 11/19/2014
Related to Foreman - Feature #15522: Support for array/multi-valued facts New 06/27/2016
Related to Salt - Refactor #15591: Use new structured facts parser New 07/06/2016
Related to Foreman - Bug #16576: Structured facts importer is extremely slow Closed 09/15/2016
Duplicated by Foreman - Bug #6414: structured facts not displayed in foreman Duplicate 06/27/2014
Duplicated by Foreman - Feature #12703: Layout aggregated fact value display format Duplicate 12/04/2015

Associated revisions

Revision c696f740
Added by Dominic Cleal over 1 year ago

fixes #4528 - import structured Puppet facts

A new structured fact importer converts nested hashes into the
FactName::SEPARATOR layout used for internal structured fact storage and
extends the regular fact importer to create the hierarchy of FactNames.
This can be used by other plugins requiring structured storage. It does
not support storage of arrays beyond current conversion to strings.

History

#1 Updated by Dominic Cleal over 3 years ago

  • Subject changed from Support structured facts to Support Facter 2 structured facts

#2 Updated by Dominic Cleal over 3 years ago

#3 Updated by Dominic Cleal over 3 years ago

  • Related to Feature #3339: Support for nested facts and specialised plugins added

#4 Updated by Greg Sutcliffe over 3 years ago

  • Description updated (diff)

#5 Updated by Dominic Cleal over 3 years ago

  • Duplicated by Bug #6414: structured facts not displayed in foreman added

#6 Updated by Stephen Benjamin over 3 years ago

ForemanChef's code "just works" for Salt's structured facts as well (after plainifying). It would be awesome to have a nested fact importer in Core that could just take the hash.

#7 Updated by Marek Hulán over 3 years ago

Let's merge it into core :-) Note that facts should still be scoped by name so we should add ForemanSalt::FactImporter inheriting from ForemanChef::FactImporter or ideally extract the generic part to some new NestedFactImporter class and let inherit both from it. Important methods that would remain importer specific: fact_name_class and support_background. Importer from ForemanChef is also lacking tests at the moment so it should be covered first.

#8 Updated by Stephen Benjamin about 3 years ago

  • Bugzilla link set to 1132402

#9 Updated by Lukas Zapletal about 3 years ago

#10 Updated by Lukas Zapletal about 3 years ago

Guys I need to get some integers in (discovery, hard disk volume sizes), can I post a patch to simply convert the number to string? See the referenced issue.

#11 Updated by Lukas Zapletal about 3 years ago

For the record, I filed a PR to convert non-string values to strings so it prevents from failure. We might want to do something smarter than this in this ticket, but since Staypuft is blocked by this I propose this solution: https://github.com/theforeman/foreman_discovery/pull/90

#12 Updated by Lukas Zapletal about 3 years ago

  • Related to Feature #3860: Implement fully automated provisioning added

#13 Updated by Lukas Zapletal about 3 years ago

Bumping up the priority a bit, if this is implemented, our Discovery rule engine is much more powerful.

#14 Updated by Joshua Hoblitt almost 3 years ago

I just smacked into this in 1.6.3. Not even string structured facts are queryable via the gui.

#15 Updated by Jeff Sparrow over 2 years ago

This has been open for sometime and puppet has really pushed to using structured data facts more often, within their future parser it makes things much easier. Hope this is included very soon as hashes and arrays are fairly common and should be supported

#16 Updated by Sven Milinski over 2 years ago

Hello,

can anyone tell me when Foreman is able to manage facter 2 array/hash values ?
Is it on the roadmap ?

Would be nice if someone can update this issue

Thanks

#17 Updated by Jeff Sparrow about 2 years ago

Any update? Not having structured facts in foreman is getting to be a big hassle to get around. Would be nice to get an update, its been almost a year. Thanks.

#18 Updated by Dominic Cleal almost 2 years ago

  • Duplicated by Feature #12703: Layout aggregated fact value display format added

#19 Updated by Michael Moll over 1 year ago

#20 Updated by Maxim Burgerhout over 1 year ago

Doesn't the foreman_ansible implementation by @dLobatog have nice structured facts already? It might be possible to use that same code?

#21 Updated by Daniel Lobato Garcia over 1 year ago

Yup - it's also partly taken from Salt so hopefully we can provide this from core to all plugins.

#22 Updated by Dominic Cleal over 1 year ago

  • Status changed from New to Assigned
  • Assigned To set to Dominic Cleal

#23 Updated by Dominic Cleal over 1 year ago

  • Related to Feature #15522: Support for array/multi-valued facts added

#24 Updated by Dominic Cleal over 1 year ago

Tickets relating to handling of non-structured, non-string facts (ints, bools) which are mentioned above:

  • #7525 changed the importer to handle non-string facts by converting their values to strings, this helps notably with integer facts.
  • #11150 changed fact searching to support integer comparisons, since Foreman stores integers as strings internally.
  • I don't propose changing the storage of non-string facts in Foreman, it's sufficient to store them as strings and convert them when used (e.g. when searching).

Tickets relating to handling of array values:

  • #15522 filed to enable storage of multiple fact values for arrays, instead of storing them as a string (e.g. ["a", "b"])

Tickets relating to handling of structured/nested/hash facts:

  • #3339 added support for structured facts - that is, facts whose value is a hash of more facts
  • This ticket will be used only to update the Puppet fact importer to store structured facts/hashes of facts.

#25 Updated by The Foreman Bot over 1 year ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3613 added

#26 Updated by Stephen Benjamin over 1 year ago

#27 Updated by Dominic Cleal over 1 year ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#28 Updated by Dominic Cleal over 1 year ago

  • Release set to 1.13.0

#29 Updated by Dominic Cleal about 1 year ago

  • Related to Bug #16576: Structured facts importer is extremely slow added

Also available in: Atom PDF