Michael Hurn wrote:

With Puppet being able to read these parameters I think that it will reduce the workload of Foreman as an ENC. As Foreman will not need to provide the parameters that it configures in Hiera.

I have had a play with https://forge.puppet.com/crayfishx/hiera_mysql

It works at a basic level but has a few limitations. I.e. If SQL query only returns one value it is returned as a string. The schema needs a type column for each row. Like the Foreman lookup_keys.key_type column.

Initially my idea is that it can be upgraded it to access the Foreman lookup_keys & lookup_values tables.
But I expect that it won't be that simple. ;)

Although the hiera_mysql module was written for MySQL it looks like it can be converted to any supported database.

If I understand Puppet + Hiera and Foreman correctly I believe that it is possible to turn Foreman into an indirect ENC. With Foreman providing a GUI to the Puppet/Hiera parameters in a database. The Puppet servers (with a Hiera enabled smart proxy) should be able to work with Hiera as the ENC and still receive the benefits of using Foreman.

Thinking about this a bit more moving to a database enabled Hiera/ENC could help a lot with large Foreman/Puppet installations. (I'm giving this a working name of HieraDB.)

Hence I think that everything that Foreman provides a Puppet Server as an ENC should be moved into HieraDB.

See also Journey to High Availability https://theforeman.org/2015/12/journey_to_high_availability.html by Christopher Pisano.

From Christopher's post, we know that memcached helps the Foreman servers. As such I think that testing HieraDB with memcached should be on the QA list.

With this we would have two key building blocks:
1, Foreman -> WebUI/Reporting/Provisioning -> Foreman DB.
2, Puppet Server with HieraDB -> Foreman DB.

Do you have an opinion if my suggestion could work in your environment?

Additional building blocks could be:
3, PuppetCA.
4, PuppetDB.
5, Foreman DB. (The Foreman backend database PostgreSQL or MySQL.)

The foreman default install would then comprise the building blocks 1, 2, 3 & 5 above.

Michael Hurn wrote:

Michael Hurn wrote:

With Puppet being able to read these parameters I think that it will reduce the workload of Foreman as an ENC. As Foreman will not need to provide the parameters that it configures in Hiera.

I have had a play with https://forge.puppet.com/crayfishx/hiera_mysql

It works at a basic level but has a few limitations. I.e. If SQL query only returns one value it is returned as a string. The schema needs a type column for each row. Like the Foreman lookup_keys.key_type column.

Initially my idea is that it can be upgraded it to access the Foreman lookup_keys & lookup_values tables.
But I expect that it won't be that simple. ;)

Although the hiera_mysql module was written for MySQL it looks like it can be converted to any supported database.

If I understand Puppet + Hiera and Foreman correctly I believe that it is possible to turn Foreman into an indirect ENC. With Foreman providing a GUI to the Puppet/Hiera parameters in a database. The Puppet servers (with a Hiera enabled smart proxy) should be able to work with Hiera as the ENC and still receive the benefits of using Foreman.

Thinking about this a bit more moving to a database enabled Hiera/ENC could help a lot with large Foreman/Puppet installations. (I'm giving this a working name of HieraDB.)

Hence I think that everything that Foreman provides a Puppet Server as an ENC should be moved into HieraDB.

See also Journey to High Availability https://theforeman.org/2015/12/journey_to_high_availability.html by Christopher Pisano.

From Christopher's post, we know that memcached helps the Foreman servers. As such I think that testing HieraDB with memcached should be on the QA list.

With this we would have two key building blocks:
1, Foreman -> WebUI/Reporting/Provisioning -> Foreman DB.
2, Puppet Server with HieraDB -> Foreman DB.

Do you have an opinion if my suggestion could work in your environment?

Additional building blocks could be:
3, PuppetCA.
4, PuppetDB.
5, Foreman DB. (The Foreman backend database PostgreSQL or MySQL.)

The foreman default install would then comprise the building blocks 1, 2, 3 & 5 above.

I don't know the inner workings of any of this but I would assume that PostgreSQL support would be the best option, especially if we are talking about scale. I think this could be a worthwhile feature though because I know Smart-Class parameters are a huge performance hit on Foreman. Especially in large environments where lots of data has to be rendered and sent off to the Puppet masters. I'm not 100% sure how HieraDB actually works but a user friendly way to create Hiera data would still be needed for those not familiar with SQL. If the answer to that is through the webUI in Foreman then I think there needs to be better version control and logging of parameter overrides etc in order to get similar behavior as versioning a Hiera data structure in Git.