Scripted processes¶

An area that Puppet/foreman currently is weak in, and can't really handle, is in creating, launching, and monitoring scripts and one-time run Puppet manifests for various deployment style tasks.  I propose that we extend and improve Foreman to fill this gap by effectively copying the functionality of Controltier (www.controltier.org) and Rundeck (www.rundeck.org) but significantly improving upon both tools by significantly leveraging the power of Puppet/foreman to significantly improve the user experience and tighten the integration between the two problem splaces.

Process¶

At the heart of this enhancement is the ability to create processes that carry out sequential steps in order to perform an operations task.  The example of this that I will use throughout this process is an existing script of mine (modified to hide details) within Rundeck:

function dependencies () { 
sudo yum install -y svn php make 
} 

function build () { 
sudo rm -r @option.location@/* 
sudo rm -r /tmp/build 
mkdir /tmp/build ; cd /tmp/build 
svn checkout https://SVN/ --username apenney --password fake --non-interactive 
cd /tmp/build/SVN/src/ 
make 
cp -R a b c d /tmp/build/SVN/build/htdocs 
sudo cp -R /tmp/build/SVN/build/htdocs/* @option.location@ 
} 

function cleanup () { 
rm -rf /tmp/build 
} 

dependencies 
build 
cleanup

sudo rm /tmp/ui-config.pp 

cat > /tmp/ui-config.pp <<EOF 
file { "/var/www/html/server/smsconfig.inc": 
ensure => present, 
source => "puppet://puppet/modules/project/var/www/html/server/@option.role@/smsconfig.inc", 
} 
EOF 

sudo -s puppet apply --modulepath=/etc/puppet/modules/@option.role@/ /tmp/ui-config.pp 
sudo rm /tmp/ui-config.pp

As you can see from the two scripts this checks code out of SVN and builds it correctly then moves it into place.  The next step does a one-time checkout of the configuration files (there are others in the original script) and then leaves the application unmanaged by Puppet so that developers can work from there.  While this might not be the best example (as it could be modelled in Puppet) you can imagine more sophisticated examples.

Creating new processes¶

For this functionality I would like a separate configuration called Processes and under here I could define a process.  A process is defined as a sequence of steps and these can be either new steps or references to existing steps.  Once you choose to add a new step you are  directed to choose between two options (to begin) called Puppet process and Script process.  If you choose Puppet process you are given a space in which to write a Puppet manifest which will only be applied when manually called by a process.  This keeps it outside the regular modules tree and ensures it won't be ran again in a regular run.  (As there's no easy way to write a class and only call it one time in Puppet currently we have to do things this way).  Once created this is executed by Foreman connecting over a pre-defined SSH user and creating a temporary .pp file, running sudo puppet apply file, then cleaning up afterwards.  Ideally we would find a way to leverage existing reports to provide results to the user.

The second type of step is the Scripted Process.  This is pretty much the same as the previous concept by instead it builds a temporary .sh script on the host and runs that instead.  (Maybe give the user an option to run the entire script via sudo or not to make life easy).  Again this will either use reports or some other method to provide detailed information on how execution went.  Rundeck allows you to see real-time tail information as well as summaries built in various ways.  For the initial feature it would be enough to just get the output at the end of the job displayed to the user.

Options¶

Here's where things get a little more complicated.  It should be possible to define options as part of various steps that can be set by the user when they choose to trigger a process.  It displays the job and the various options for them to trigger.  I haven't thought this part fully through because I would like to, if possible, leverage all the awesome work Ohad has already done on options throughout foreman.  It should inherit all the parameters a node has when you select where to run the job and use those for defaults for example.  Like Rundeck it would be nice to define a pre-existing list of options or a text box for each option (so I can select /var/www/html as my destination in a text box, or prehaps select an environment from the drop-down).  Where possible we should allow users to select options from information Foreman already has.  I would also like to leverage facter in an easy fashion so that within my script I can simply call out something like x = $facter.operatingsystem as a bash if statement and have the script dynamically fill in all the various $facter calls.

Constraints¶

Unlike Rundeck we should use all the awesome of Foreman to apply process constraints.  I would like to be able to force the inclusion of a Puppet class if a job is to be run.  Perhaps you just choose the classes that must existing on the node level for a job to be valid and then display invalid jobs in a different format.  You get the idea, this would be a massive victory over Rundeck and be incredibly powerful.  I would like to be able to constrain a process by things like "Must contain project::ui, project::core, be x86_64, have > 2G of ram and have a report from the node in the last 24 hours (to ensure Puppet is working)).  Things like that would be extremely useful in my environment.

Dependencies¶

If we were getting fancy it would be nice to define process dependencies - a process that configures an application would not be allowed to run if there was no record of the build process having been ran on a node.  I'm not sure how clever this could get but simple 'block unless x,y,z' like the contraints.  Maybe this just belongs up there in constraints.

Launching Processes¶

Here we would leverage the node page as well as the multi select node options to trigger processes.  When you drill into a node it would have an extra box called processes that would list all processes allowed by the constraints.  You could then select and trigger multiple processes (order here would count\!) and fire them off.  I think this would best be an option off the main node page (where the report currently is) that says Processes.  Doesn't really fit into Edit.

Monitoring Processes¶

This would ideally hook into reports in some fashion.  I have no idea how this would look but you'd want to be able to see the output of the entire ssh session from start to finish I think.