History

Jenkins

The Foreman project maintains its own Jenkins instance for continuous integration at http://ci.theforeman.org/

It runs the following types of tests:

  • Unit tests on the main develop and recent stable branches of core Foreman projects
  • Pull request tests against core Foreman projects
  • Unit and pull request tests for many Foreman plugins
  • Nightly and release package building and release processes
  • Package smoke and integration testing ("systest")

Quick reference for maintainers

Current PR test jobs (used on Foreman itself) support these commands:

  • ok to test - run tests for an unknown user, if the code within the patch is not malicious
  • add to PR whitelist - automatically run all PRs in future from this user
  • [test STATUS-NAME], e.g. [test foreman] to re-run a particular set of tests

Older PR tests, those that use the status name "default", supports this command:

  • [test] - run tests for an unknown user, or re-run all tests

Quick reference for plugin maintainers

Foreman plugin testing

Foreman plugins are tested by adding the plugin to a Foreman checkout and running core tests, so it checks that existing behaviours still work and new plugin tests are run too. The test_plugin_matrix job copies the core jobs, but adds a plugin from a given git repo/branch and is usually used to test plugins in a generic way.

Each plugin should have a job defined in JJB that calls test_plugin_matrix here: http://ci.theforeman.org/view/Plugins/

Foreman plugin PR testing

To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_plugin_pull_request/

To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner. Also enable the prprocessor webhook on the GitHub repo for immediate builds.

Adding a new Foreman plugin

For a plugin "foreman_example", first create a job that tests the main (master or develop) branch.

  1. ensure plugin tests (if any) run when rake jenkins:unit is called, see the example plugin and testing a plugin for help
  2. create a foreman_example.yaml file in foreman-infra/JJB
  3. ensure the job is green by fixing bugs, installing dependencies etc.
  4. add hook to GitHub repo, see GitHub repo hook

Next, set up PR testing for the plugin:

  1. add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
  2. add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
  3. add project to PrProcessor repos.yaml config file with "pr_scanner: true"

An org admin must then:

  1. add PrProcessor webhook to GitHub repo
  2. add the repo to the Bots team with write access

Smart proxy plugin testing

Proxy plugins are tested like ordinary gems with tests run entirely from the plugin directory, installing the smart proxy as a dependency (via bundler's git support). The test_proxy_plugin_matrix job is usually used to test plugins in a generic way.

Each plugin should have a job defined in JJB that calls test_proxy_plugin_matrix here: http://ci.theforeman.org/view/Plugins/

Smart proxy plugin PR testing

To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_proxy_plugin_pull_request/

To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner. Also enable the prprocessor webhook on the GitHub repo for immediate builds.

Adding a new smart proxy plugin

For a plugin "smart_proxy_example", first create a job that tests the main (master or develop) branch.

  1. ensure plugin tests run when doing bundle install and rake test, see testing a plugin for help
    • if different branches rely on different versions of smart proxy, specify :branch in Gemfile on those branches
  2. create a smart_proxy_example.yaml file in foreman-infra/JJB
  3. ensure the job is green by fixing bugs, installing dependencies etc.
  4. add hook to GitHub repo, see GitHub repo hook

Next, set up PR testing for the plugin:

  1. add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
  2. add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
  3. add project to PrProcessor repos.yaml config file with "pr_scanner: true"

An org admin must then:

  1. add PrProcessor webhook to GitHub repo
  2. add the repo to the Bots team with write access

Job configurations

Testing develop

All repos with an associated job that tests their master/develop branch should have a hook added to the repo to trigger immediate builds.

To set up the hook, an org/repo admin must:

  1. View the repository settings
  2. Click Webhooks & Services
  3. Add service dropdown, select Jenkins (Git plugin)
  4. For the URL, enter http://ci.theforeman.org/

Pull request testing

Core Foreman projects have GitHub pull requests tested on our Jenkins instance so it's identical to the way we test the primary development branches themselves. Simpler and quieter projects (such as installer modules) should use Travis CI which supports PR testing and reduces the load on our own infrastructure.

Every project that needs PR testing has at least two Jenkins jobs. Taking core Foreman as an example:

The results from these PR jobs are only stored for a few weeks, sufficient for reviews. The PR jobs are set up in a few different ways, as it's changed over time:

  1. Version 3 jobs use the GitHub Pull Request Builder (GHPRB) and merge the branch to the project branch
  2. Version 2 jobs use the PR scanner and merge the branch to the primary project branch
  3. Version 1 jobs use the PR scanner and use git apply with the patch from the PR

Version 3 (GHPRB)

The GHPRB plugin uses webhooks installed on the repo to trigger a build, then it runs any job configured with the GHPRB trigger and a matching GitHub project set.

The plugin tests the latest commit on the PR branch only, it does not merge the PR with the base branch. The webhook may also trigger multiple jobs, and jobs may use different GitHub commit status names to easily test and report status on different types of tests.

PR jobs should be set up identically to primary branch tests, except for the SCM (which checks out ${sha1}) and to add the GHPRB trigger (see the github_pr macro in JJB).

To set up the hook, an org/repo admin goes to the repository settings, then Webhooks & Services and adds a webhook with these settings:

  • Payload URL: http://ci.theforeman.org/ghprbhook/
  • Content type: application/json
  • Secret: redacted
  • Events: Let me select individual events, Pull request, Issue comment

An org admin must then change the org teams:

  • Add the repository to the Bots team with write access

Version 2 (scanner + git URL/ref)

The PR test job takes the following parameters:

  • pr_git_url: URL to the repository containing the PR to clone
  • pr_git_ref: branch name in the above repository for the PR
  • pr_number: optional, for informational purposes only (e.g. tracking from a job back to a PR)

Under source code management in the job configuration, set up:

  1. Project git repository:
    • Repository URL: https://github.com/theforeman/foreman
  2. Branches to build: develop
  3. Additional Behaviours > Prune stale remote-tracking branches
  4. Additional Behaviours > Wipe out repository & force clone

As the first build step, add:

#!/bin/bash -ex
if [ -n "${pr_git_url}" ]; then
  git remote add pr ${pr_git_url}
  git fetch pr
  git merge pr/${pr_git_ref}
fi

Version 1 (scanner + .patch)

The PR test job takes the PR number parameter, downloads the patch from GitHub (by appending a .patch extension to the PR URL), applies it to the local checkout of the project and then builds as normal. This process means PRs are effectively rebased onto the current development branch before tests are run, rather than testing the branch as-is. GitHub tracks "mergeability" so we don't test PRs that can't be merged cleanly.

PR scanner (version 1, 2)

To initiate the PR tests, the test-pull-requests script is used to scan for open PRs, check they're mergeable and then trigger the Jenkins job. The script is a fork from the OpenShift upstream, enhanced in a few areas including changing from comment-based updates to the GitHub status API.

The script runs under the pull_request_scanner job under Jenkins and is set to run a few times every hour. It scans all configured projects for PRs and then exits, leaving the PR test jobs themselves to execute asynchronously.

The configuration files are deployed via our Puppet infrastructure for each project, and mostly just detail the GitHub repos, branches and Jenkins job names. These are managed in the slave foreman-infra module in slave itself and templates/.

After a PR test job completes, the Jenkins jobs are configured to build the PR scanner job again, which means immediately after the PR test results come in, the PR scanner script is able to update the status on the GitHub PR. A kind of feedback loop, if you will.

PR scanner repo hooks

In addition to regular scheduled runs of the PR scanner, a hook is added to the GitHub repositories to kick the PR scanner "build" when a PR is opened or synchronised.

See PrProcessor for information on enabling the prprocessor webhook. The repo must be listed in the prprocessor config file with "pr_scanner: true".

Adding a new generic project

For a project "foo":

  1. create a test_foo job that tests the primary development branch, enable IRC notifications in post build
  2. clone to test_foo_pull_request
    • without IRC notifications
    • add pr_number parameter
    • add initial build step to download the patch and apply
  3. add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
  4. add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
  5. add project to PrProcessor repos.yaml config file with "pr_scanner: true"

An org admin must then:

  1. add PrProcessor webhook to GitHub repo
  2. add the repo to the Bots team with write access

System testing with foreman-bats

Some system tests are performed on the complete all-in-one Foreman setup, which includes packages, the installer, the CLI and related components.

These tests are currently in the foreman-bats project and use the BATS test framework (based on bash).

They are intended to be smoke and integration tests only, not in depth testing of any component of the stack. Most components have their own unit tests, which are cheaper to execute and are run closer to where the code is developed, reducing the turnaround time for fixes.

systest jobs

The systest_* jobs in Jenkins run the system tests, with systest_foreman being the main parameterised job for executing them. Other jobs call this and it's used in two ways.

Package smoke testing

For both types of packages, the systest_foreman job is run as an intermediate test phase between the repo being generated and published to the public web site. The aim is to ensure that the packages install and the application starts up in its default configuration.

During official releases it's called by release_test, and for nightly packages it's called by release_nightly_test_rpm and release_nightly_test_deb. Once the tests have run, the test jobs trigger publishing jobs to push the packages to the public repositories.

Scenario testing

A set of further integration tests are run against the published packages for different scenarios. These usually run once or a couple of times a week against the published nightly packages - since they run on every OS, we don't run them too often due to limited infrastructure. They run against published packages as any failure shouldn't be enough to block the publishing of nightly packages, but should probably be considered as high priority for releases.

We currently test the following:

  • Hammer integration (fb-hammer-tests.bats): runs the hammer-tests suite
  • MySQL database support: installs Foreman with --foreman-db-type=mysql
  • Puppet integration (fb-puppet-tests.bats): checking that modules can be installed, imported and applied to hosts
  • Puppet nightlies integration: checking that Foreman installs and functions with nightly Puppet packages (currently disabled)
  • umask 077 (fb-setup-umask.bats): installs Foreman under a umask of 077

Vagrant setup

foreman-bats is very simple - it's a bash script that executes as root on a host, installs Foreman and tests the result. Since we don't want to install Foreman directly on slaves, this is run on the Rackspace public cloud, under a project account.

The Jenkins jobs use Vagrant to create hosts on Rackspace and to run the foreman-bats project on it. Vagrant is installed via the foreman-infra Puppet modules along with the vagrant-rackspace plugin. This launches a standard Rackspace image for the OS under test, Vagrant rsyncs the current directory (workspace) to it containing foreman-bats, and the job script then executes the foreman-bats test over vagrant ssh.

The same Vagrant setup can be used to run foreman-bats locally via vagrant-libvirt or other plugins.

CentOS QA service

  • This change is in development, not currently in use.

CentOS offer access to their QA CI infrastructure to Foreman and other projects, which gives on-demand access to run tests on physical hardware running CentOS. It consists of three main components:

  1. ci.centos.org - a Jenkins instance which we can manage jobs on
  2. foreman@slave01 - a user account on a slave which our jobs all run on
  3. Duffy - on-demand provisioning of physical test servers running CentOS, available from foreman@slave01 jobs

Replacing Rackspace Cloud with Duffy

Our CI today runs the systest_foreman job on a slave, which uses vagrant-rackspace to create a VM of each supported OS on Rackspace on which tests are executed. These tests could be moved to Duffy hardware, reducing our usage of the Rackspace account, so we can run more static slaves for unit tests, and allowing us to run systest jobs more frequently or in more combinations (inc. nested virt).

Since Duffy is CentOS-only, we can use libvirt (preferably with vagrant-libvirt) to access each of our supported OSes. Vagrant could be installed on the Duffy-provisioned host, or once packaged via the SCLo SIG, even onto slave01 to save time. Suggested plan:

  1. Create systest_foreman_duffy on CentOS CI, accepting all the usual parameters, preferably via JJB
  2. systest_foreman_duffy has three build tasks:
    1. Execute shell (Python?): request new host from Duffy with /Node/get (example script)
    2. Execute shell: run usual vagrant up commands on remote Duffy host over SSH
    3. Post-build shell task: destroy the Duffy built host
  3. systest_foreman_* or release_test_* jobs on Foreman CI can call systest_foreman_duffy remotely via the remote trigger plugin

Managing jobs

Jenkins itself is deployed onto one master VM from foreman-infra. Jobs are maintained in two ways:

  1. Updated by hand - some developers have accounts to log in and modify jobs
  2. Jenkins Job Builder - a set of YAML configuration files in foreman-infra that generate jobs

New jobs should be written via Jenkins Job Builder if possible.

Jenkins Job Builder

Jenkins Job Builder (JJB) is an OpenStack tool for generating Jenkins job definitions (an XML file) from a set of YAML job descriptions, which we store in version control.

Puppet deploys these onto our Jenkins server (a recursive file copy) and when they change, it runs the JJB tool to update the jobs in the live instance. It also refreshes them daily to overwrite manual changes.

We use a fork of JJB with some extra support for plugins we use:

To test it:

  1. check out the above JJB repo and local branch
  2. recommended: create a Python virtualenv and activate it
    1. virtualenv jjb
    2. source jjb/bin/activate
  3. easy_install .
  4. check out foreman-infra
  5. cd foreman-infra/puppet/modules/jenkins_job_builder/files
  6. jenkins-jobs -l debug test -r . -o /tmp/jobs

JJB will populate /tmp/jobs with the proposed XML files.

Useful resources:

  1. Job definitions, templates etc.
  2. Modules, e.g. SCM, publishers, builders

Slaves

Configuration management

All slaves are maintained through our own Foreman instance using Puppet. The Foreman instance has a host group called "Base/Builders" and "Base/Builders/Red Hat" and "Base/Builders/Debian" which have the "slave" and other classes assigned to them.

https://github.com/theforeman/foreman-infra/tree/master/puppet/modules contains the source for all Puppet modules.

Slave requirements

  • CentOS 7
    • Clean, minimal base installation or the option to reinstall it
  • 2GB of RAM per vCPU (4 vCPU + 8GB RAM is typical)
  • 60GB disk (minimum), SSD preferred
  • ~20GB/month bandwidth
  • Public facing IP address
  • Root access

Configuring a new slave

If using Rackspace, start the new server via the web UI or the rumm CLI utility. Ensure you select:

  • Image: CentOS 7 (PVHVM)
  • Flavor: 4 GB General Purpose v1

Flavors with separate root/data disks only: set up the data partition for the Jenkins workspace:

  1. mkfs.ext4 -L data /dev/xvde1
  2. echo "LABEL=data /var/lib/workspace ext4 defaults,noatime 1 2" >> /etc/fstab
  3. mkdir /var/lib/workspace && mount /var/lib/workspace

Then:

  1. Ensure EPEL is configured: epel-release
  2. Ensure yum.pl.com is configured: puppet5-release
  3. yum -y install puppet-agent
  4. echo "server = puppetmaster.theforeman.org" >> /etc/puppetlabs/puppet/puppet.conf
  5. Relog into the machine so that the 'puppet' command is available
  6. puppet agent --test
  7. Sign the certificate on the puppetmaster or via Foreman
  8. puppet agent --test
  9. Set the host group to "Base/Builders/Red Hat" in Foreman
  10. Run puppet agent --test twice (second run is important, due to the rvm module behaviour)
  11. Add the node by IP address to Jenkins, copying an existing slave (except slave01.rackspace and slave02.rackspace)