Jenkins » History » Version 4

« Previous - Version 4/53 (diff) - Next » - Current version
Dominic Cleal, 11/05/2013 09:58 PM


The Foreman project maintains its own Jenkins instance for continuous integration at

Pull request testing

Core Foreman projects have GitHub pull requests tested on our Jenkins instance so it's identical to the way we test the primary development branches themselves. Less significant projects (such as installer submodules) may use Travis CI.

PR jobs

Every project that needs PR testing requires two Jenkins jobs. Taking core Foreman as an example:

The PR test job takes the PR number parameter, downloads the patch from GitHub (by appending a .patch extension to the PR URL), applies it to the local checkout of the project and then builds as normal. This process means PRs are effectively rebased onto the current development branch before tests are run, rather than testing the branch as-is. GitHub tracks "mergeability" so we don't test PRs that can't be merged cleanly.

The results from these PR jobs are only stored for a few weeks, sufficient for reviews.

PR scanner

To initiate the PR tests, the test-pull-requests script is used to scan for open PRs, check they're mergeable and then trigger the Jenkins job. The script is a fork from the OpenShift upstream, enhanced in a few areas including changing from comment-based updates to the GitHub status API.

The script runs under the pull_request_scanner job under Jenkins and is set to run a few times every hour. It scans all configured projects for PRs and then exits, leaving the PR test jobs themselves to execute asynchronously.

The configuration files are deployed via our Puppet infrastructure for each project, and mostly just detail the GitHub repos, branches and Jenkins job names. These are managed in the slave foreman-infra module in slave itself and templates/.

After a PR test job completes, the Jenkins jobs are configured to build the PR scanner job again, which means immediately after the PR test results come in, the PR scanner script is able to update the status on the GitHub PR. A kind of feedback loop, if you will.

PR scanner repo hooks

In addition to regular scheduled runs of the PR scanner, hooks are added to the GitHub repositories to kick the PR scanner "build" when a PR is opened or synchronised.

They currently point to a very simple sinatra app running on OpenShift, which reads in the source repo from the hook payload and then runs the PR scanner build remotely via a POST request to Jenkins. This means PRs begin building within a minute or two of the PR being opened, giving better feedback to developers.

Adding a new project

For project "foo":

  1. create a test_foo job that tests the primary development branch, enable IRC notifications in post build
  2. clone to test_foo_pull_request
    • without IRC notifications
    • add pr_number parameter
    • add initial build step to download the patch and apply
  3. add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
  4. add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
  5. add pull_request hook to GitHub repo