Project

General

Profile

Jenkins » History » Version 33

Dominic Cleal, 06/03/2015 07:18 AM
optional tests

1 1 Dominic Cleal
h1. Jenkins
2 1 Dominic Cleal
3 1 Dominic Cleal
The Foreman project maintains its own Jenkins instance for continuous integration at http://ci.theforeman.org/
4 1 Dominic Cleal
5 11 Dominic Cleal
{{toc}}
6 11 Dominic Cleal
7 23 Dominic Cleal
h2. Managing jobs
8 23 Dominic Cleal
9 23 Dominic Cleal
Jenkins itself is deployed onto one master VM from foreman-infra.  Jobs are maintained in two ways:
10 23 Dominic Cleal
11 23 Dominic Cleal
# Updated by hand - some developers have accounts to log in and modify jobs
12 23 Dominic Cleal
# Jenkins Job Builder - a set of YAML configuration files in foreman-infra that generate jobs
13 23 Dominic Cleal
14 23 Dominic Cleal
New jobs should be written via Jenkins Job Builder if possible.
15 23 Dominic Cleal
16 23 Dominic Cleal
h3. Jenkins Job Builder
17 23 Dominic Cleal
18 23 Dominic Cleal
"Jenkins Job Builder":http://docs.openstack.org/infra/jenkins-job-builder/ (JJB) is an OpenStack tool for generating Jenkins job definitions (an XML file) from a set of YAML job descriptions, which we store in version control.
19 23 Dominic Cleal
20 23 Dominic Cleal
* "JJB YAML files in foreman-infra":https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/config
21 23 Dominic Cleal
22 23 Dominic Cleal
Puppet deploys these onto our Jenkins server (a recursive file copy) and when they change, it runs the JJB tool to update the jobs in the live instance.  It also refreshes them daily to overwrite manual changes.
23 23 Dominic Cleal
24 23 Dominic Cleal
We use a fork of JJB with some extra support for plugins we use:
25 23 Dominic Cleal
26 23 Dominic Cleal
* "theforeman/jenkins-job-builder, local branch":https://github.com/theforeman/jenkins-job-builder/tree/local
27 23 Dominic Cleal
28 23 Dominic Cleal
To test it:
29 23 Dominic Cleal
30 23 Dominic Cleal
# check out the above JJB repo and local branch
31 23 Dominic Cleal
# recommended: "create a Python virtualenv and activate it":https://virtualenv.pypa.io/en/latest/userguide.html#usage
32 23 Dominic Cleal
## @virtualenv jjb@
33 23 Dominic Cleal
## @source jjb/bin/activate@
34 23 Dominic Cleal
# @easy_install .@
35 23 Dominic Cleal
# check out foreman-infra
36 23 Dominic Cleal
# @cd foreman-infra/puppet/modules/jenkins_job_builder/files@
37 23 Dominic Cleal
# @jenkins-jobs -l debug test -r . -o /tmp/jobs@
38 23 Dominic Cleal
39 23 Dominic Cleal
JJB will populate /tmp/jobs with the proposed XML files.
40 23 Dominic Cleal
41 23 Dominic Cleal
Useful resources:
42 23 Dominic Cleal
43 23 Dominic Cleal
# "Job definitions, templates etc.":http://docs.openstack.org/infra/jenkins-job-builder/definition.html
44 23 Dominic Cleal
# "Modules, e.g. SCM, publishers, builders":http://docs.openstack.org/infra/jenkins-job-builder/definition.html#modules
45 23 Dominic Cleal
46 31 Dominic Cleal
h2. GitHub repo hook
47 31 Dominic Cleal
48 31 Dominic Cleal
All repos with an associated job that tests their master/develop branch should have a hook added to the repo to trigger immediate builds.
49 31 Dominic Cleal
50 32 Dominic Cleal
To set up the hook, an org/repo admin goes to the repository settings, then _Webhooks & Services_, _Add service_ and select _Jenkins (Git plugin)_.  For the URL, enter @http://ci.theforeman.org/@
51 31 Dominic Cleal
52 1 Dominic Cleal
h2. Pull request testing
53 1 Dominic Cleal
54 31 Dominic Cleal
Core Foreman projects have GitHub pull requests tested on our Jenkins instance so it's identical to the way we test the primary development branches themselves.  Simpler and quieter projects (such as installer modules) should use Travis CI which supports PR testing and reduces the load on our own infrastructure.
55 1 Dominic Cleal
56 1 Dominic Cleal
h3. PR jobs
57 1 Dominic Cleal
58 1 Dominic Cleal
Every project that needs PR testing requires two Jenkins jobs.  Taking core Foreman as an example:
59 1 Dominic Cleal
60 1 Dominic Cleal
* Test job for the main development branch (develop or master): "test_develop":http://ci.theforeman.org/job/test_develop/
61 7 Dominic Cleal
* Test job taking parameters about the PR, which is merged and tested: "test_develop_pull_request":http://ci.theforeman.org/job/test_develop_pull_request/
62 1 Dominic Cleal
63 7 Dominic Cleal
The results from these PR jobs are only stored for a few weeks, sufficient for reviews.
64 7 Dominic Cleal
65 7 Dominic Cleal
The jobs are being transitioned at the moment from an old style, which used basic @git apply@ + patches to a proper merge of the branch under test to the primary project branch, which is far more reliable.
66 7 Dominic Cleal
67 7 Dominic Cleal
h4. Old style (.patch)
68 7 Dominic Cleal
69 1 Dominic Cleal
The PR test job takes the PR number parameter, downloads the patch from GitHub (by appending a .patch extension to the PR URL), applies it to the local checkout of the project and then builds as normal.  This process means PRs are effectively rebased onto the current development branch before tests are run, rather than testing the branch as-is.  GitHub tracks "mergeability" so we don't test PRs that can't be merged cleanly.
70 1 Dominic Cleal
71 7 Dominic Cleal
h4. New style (git URL/ref)
72 7 Dominic Cleal
73 7 Dominic Cleal
The PR test job takes the following parameters:
74 7 Dominic Cleal
75 9 Dominic Cleal
* pr_git_url: URL to the repository containing the PR to clone
76 9 Dominic Cleal
* pr_git_ref: branch name in the above repository for the PR
77 7 Dominic Cleal
* pr_number: optional, for informational purposes only (e.g. tracking from a job back to a PR)
78 7 Dominic Cleal
79 7 Dominic Cleal
Under source code management in the job configuration, set up:
80 7 Dominic Cleal
81 8 Dominic Cleal
# Project git repository:
82 8 Dominic Cleal
#* Repository URL: @https://github.com/theforeman/foreman@
83 7 Dominic Cleal
# Branches to build: @develop@
84 8 Dominic Cleal
# Additional Behaviours > Prune stale remote-tracking branches
85 8 Dominic Cleal
# Additional Behaviours > Wipe out repository & force clone
86 8 Dominic Cleal
87 8 Dominic Cleal
As the first build step, add:
88 8 Dominic Cleal
89 8 Dominic Cleal
<pre>
90 8 Dominic Cleal
#!/bin/bash -ex
91 10 Dominic Cleal
if [ -n "${pr_git_url}" ]; then
92 10 Dominic Cleal
  git remote add pr ${pr_git_url}
93 10 Dominic Cleal
  git fetch pr
94 10 Dominic Cleal
  git merge pr/${pr_git_ref}
95 10 Dominic Cleal
fi
96 8 Dominic Cleal
</pre>
97 1 Dominic Cleal
98 1 Dominic Cleal
h3. PR scanner
99 1 Dominic Cleal
100 2 Dominic Cleal
To initiate the PR tests, the "test-pull-requests":https://github.com/theforeman/test-pull-requests script is used to scan for open PRs, check they're mergeable and then trigger the Jenkins job.  The script is a fork from the OpenShift upstream, enhanced in a few areas including changing from comment-based updates to the GitHub status API.
101 2 Dominic Cleal
102 2 Dominic Cleal
The script runs under the "pull_request_scanner":http://ci.theforeman.org/job/pull_request_scanner/ job under Jenkins and is set to run a few times every hour.  It scans all configured projects for PRs and then exits, leaving the PR test jobs themselves to execute asynchronously.
103 2 Dominic Cleal
104 2 Dominic Cleal
The configuration files are deployed via our Puppet infrastructure for each project, and mostly just detail the GitHub repos, branches and Jenkins job names.  These are managed in the "slave foreman-infra module":https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/slave in slave itself and templates/.
105 3 Dominic Cleal
106 3 Dominic Cleal
After a PR test job completes, the Jenkins jobs are configured to build the PR scanner job again, which means immediately after the PR test results come in, the PR scanner script is able to update the status on the GitHub PR.  A kind of feedback loop, if you will.
107 3 Dominic Cleal
108 3 Dominic Cleal
h3. PR scanner repo hooks
109 3 Dominic Cleal
110 3 Dominic Cleal
In addition to regular scheduled runs of the PR scanner, hooks are added to the GitHub repositories to kick the PR scanner "build" when a PR is opened or synchronised.
111 3 Dominic Cleal
112 3 Dominic Cleal
They currently point to a very simple sinatra app running on OpenShift, which reads in the source repo from the hook payload and then runs the PR scanner build remotely via a POST request to Jenkins.  This means PRs begin building within a minute or two of the PR being opened, giving better feedback to developers.
113 3 Dominic Cleal
114 16 Dominic Cleal
The app is prprocessor, hosted on the project's OpenShift account and the source is available here: https://github.com/theforeman/prprocessor (please keep this in sync with OpenShift's repo)
115 15 Dominic Cleal
116 29 Dominic Cleal
To set up the hook, an org/repo admin goes to the repository settings, then Webhooks & Services and adds a webhook with these settings:
117 29 Dominic Cleal
118 29 Dominic Cleal
* Payload URL: @http://prprocessor-theforeman.rhcloud.com/pull_request@
119 29 Dominic Cleal
* Content type: @application/json@
120 29 Dominic Cleal
* Secret: copy value of GITHUB_SECRET_TOKEN from @rhc env list -a prprocessor -n theforeman@
121 29 Dominic Cleal
* Events: _Let me select individual events_, _Pull Request_
122 29 Dominic Cleal
123 25 Dominic Cleal
h3. Adding a new generic project
124 4 Dominic Cleal
125 24 Dominic Cleal
For a project "foo":
126 4 Dominic Cleal
127 4 Dominic Cleal
# create a test_foo job that tests the primary development branch, enable IRC notifications in post build
128 4 Dominic Cleal
# clone to test_foo_pull_request
129 4 Dominic Cleal
#* without IRC notifications
130 4 Dominic Cleal
#* add pr_number parameter
131 1 Dominic Cleal
#* add initial build step to download the patch and apply
132 24 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
133 24 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
134 24 Dominic Cleal
# add pull_request hook to GitHub repo
135 24 Dominic Cleal
136 25 Dominic Cleal
h3. Foreman plugin testing
137 1 Dominic Cleal
138 25 Dominic Cleal
Foreman plugins are tested by adding the plugin to a Foreman checkout and running core tests, so it checks that existing behaviours still work and new plugin tests are run too.  The "test_plugin_matrix job":http://ci.theforeman.org/job/test_plugin_matrix/ copies the core jobs, but adds a plugin from a given git repo/branch and is usually used to test plugins in a generic way.
139 25 Dominic Cleal
140 25 Dominic Cleal
Each plugin should have a job defined in JJB that calls test_plugin_matrix here: http://ci.theforeman.org/view/Plugins/
141 25 Dominic Cleal
142 25 Dominic Cleal
h4. Foreman plugin PR testing
143 25 Dominic Cleal
144 25 Dominic Cleal
To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_plugin_pull_request/
145 25 Dominic Cleal
146 25 Dominic Cleal
To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner.  Also enable the prprocessor webhook on the GitHub repo for immediate builds.
147 25 Dominic Cleal
148 25 Dominic Cleal
h4. Adding a new Foreman plugin
149 25 Dominic Cleal
150 27 Dominic Cleal
For a plugin "foreman_example", first create a job that tests the main (master or develop) branch.
151 17 Dominic Cleal
152 33 Dominic Cleal
# ensure plugin tests (if any) run when @rake jenkins:unit@ is called, see "the example plugin":https://github.com/theforeman/foreman_plugin_template/ and "testing a plugin":http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin#Testing for help
153 17 Dominic Cleal
# create a foreman_example.yaml file in foreman-infra/JJB
154 17 Dominic Cleal
#* https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/config/yaml/jobs/plugins
155 1 Dominic Cleal
#* This will create a "test_plugin_foreman_example_master" job in Jenkins to test the master branch.
156 1 Dominic Cleal
# ensure the job is green by fixing bugs, installing dependencies etc.
157 31 Dominic Cleal
# add hook to GitHub repo, see "GitHub repo hook":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#GitHub-repo-hook
158 27 Dominic Cleal
159 27 Dominic Cleal
Next, set up PR testing for the plugin:
160 27 Dominic Cleal
161 17 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
162 17 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
163 30 Dominic Cleal
# add pull_request hook to GitHub repo, see "PR scanner repo hooks":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#PR-scanner-repo-hooks
164 17 Dominic Cleal
165 28 Dominic Cleal
h3. Smart proxy plugin testing
166 28 Dominic Cleal
167 28 Dominic Cleal
Proxy plugins are tested like ordinary gems with tests run entirely from the plugin directory, installing the smart proxy as a dependency (via bundler's git support).  The "test_proxy_plugin_matrix job":http://ci.theforeman.org/job/test_proxy_plugin_matrix/ is usually used to test plugins in a generic way.
168 28 Dominic Cleal
169 28 Dominic Cleal
Each plugin should have a job defined in JJB that calls test_proxy_plugin_matrix here: http://ci.theforeman.org/view/Plugins/
170 28 Dominic Cleal
171 28 Dominic Cleal
h4. Smart proxy plugin PR testing
172 28 Dominic Cleal
173 28 Dominic Cleal
To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_proxy_plugin_pull_request/
174 28 Dominic Cleal
175 28 Dominic Cleal
To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner.  Also enable the prprocessor webhook on the GitHub repo for immediate builds.
176 28 Dominic Cleal
177 28 Dominic Cleal
h4. Adding a new smart proxy plugin
178 28 Dominic Cleal
179 28 Dominic Cleal
For a plugin "smart_proxy_example", first create a job that tests the main (master or develop) branch.
180 28 Dominic Cleal
181 28 Dominic Cleal
# ensure plugin tests run when doing @bundle install@ and @rake test@, see "testing a plugin":http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Smart-Proxy_Plugin#Testing for help
182 28 Dominic Cleal
#* if different branches rely on different versions of smart proxy, specify @:branch@ in Gemfile on those branches
183 28 Dominic Cleal
# create a smart_proxy_example.yaml file in foreman-infra/JJB
184 28 Dominic Cleal
#* https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/config/yaml/jobs/plugins
185 28 Dominic Cleal
#* This will create a "test_proxy_plugin_smart_proxy_example_master" job in Jenkins to test the master branch.
186 1 Dominic Cleal
# ensure the job is green by fixing bugs, installing dependencies etc.
187 31 Dominic Cleal
# add hook to GitHub repo, see "GitHub repo hook":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#GitHub-repo-hook
188 28 Dominic Cleal
189 28 Dominic Cleal
Next, set up PR testing for the plugin:
190 28 Dominic Cleal
191 28 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
192 28 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
193 30 Dominic Cleal
# add pull_request hook to GitHub repo, see "PR scanner repo hooks":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#PR-scanner-repo-hooks
194 28 Dominic Cleal
195 5 Dominic Cleal
h2. System testing with foreman-bats
196 5 Dominic Cleal
197 6 Dominic Cleal
Some system tests are performed on the complete all-in-one Foreman setup, which includes packages, the installer, the CLI and related components.
198 5 Dominic Cleal
199 6 Dominic Cleal
These tests are currently in the "foreman-bats":http://github.com/theforeman/foreman-bats project and use the BATS test framework (based on bash).
200 1 Dominic Cleal
201 6 Dominic Cleal
They are intended to be smoke tests only, not in depth testing of any component of the stack.  Most components have their own unit tests, which are cheaper to execute and are run closer to where the code is developed, reducing the turnaround time for fixes.
202 5 Dominic Cleal
203 5 Dominic Cleal
h3. systest jobs
204 5 Dominic Cleal
205 6 Dominic Cleal
The systest_* jobs in Jenkins run the system tests, with "systest_foreman":http://ci.theforeman.org/job/systest_foreman/ being the main parameterised job for executing them.
206 5 Dominic Cleal
207 6 Dominic Cleal
For RPM-based distributions, the systest_foreman job is run from the packaging_publish_rpm job as an intermediate test phase between the repo being generated and published to the public web site.
208 5 Dominic Cleal
209 6 Dominic Cleal
For Debian distributions, the systest_foreman_debian job runs nightly against the published repos to report problems.  This is a matrix job that runs once for each Debian-based distribution.
210 5 Dominic Cleal
211 5 Dominic Cleal
h3. Vagrant setup
212 5 Dominic Cleal
213 6 Dominic Cleal
foreman-bats is very simple - it's a bash script that executes as root on a host, installs Foreman and tests the result.  Since we don't want to install Foreman directly on slaves, this is run on the "Rackspace public cloud":http://www.rackspace.com/cloud/servers/, under a project account.
214 5 Dominic Cleal
215 6 Dominic Cleal
The Jenkins jobs use "Vagrant":http://vagrantup.com/ to create hosts on Rackspace and to run the foreman-bats project on it.  Vagrant is installed via the "foreman-infra Puppet modules":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/slave/manifests/vagrant.pp along with the "vagrant-rackspace":https://github.com/mitchellh/vagrant-rackspace/ plugin.  This launches a standard Rackspace image for the OS under test, Vagrant rsyncs the current directory (workspace) to it containing foreman-bats, and the job script then executes the foreman-bats test over @vagrant ssh@.
216 5 Dominic Cleal
217 6 Dominic Cleal
The same Vagrant setup can be used to run foreman-bats locally via vagrant-libvirt or other plugins.
218 11 Dominic Cleal
219 11 Dominic Cleal
h2. Slaves
220 11 Dominic Cleal
221 11 Dominic Cleal
h3. Configuration management
222 11 Dominic Cleal
223 11 Dominic Cleal
All slaves are maintained through our own Foreman instance using Puppet.  The Foreman instance has a host group called "Base/Builders" and "Base/Builders/Red Hat" and "Base/Builders/Debian" which have the "slave" and other classes assigned to them.
224 11 Dominic Cleal
225 11 Dominic Cleal
https://github.com/theforeman/foreman-infra/tree/master/puppet/modules contains the source for all Puppet modules.
226 11 Dominic Cleal
227 18 Dominic Cleal
h3. Slave requirements
228 18 Dominic Cleal
229 18 Dominic Cleal
* CentOS 6 currently (or 7 once we've tested and updated our Puppet modules)
230 20 Dominic Cleal
** Clean, minimal base installation or the option to reinstall it
231 18 Dominic Cleal
* 1GB of RAM per vCPU (4 vCPU + 4GB RAM is typical)
232 19 Dominic Cleal
* 40GB disk (minimum), SSD preferred
233 22 Dominic Cleal
* ~20GB/month bandwidth
234 21 Dominic Cleal
* Public facing IP address
235 20 Dominic Cleal
* Root access
236 18 Dominic Cleal
237 18 Dominic Cleal
h3. Configuring a new slave
238 11 Dominic Cleal
239 11 Dominic Cleal
If using Rackspace, start the new server via the web UI or the "rumm":https://github.com/rackspace/rumm CLI utility.  Ensure you select:
240 11 Dominic Cleal
241 11 Dominic Cleal
* Image: CentOS 6.5
242 11 Dominic Cleal
* Flavor: Performance 1 (4GB)
243 11 Dominic Cleal
244 11 Dominic Cleal
Set up the data partition for the Jenkins workspace:
245 11 Dominic Cleal
246 12 Dominic Cleal
# @mkfs.ext4 -L data /dev/xvde1@
247 12 Dominic Cleal
# @echo "LABEL=data /var/lib/workspace ext4 defaults,noatime 1 2" >> /etc/fstab@
248 12 Dominic Cleal
# @mkdir /var/lib/workspace && mount /var/lib/workspace@
249 11 Dominic Cleal
250 11 Dominic Cleal
Then:
251 11 Dominic Cleal
252 12 Dominic Cleal
# Ensure EPEL is configured: "epel-release":http://dl.fedoraproject.org/pub/epel/6/x86_64/repoview/epel-release.html
253 12 Dominic Cleal
# Ensure yum.pl.com is configured: "puppetlabs-release":http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
254 13 Dominic Cleal
# @yum -y install puppet@
255 12 Dominic Cleal
# @echo "server = puppetmaster.theforeman.org" >> /etc/puppet/puppet.conf@
256 12 Dominic Cleal
# @puppet agent -t@
257 12 Dominic Cleal
# Sign the certificate on the puppetmaster or via Foreman
258 14 Dominic Cleal
# @puppet agent -t@
259 14 Dominic Cleal
# Set the host group to "Base/Builders/Red Hat" in Foreman
260 12 Dominic Cleal
# Run @puppet agent -t@ twice (second run is important, due to the rvm module behaviour)
261 12 Dominic Cleal
# Add the node by IP address to Jenkins, copying an existing slave (except slave01)