Project

General

Profile

Jenkins » History » Version 49

Eric Helms, 06/11/2018 08:20 PM

1 1 Dominic Cleal
h1. Jenkins
2 1 Dominic Cleal
3 1 Dominic Cleal
The Foreman project maintains its own Jenkins instance for continuous integration at http://ci.theforeman.org/
4 1 Dominic Cleal
5 41 Dominic Cleal
It runs the following types of tests:
6 41 Dominic Cleal
7 41 Dominic Cleal
* Unit tests on the main develop and recent stable branches of core Foreman projects
8 41 Dominic Cleal
* Pull request tests against core Foreman projects
9 41 Dominic Cleal
* Unit and pull request tests for many Foreman plugins
10 41 Dominic Cleal
* Nightly and release package building and release processes
11 41 Dominic Cleal
* Package smoke and integration testing ("systest")
12 41 Dominic Cleal
13 23 Dominic Cleal
{{toc}}
14 1 Dominic Cleal
15 41 Dominic Cleal
h1. Quick reference for maintainers
16 1 Dominic Cleal
17 41 Dominic Cleal
Current PR test jobs (used on Foreman itself) support these commands:
18 1 Dominic Cleal
19 41 Dominic Cleal
* @ok to test@ - run tests for an unknown user, if the code within the patch is not malicious
20 42 Dominic Cleal
* @add to PR whitelist@ - automatically run all PRs in future from this user
21 41 Dominic Cleal
* @[test STATUS-NAME]@, e.g. @[test foreman]@ to re-run a particular set of tests
22 1 Dominic Cleal
23 41 Dominic Cleal
Older PR tests, those that use the status name "default", supports this command:
24 1 Dominic Cleal
25 41 Dominic Cleal
* @[test]@ - run tests for an unknown user, or re-run all tests
26 1 Dominic Cleal
27 41 Dominic Cleal
h1. Quick reference for plugin maintainers
28 23 Dominic Cleal
29 41 Dominic Cleal
h2. Foreman plugin testing
30 1 Dominic Cleal
31 41 Dominic Cleal
Foreman plugins are tested by adding the plugin to a Foreman checkout and running core tests, so it checks that existing behaviours still work and new plugin tests are run too.  The "test_plugin_matrix job":http://ci.theforeman.org/job/test_plugin_matrix/ copies the core jobs, but adds a plugin from a given git repo/branch and is usually used to test plugins in a generic way.
32 1 Dominic Cleal
33 41 Dominic Cleal
Each plugin should have a job defined in JJB that calls test_plugin_matrix here: http://ci.theforeman.org/view/Plugins/
34 1 Dominic Cleal
35 41 Dominic Cleal
h3. Foreman plugin PR testing
36 1 Dominic Cleal
37 41 Dominic Cleal
To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_plugin_pull_request/
38 1 Dominic Cleal
39 41 Dominic Cleal
To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner.  Also enable the prprocessor webhook on the GitHub repo for immediate builds.
40 1 Dominic Cleal
41 41 Dominic Cleal
h3. Adding a new Foreman plugin
42 1 Dominic Cleal
43 41 Dominic Cleal
For a plugin "foreman_example", first create a job that tests the main (master or develop) branch.
44 1 Dominic Cleal
45 41 Dominic Cleal
# ensure plugin tests (if any) run when @rake jenkins:unit@ is called, see "the example plugin":https://github.com/theforeman/foreman_plugin_template/ and "testing a plugin":http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin#Testing for help
46 41 Dominic Cleal
# create a foreman_example.yaml file in foreman-infra/JJB
47 41 Dominic Cleal
#* https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/plugins
48 41 Dominic Cleal
#* This will create a "test_plugin_foreman_example_master" job in Jenkins to test the master branch.
49 41 Dominic Cleal
# ensure the job is green by fixing bugs, installing dependencies etc.
50 47 Dominic Cleal
# add hook to GitHub repo, see "GitHub repo hook":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#Testing-develop
51 1 Dominic Cleal
52 41 Dominic Cleal
Next, set up PR testing for the plugin:
53 1 Dominic Cleal
54 41 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
55 41 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
56 46 Dominic Cleal
# add project to [[PrProcessor]] "repos.yaml":https://github.com/theforeman/prprocessor/blob/master/config/repos.yaml config file with "pr_scanner: true"
57 1 Dominic Cleal
58 46 Dominic Cleal
An org admin must then:
59 46 Dominic Cleal
60 46 Dominic Cleal
# add [[PrProcessor]] webhook to GitHub repo
61 46 Dominic Cleal
# add the repo to the "Bots team":https://github.com/orgs/theforeman/teams/bots/repositories with +write+ access
62 46 Dominic Cleal
63 41 Dominic Cleal
h2. Smart proxy plugin testing
64 41 Dominic Cleal
65 41 Dominic Cleal
Proxy plugins are tested like ordinary gems with tests run entirely from the plugin directory, installing the smart proxy as a dependency (via bundler's git support).  The "test_proxy_plugin_matrix job":http://ci.theforeman.org/job/test_proxy_plugin_matrix/ is usually used to test plugins in a generic way.
66 41 Dominic Cleal
67 41 Dominic Cleal
Each plugin should have a job defined in JJB that calls test_proxy_plugin_matrix here: http://ci.theforeman.org/view/Plugins/
68 41 Dominic Cleal
69 41 Dominic Cleal
h3. Smart proxy plugin PR testing
70 41 Dominic Cleal
71 41 Dominic Cleal
To test pull requests, a separate job is used that also takes the PR details: http://ci.theforeman.org/view/Plugins/job/test_proxy_plugin_pull_request/
72 41 Dominic Cleal
73 41 Dominic Cleal
To set this up for a plugin, make the template + manifest addition as noted in the PR section below to configure the PR scanner.  Also enable the prprocessor webhook on the GitHub repo for immediate builds.
74 41 Dominic Cleal
75 41 Dominic Cleal
h3. Adding a new smart proxy plugin
76 41 Dominic Cleal
77 41 Dominic Cleal
For a plugin "smart_proxy_example", first create a job that tests the main (master or develop) branch.
78 41 Dominic Cleal
79 41 Dominic Cleal
# ensure plugin tests run when doing @bundle install@ and @rake test@, see "testing a plugin":http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Smart-Proxy_Plugin#Testing for help
80 41 Dominic Cleal
#* if different branches rely on different versions of smart proxy, specify @:branch@ in Gemfile on those branches
81 41 Dominic Cleal
# create a smart_proxy_example.yaml file in foreman-infra/JJB
82 41 Dominic Cleal
#* https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/plugins
83 41 Dominic Cleal
#* This will create a "test_proxy_plugin_smart_proxy_example_master" job in Jenkins to test the master branch.
84 41 Dominic Cleal
# ensure the job is green by fixing bugs, installing dependencies etc.
85 47 Dominic Cleal
# add hook to GitHub repo, see "GitHub repo hook":http://projects.theforeman.org/projects/foreman/wiki/Jenkins#Testing-develop
86 41 Dominic Cleal
87 41 Dominic Cleal
Next, set up PR testing for the plugin:
88 41 Dominic Cleal
89 1 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
90 41 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
91 46 Dominic Cleal
# add project to [[PrProcessor]] "repos.yaml":https://github.com/theforeman/prprocessor/blob/master/config/repos.yaml config file with "pr_scanner: true"
92 41 Dominic Cleal
93 46 Dominic Cleal
An org admin must then:
94 1 Dominic Cleal
95 46 Dominic Cleal
# add [[PrProcessor]] webhook to GitHub repo
96 46 Dominic Cleal
# add the repo to the "Bots team":https://github.com/orgs/theforeman/teams/bots/repositories with +write+ access
97 46 Dominic Cleal
98 41 Dominic Cleal
h1. Job configurations
99 41 Dominic Cleal
100 41 Dominic Cleal
h2. Testing develop
101 41 Dominic Cleal
102 1 Dominic Cleal
All repos with an associated job that tests their master/develop branch should have a hook added to the repo to trigger immediate builds.
103 1 Dominic Cleal
104 41 Dominic Cleal
To set up the hook, an org/repo admin must:
105 1 Dominic Cleal
106 41 Dominic Cleal
# View the repository settings
107 41 Dominic Cleal
# Click _Webhooks & Services_
108 41 Dominic Cleal
# _Add service_ dropdown, select _Jenkins (Git plugin)_
109 41 Dominic Cleal
# For the URL, enter @http://ci.theforeman.org/@
110 41 Dominic Cleal
111 1 Dominic Cleal
h2. Pull request testing
112 3 Dominic Cleal
113 3 Dominic Cleal
Core Foreman projects have GitHub pull requests tested on our Jenkins instance so it's identical to the way we test the primary development branches themselves.  Simpler and quieter projects (such as installer modules) should use Travis CI which supports PR testing and reduces the load on our own infrastructure.
114 3 Dominic Cleal
115 41 Dominic Cleal
Every project that needs PR testing has at least two Jenkins jobs.  Taking core Foreman as an example:
116 15 Dominic Cleal
117 29 Dominic Cleal
* Test job for the main development branch (develop or master): "test_develop":http://ci.theforeman.org/job/test_develop/
118 41 Dominic Cleal
* Test job for each PR: "test_develop_pr_core":http://ci.theforeman.org/job/test_develop_pr_core/
119 29 Dominic Cleal
120 41 Dominic Cleal
The results from these PR jobs are only stored for a few weeks, sufficient for reviews. The PR jobs are set up in a few different ways, as it's changed over time:
121 29 Dominic Cleal
122 41 Dominic Cleal
# _Version 3_ jobs use the _GitHub Pull Request Builder_ (GHPRB) and merge the branch to the project branch
123 41 Dominic Cleal
# _Version 2_ jobs use the _PR scanner_ and merge the branch to the primary project branch
124 41 Dominic Cleal
# _Version 1_ jobs use the _PR scanner_ and use @git apply@ with the patch from the PR
125 35 Dominic Cleal
126 41 Dominic Cleal
h3. Version 3 (GHPRB)
127 4 Dominic Cleal
128 41 Dominic Cleal
The GHPRB plugin uses webhooks installed on the repo to trigger a build, then it runs any job configured with the GHPRB trigger and a matching GitHub project set.
129 4 Dominic Cleal
130 44 Dominic Cleal
The plugin tests the latest commit on the PR branch only, it does not merge the PR with the base branch. The webhook may also trigger multiple jobs, and jobs may use different GitHub commit status names to easily test and report status on different types of tests.
131 4 Dominic Cleal
132 41 Dominic Cleal
PR jobs should be set up identically to primary branch tests, except for the SCM (which checks out @${sha1}@) and to add the GHPRB trigger (see the @github_pr@ macro in JJB).
133 41 Dominic Cleal
134 41 Dominic Cleal
To set up the hook, an org/repo admin goes to the repository settings, then Webhooks & Services and adds a webhook with these settings:
135 41 Dominic Cleal
136 41 Dominic Cleal
* Payload URL: @http://ci.theforeman.org/ghprbhook/@
137 41 Dominic Cleal
* Content type: @application/json@
138 41 Dominic Cleal
* Secret: _redacted_
139 41 Dominic Cleal
* Events: _Let me select individual events_, _Pull request_, _Issue comment_
140 41 Dominic Cleal
141 41 Dominic Cleal
An org admin must then change the org teams:
142 41 Dominic Cleal
143 41 Dominic Cleal
* Add the repository to the "Bots team":https://github.com/orgs/theforeman/teams/bots/repositories with +write+ access
144 41 Dominic Cleal
145 41 Dominic Cleal
h3. Version 2 (scanner + git URL/ref)
146 41 Dominic Cleal
147 25 Dominic Cleal
The PR test job takes the following parameters:
148 25 Dominic Cleal
149 25 Dominic Cleal
* pr_git_url: URL to the repository containing the PR to clone
150 25 Dominic Cleal
* pr_git_ref: branch name in the above repository for the PR
151 25 Dominic Cleal
* pr_number: optional, for informational purposes only (e.g. tracking from a job back to a PR)
152 25 Dominic Cleal
153 25 Dominic Cleal
Under source code management in the job configuration, set up:
154 25 Dominic Cleal
155 25 Dominic Cleal
# Project git repository:
156 25 Dominic Cleal
#* Repository URL: @https://github.com/theforeman/foreman@
157 25 Dominic Cleal
# Branches to build: @develop@
158 27 Dominic Cleal
# Additional Behaviours > Prune stale remote-tracking branches
159 17 Dominic Cleal
# Additional Behaviours > Wipe out repository & force clone
160 33 Dominic Cleal
161 17 Dominic Cleal
As the first build step, add:
162 1 Dominic Cleal
163 37 Dominic Cleal
<pre>
164 1 Dominic Cleal
#!/bin/bash -ex
165 1 Dominic Cleal
if [ -n "${pr_git_url}" ]; then
166 1 Dominic Cleal
  git remote add pr ${pr_git_url}
167 1 Dominic Cleal
  git fetch pr
168 31 Dominic Cleal
  git merge pr/${pr_git_ref}
169 27 Dominic Cleal
fi
170 27 Dominic Cleal
</pre>
171 27 Dominic Cleal
172 41 Dominic Cleal
h3. Version 1 (scanner + .patch)
173 17 Dominic Cleal
174 41 Dominic Cleal
The PR test job takes the PR number parameter, downloads the patch from GitHub (by appending a .patch extension to the PR URL), applies it to the local checkout of the project and then builds as normal.  This process means PRs are effectively rebased onto the current development branch before tests are run, rather than testing the branch as-is.  GitHub tracks "mergeability" so we don't test PRs that can't be merged cleanly.
175 41 Dominic Cleal
176 41 Dominic Cleal
h3. PR scanner (version 1, 2)
177 28 Dominic Cleal
178 1 Dominic Cleal
To initiate the PR tests, the "test-pull-requests":https://github.com/theforeman/test-pull-requests script is used to scan for open PRs, check they're mergeable and then trigger the Jenkins job.  The script is a fork from the OpenShift upstream, enhanced in a few areas including changing from comment-based updates to the GitHub status API.
179 1 Dominic Cleal
180 1 Dominic Cleal
The script runs under the "pull_request_scanner":http://ci.theforeman.org/job/pull_request_scanner/ job under Jenkins and is set to run a few times every hour.  It scans all configured projects for PRs and then exits, leaving the PR test jobs themselves to execute asynchronously.
181 1 Dominic Cleal
182 1 Dominic Cleal
The configuration files are deployed via our Puppet infrastructure for each project, and mostly just detail the GitHub repos, branches and Jenkins job names.  These are managed in the "slave foreman-infra module":https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/slave in slave itself and templates/.
183 1 Dominic Cleal
184 45 Dominic Cleal
After a PR test job completes, the Jenkins jobs are configured to build the PR scanner job again, which means immediately after the PR test results come in, the PR scanner script is able to update the status on the GitHub PR.  A kind of feedback loop, if you will.
185 1 Dominic Cleal
186 1 Dominic Cleal
h3. PR scanner repo hooks
187 1 Dominic Cleal
188 46 Dominic Cleal
In addition to regular scheduled runs of the PR scanner, a hook is added to the GitHub repositories to kick the PR scanner "build" when a PR is opened or synchronised.
189 1 Dominic Cleal
190 46 Dominic Cleal
See [[PrProcessor#GitHub-integration]] for information on enabling the prprocessor webhook. The repo must be listed in the prprocessor config file with "pr_scanner: true".
191 1 Dominic Cleal
192 1 Dominic Cleal
h3. Adding a new generic project
193 41 Dominic Cleal
194 1 Dominic Cleal
For a project "foo":
195 41 Dominic Cleal
196 41 Dominic Cleal
# create a test_foo job that tests the primary development branch, enable IRC notifications in post build
197 1 Dominic Cleal
# clone to test_foo_pull_request
198 1 Dominic Cleal
#* without IRC notifications
199 1 Dominic Cleal
#* add pr_number parameter
200 1 Dominic Cleal
#* add initial build step to download the patch and apply
201 1 Dominic Cleal
# add template to foreman-infra/puppet/modules/slave/templates, update branch, job names etc.
202 1 Dominic Cleal
# add project to slave::pr_test_config list in foreman-infra/puppet/modules/slave/manifests/init.pp
203 46 Dominic Cleal
# add project to [[PrProcessor]] "repos.yaml":https://github.com/theforeman/prprocessor/blob/master/config/repos.yaml config file with "pr_scanner: true"
204 46 Dominic Cleal
205 46 Dominic Cleal
An org admin must then:
206 46 Dominic Cleal
207 46 Dominic Cleal
# add [[PrProcessor]] webhook to GitHub repo
208 46 Dominic Cleal
# add the repo to the "Bots team":https://github.com/orgs/theforeman/teams/bots/repositories with +write+ access
209 1 Dominic Cleal
210 1 Dominic Cleal
h2. System testing with foreman-bats
211 1 Dominic Cleal
212 1 Dominic Cleal
Some system tests are performed on the complete all-in-one Foreman setup, which includes packages, the installer, the CLI and related components.
213 28 Dominic Cleal
214 28 Dominic Cleal
These tests are currently in the "foreman-bats":http://github.com/theforeman/foreman-bats project and use the BATS test framework (based on bash).
215 28 Dominic Cleal
216 28 Dominic Cleal
They are intended to be smoke and integration tests only, not in depth testing of any component of the stack.  Most components have their own unit tests, which are cheaper to execute and are run closer to where the code is developed, reducing the turnaround time for fixes.
217 28 Dominic Cleal
218 28 Dominic Cleal
h3. systest jobs
219 28 Dominic Cleal
220 28 Dominic Cleal
The systest_* jobs in Jenkins run the system tests, with "systest_foreman":http://ci.theforeman.org/job/systest_foreman/ being the main parameterised job for executing them.  Other jobs call this and it's used in two ways.
221 28 Dominic Cleal
222 28 Dominic Cleal
h4. Package smoke testing
223 28 Dominic Cleal
224 28 Dominic Cleal
For both types of packages, the systest_foreman job is run as an intermediate test phase between the repo being generated and published to the public web site.  The aim is to ensure that the packages install and the application starts up in its default configuration.
225 28 Dominic Cleal
226 28 Dominic Cleal
During official releases it's called by release_test, and for nightly packages it's called by release_nightly_test_rpm and release_nightly_test_deb.  Once the tests have run, the test jobs trigger publishing jobs to push the packages to the public repositories.
227 28 Dominic Cleal
228 37 Dominic Cleal
h4. Scenario testing
229 28 Dominic Cleal
230 1 Dominic Cleal
A set of further integration tests are run against the published packages for different scenarios.  These usually run once or a couple of times a week against the published nightly packages - since they run on every OS, we don't run them too often due to limited infrastructure.  They run against published packages as any failure shouldn't be enough to block the publishing of nightly packages, but should probably be considered as high priority for releases.
231 31 Dominic Cleal
232 28 Dominic Cleal
We currently test the following:
233 28 Dominic Cleal
234 28 Dominic Cleal
* Hammer integration (fb-hammer-tests.bats): runs the "hammer-tests":https://github.com/theforeman/hammer-tests suite
235 28 Dominic Cleal
* MySQL database support: installs Foreman with --foreman-db-type=mysql
236 28 Dominic Cleal
* Puppet integration (fb-puppet-tests.bats): checking that modules can be installed, imported and applied to hosts
237 30 Dominic Cleal
* Puppet nightlies integration: checking that Foreman installs and functions with nightly Puppet packages (currently disabled)
238 28 Dominic Cleal
* umask 077 (fb-setup-umask.bats): installs Foreman under a umask of 077
239 5 Dominic Cleal
240 5 Dominic Cleal
h3. Vagrant setup
241 6 Dominic Cleal
242 5 Dominic Cleal
foreman-bats is very simple - it's a bash script that executes as root on a host, installs Foreman and tests the result.  Since we don't want to install Foreman directly on slaves, this is run on the "Rackspace public cloud":http://www.rackspace.com/cloud/servers/, under a project account.
243 6 Dominic Cleal
244 1 Dominic Cleal
The Jenkins jobs use "Vagrant":http://vagrantup.com/ to create hosts on Rackspace and to run the foreman-bats project on it.  Vagrant is installed via the "foreman-infra Puppet modules":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/slave/manifests/vagrant.pp along with the "vagrant-rackspace":https://github.com/mitchellh/vagrant-rackspace/ plugin.  This launches a standard Rackspace image for the OS under test, Vagrant rsyncs the current directory (workspace) to it containing foreman-bats, and the job script then executes the foreman-bats test over @vagrant ssh@.
245 34 Dominic Cleal
246 5 Dominic Cleal
The same Vagrant setup can be used to run foreman-bats locally via vagrant-libvirt or other plugins.
247 5 Dominic Cleal
248 5 Dominic Cleal
h3. CentOS QA service
249 34 Dominic Cleal
250 1 Dominic Cleal
* _This change is in development, not currently in use._
251 34 Dominic Cleal
252 1 Dominic Cleal
CentOS offer access to their "QA CI infrastructure":http://wiki.centos.org/QaWiki/CI to Foreman and other projects, which gives on-demand access to run tests on physical hardware running CentOS.  It consists of three main components:
253 34 Dominic Cleal
254 34 Dominic Cleal
# "ci.centos.org":https://ci.centos.org - a Jenkins instance which we can manage jobs on
255 34 Dominic Cleal
# foreman@slave01 - a user account on a slave which our jobs all run on
256 34 Dominic Cleal
# "Duffy":http://wiki.centos.org/QaWiki/CI/Duffy - on-demand provisioning of physical test servers running CentOS, available from foreman@slave01 jobs
257 34 Dominic Cleal
258 34 Dominic Cleal
h4. Replacing Rackspace Cloud with Duffy
259 34 Dominic Cleal
260 34 Dominic Cleal
Our CI today runs the systest_foreman job on a slave, which uses vagrant-rackspace to create a VM of each supported OS on Rackspace on which tests are executed.  These tests could be moved to Duffy hardware, reducing our usage of the Rackspace account, so we can run more static slaves for unit tests, and allowing us to run systest jobs more frequently or in more combinations (inc. nested virt).
261 34 Dominic Cleal
262 34 Dominic Cleal
Since Duffy is CentOS-only, we can use libvirt (preferably with vagrant-libvirt) to access each of our supported OSes.  Vagrant could be installed on the Duffy-provisioned host, or once packaged via the SCLo SIG, even onto slave01 to save time.  Suggested plan:
263 34 Dominic Cleal
264 34 Dominic Cleal
# Create systest_foreman_duffy on CentOS CI, accepting all the usual parameters, preferably via JJB
265 34 Dominic Cleal
# systest_foreman_duffy has three build tasks:
266 34 Dominic Cleal
## Execute shell (Python?): request new host from Duffy with /Node/get ("example script":https://github.com/kbsingh/centos-ci-scripts/blob/master/build_python_script.py)
267 34 Dominic Cleal
## Execute shell: run "usual vagrant up commands":https://github.com/theforeman/foreman-infra/blob/26da5851/puppet/modules/jenkins_job_builder/files/config/scripts/systest_foreman.sh on remote Duffy host over SSH
268 5 Dominic Cleal
## Post-build shell task: destroy the Duffy built host
269 5 Dominic Cleal
# systest_foreman_* or release_test_* jobs on Foreman CI can call systest_foreman_duffy *remotely* via the "remote trigger plugin":https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Remote+Trigger+Plugin
270 41 Dominic Cleal
271 41 Dominic Cleal
h2. Managing jobs
272 41 Dominic Cleal
273 41 Dominic Cleal
Jenkins itself is deployed onto one master VM from foreman-infra.  Jobs are maintained in two ways:
274 41 Dominic Cleal
275 41 Dominic Cleal
# Updated by hand - some developers have accounts to log in and modify jobs
276 41 Dominic Cleal
# Jenkins Job Builder - a set of YAML configuration files in foreman-infra that generate jobs
277 41 Dominic Cleal
278 41 Dominic Cleal
New jobs should be written via Jenkins Job Builder if possible.
279 41 Dominic Cleal
280 41 Dominic Cleal
h3. Jenkins Job Builder
281 41 Dominic Cleal
282 41 Dominic Cleal
"Jenkins Job Builder":http://docs.openstack.org/infra/jenkins-job-builder/ (JJB) is an OpenStack tool for generating Jenkins job definitions (an XML file) from a set of YAML job descriptions, which we store in version control.
283 41 Dominic Cleal
284 41 Dominic Cleal
* "JJB YAML files in foreman-infra":https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files
285 41 Dominic Cleal
286 41 Dominic Cleal
Puppet deploys these onto our Jenkins server (a recursive file copy) and when they change, it runs the JJB tool to update the jobs in the live instance.  It also refreshes them daily to overwrite manual changes.
287 41 Dominic Cleal
288 41 Dominic Cleal
We use a fork of JJB with some extra support for plugins we use:
289 41 Dominic Cleal
290 41 Dominic Cleal
* "theforeman/jenkins-job-builder, local branch":https://github.com/theforeman/jenkins-job-builder/tree/local
291 41 Dominic Cleal
292 41 Dominic Cleal
To test it:
293 41 Dominic Cleal
294 41 Dominic Cleal
# check out the above JJB repo and local branch
295 41 Dominic Cleal
# recommended: "create a Python virtualenv and activate it":https://virtualenv.pypa.io/en/latest/userguide.html#usage
296 41 Dominic Cleal
## @virtualenv jjb@
297 41 Dominic Cleal
## @source jjb/bin/activate@
298 41 Dominic Cleal
# @easy_install .@
299 41 Dominic Cleal
# check out foreman-infra
300 41 Dominic Cleal
# @cd foreman-infra/puppet/modules/jenkins_job_builder/files@
301 41 Dominic Cleal
# @jenkins-jobs -l debug test -r . -o /tmp/jobs@
302 41 Dominic Cleal
303 41 Dominic Cleal
JJB will populate /tmp/jobs with the proposed XML files.
304 41 Dominic Cleal
305 41 Dominic Cleal
Useful resources:
306 41 Dominic Cleal
307 41 Dominic Cleal
# "Job definitions, templates etc.":http://docs.openstack.org/infra/jenkins-job-builder/definition.html
308 41 Dominic Cleal
# "Modules, e.g. SCM, publishers, builders":http://docs.openstack.org/infra/jenkins-job-builder/definition.html#modules
309 18 Dominic Cleal
310 40 Dominic Cleal
h2. Slaves
311 20 Dominic Cleal
312 18 Dominic Cleal
h3. Configuration management
313 40 Dominic Cleal
314 22 Dominic Cleal
All slaves are maintained through our own Foreman instance using Puppet.  The Foreman instance has a host group called "Base/Builders" and "Base/Builders/Red Hat" and "Base/Builders/Debian" which have the "slave" and other classes assigned to them.
315 21 Dominic Cleal
316 20 Dominic Cleal
https://github.com/theforeman/foreman-infra/tree/master/puppet/modules contains the source for all Puppet modules.
317 18 Dominic Cleal
318 18 Dominic Cleal
h3. Slave requirements
319 11 Dominic Cleal
320 11 Dominic Cleal
* CentOS 7
321 11 Dominic Cleal
** Clean, minimal base installation or the option to reinstall it
322 43 Dominic Cleal
* 2GB of RAM per vCPU (4 vCPU + 8GB RAM is typical)
323 40 Dominic Cleal
* 60GB disk (minimum), SSD preferred
324 11 Dominic Cleal
* ~20GB/month bandwidth
325 40 Dominic Cleal
* Public facing IP address
326 11 Dominic Cleal
* Root access
327 12 Dominic Cleal
328 12 Dominic Cleal
h3. Configuring a new slave
329 12 Dominic Cleal
330 11 Dominic Cleal
If using Rackspace, start the new server via the web UI or the "rumm":https://github.com/rackspace/rumm CLI utility.  Ensure you select:
331 11 Dominic Cleal
332 11 Dominic Cleal
* Image: CentOS 7 (PVHVM)
333 38 Dominic Cleal
* Flavor: 4 GB General Purpose v1
334 38 Dominic Cleal
335 13 Dominic Cleal
_Flavors with separate root/data disks only:_ set up the data partition for the Jenkins workspace:
336 12 Dominic Cleal
337 12 Dominic Cleal
# @mkfs.ext4 -L data /dev/xvde1@
338 12 Dominic Cleal
# @echo "LABEL=data /var/lib/workspace ext4 defaults,noatime 1 2" >> /etc/fstab@
339 14 Dominic Cleal
# @mkdir /var/lib/workspace && mount /var/lib/workspace@
340 14 Dominic Cleal
341 12 Dominic Cleal
Then:
342 12 Dominic Cleal
343 1 Dominic Cleal
# Ensure EPEL is configured: "epel-release":http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
344 48 Eric Helms
# Ensure yum.pl.com is configured: "puppetlabs-release-pc1":http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
345 1 Dominic Cleal
# @yum -y install puppet@
346 49 Eric Helms
# @echo "server = puppetmaster.theforeman.org" >> /etc/puppetlabs/puppet/puppet.conf@
347 1 Dominic Cleal
# @puppet agent -t@
348 1 Dominic Cleal
# Sign the certificate on the puppetmaster or via Foreman
349 1 Dominic Cleal
# @puppet agent -t@
350 1 Dominic Cleal
# Set the host group to "Base/Builders/Red Hat" in Foreman
351 1 Dominic Cleal
# Run @puppet agent -t@ twice (second run is important, due to the rvm module behaviour)
352 1 Dominic Cleal
# Add the node by IP address to Jenkins, copying an existing slave (except slave01)