Project

General

Profile

Debian Packaging » History » Version 20

Dominic Cleal, 04/10/2015 10:23 AM

1 1 Greg Sutcliffe
h1. Debian Packaging
2
3
{{toc}}
4
5
h2. Artifacts
6
7 7 Greg Sutcliffe
The following artifacts are generated from the Deb build process and form part of any release:
8 1 Greg Sutcliffe
9
* foreman, including database and compute resource subpackages
10
* foreman-installer
11
* foreman-proxy
12 2 Greg Sutcliffe
* dependencies
13 1 Greg Sutcliffe
14 6 Greg Sutcliffe
h2. Jenkins
15 1 Greg Sutcliffe
16
Jenkins is used to automate the package builds, using the jobs:
17
18 6 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_dependency (for gems)
19 16 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_plugin (for plugins)
20 6 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_coreproject (for core/proxy/installer)
21
22
These jobs rely on matching the branching structure of the core projects to the branching in foreman-packaging.
23
24 8 Greg Sutcliffe
In addition these matrix jobs build regular 'scratch' packages from the develop HEAD
25 1 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman_matrix
26
* http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman-proxy_matrix
27
28 6 Greg Sutcliffe
The matrix jobs only presently build the nightly packages, thus RC/Stable releases will be queued manually using the basic jobs. The configuration depends on the build you want:
29 1 Greg Sutcliffe
30 9 Dominic Cleal
h3. Scratch build parameters
31 1 Greg Sutcliffe
32 9 Dominic Cleal
h4. Want to test your own branch?
33
34 6 Greg Sutcliffe
Push your packaging updates to your fork of foreman-packaging:deb/develop
35
Change repoowner to your GitHub account
36
Select project (core/proxy/installer)
37
Select OS ('all', or a specific target)
38 1 Greg Sutcliffe
branch should be 'develop'
39 8 Greg Sutcliffe
leave PR field blank
40 6 Greg Sutcliffe
gitrelease should be ticked
41
42 10 Dominic Cleal
The packages will be on stagingdeb.theforeman.org/<os>/<your github user>.  To use in apt, add this to sources.list:
43
44 12 Greg Sutcliffe
    deb http://stagingdeb.theforeman.org/ os username
45 1 Greg Sutcliffe
46
h4. Want to test a PR?
47 9 Dominic Cleal
48 8 Greg Sutcliffe
repoowner must be theforeman
49
Select project (core/proxy/installer)
50
Select OS ('all', or a specific target)
51
branch must be x.y (e.g 1.3) - this is both the branch of foreman-packaging _and_ the target component on deb.theforeman.org
52
set PR field to the number of the PR (e.g 113)
53
gitrelease must be ticked
54
55
Final packages will be pushed to stagingdeb.theforeman.org/<os>/theforeman and marked with the string "pr(#number)"
56 1 Greg Sutcliffe
57 8 Greg Sutcliffe
h4. Release (RC or final) build parameters
58 9 Dominic Cleal
59 6 Greg Sutcliffe
This should only be done by the release nanny.
60
61
Make necessary merges from theforeman/foreman-packaging:deb/develop to deb/<release> (e.g. '1.3')
62
repoowner must be theforeman
63
Select project (core/proxy/installer)
64 1 Greg Sutcliffe
Select OS ('all', or a specific target)
65 6 Greg Sutcliffe
branch must be x.y (e.g 1.3) - this is both the branch of foreman-packaging _and_ the target component on deb.theforeman.org
66 1 Greg Sutcliffe
leave PR field blank
67
gitrelease must be unticked
68
69
Final packages will be pushed to deb.theforeman.org/<os>/<branch>
70
71 9 Dominic Cleal
h3. Nightly build parameters
72
73 8 Greg Sutcliffe
These are triggered normally by other jobs in Jenkins, e.g. the test_develop job triggers the Deb matrix job with the following parameters.
74 6 Greg Sutcliffe
75 8 Greg Sutcliffe
repoowner will be theforeman
76
project is determined by the upstream job (eg test_develop will trigger project=foreman)
77
branch will be develop
78
pr be blank
79
gitrelease will be ticked
80 6 Greg Sutcliffe
81
h2. Tooling
82 1 Greg Sutcliffe
83
h3. Pbuilder
84
85
"Pbuilder":http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html is the system used internally by jenkins to build the packages for each of the architectures. It should just work, but if there are problems, you can look at the pbuilder module in foreman-infra for more detail.
86 6 Greg Sutcliffe
87
h3. Jenkins
88
89
Jenkins is used to prepare the package sources, and the upstream sources, execute the appropriate pbuilder OS environment, and then upload the resulting packages to the appropriate deb repo.
90
91
h3. Freight
92
93
Freight is used to add individual debs to the overall repo. The following actions are performed:
94
95
* Packages are rsync'ed to (staging)deb.theforeman.org from the Jenkins package build
96 20 Dominic Cleal
* @freight-add <debs> apt/<os>/<repo>@ is used to stage the debs for adding
97
* @freight-cache apt/<os>@ is used to rebuild the repo and publish it
98
99
Key locations:
100
101
* /home/freight(stage) is the home directory of the user managing the repo, contains freight.conf
102
* /var/www/freight(stage) is the freight library, where new packages are staged to with freight-add
103
* /var/www/vhosts/(stage)deb/htdocs/ is the freight cache, the public viewable archive
104 6 Greg Sutcliffe
105
There are cron jobs which clean out old nightly, and scratch debs after 7 days.
106
107
This should all happen automatically, this information is only provided for background. See foreman-infra/puppet/modules/freight for more code.
108 1 Greg Sutcliffe
109
h2. Project sources
110
111
h3. foreman
112 6 Greg Sutcliffe
113 1 Greg Sutcliffe
Foreman{,proxy,installer} is built from the packaging constructs in the foreman-packaging repo:
114 6 Greg Sutcliffe
115 1 Greg Sutcliffe
* https://github.com/theforeman/foreman-packaging/tree/deb/develop (or 1.2/1.3/etc)
116
117
This repo is pulled in by Jenkins, thefore build preparation consists of appropriately updating this repo before queueing the Jenkins jobs.
118
119
h4. Initial tests
120 6 Greg Sutcliffe
121 1 Greg Sutcliffe
The nightly repo tracks the develop branch. At the point when a release candidate is forked, nightly will most likely be at the same commit, and will already be built (as it is triggered by a commit to develop). Thus, if the nightly packages work, one can conclude the packaging constructs in deb/nightly are probably sane. Minimal testing using a scratch build from your own repo is recommended, as above.
122
123
h4. Release candidate
124
125
The following steps are used to step up a release candidate build from nightly
126 4 Greg Sutcliffe
127 6 Greg Sutcliffe
* Cherry-pick appropriate commits from deb/develop into deb/<version> (e.g. deb/1.3)
128 1 Greg Sutcliffe
* Update the UPSTREAM variable in <package>/build_vars.sh to point to correct git-tag for the release (e.g. UPSTREAM=1.3.0-RC1) (TODO: make this a jenkins option like the rpms?)
129 8 Greg Sutcliffe
* Create a new dir for the new release (e.g. 1.3) and add an "rc" symlink to it (eg staged/apt/<os>/1.3 and staged/apt/<os>/rc -> 1.3)
130 13 Greg Sutcliffe
** Something like this should copy all the current deps from nightly:
131 17 Greg Sutcliffe
*** Log in to deb.theforeman.org
132
*** sudo -u freight -s /bin/bash
133
*** cd /var/www/freight/apt/wheezy && cp -r nightly 1.4 && ln -snf 1.4 rc && rm -f rc/*9999*
134
*** delete older versions of packages from rc/* (eg where you have ruby-clamp_0.6.2-1_all.deb & ruby-clamp_0.6.2-2_all.deb, delete 0.6.2-1)
135
*** cd /var/www/vhosts/deb/htdocs/pool/ && rm -rf wheezy/rc  # clean up the web cache too
136
** Repeat for the other debian OSs and the plugins repo
137 13 Greg Sutcliffe
** Use freight-cache to create new rc repo:
138 19 Dominic Cleal
*** freight cache -v -c ~/freight.conf
139 18 Greg Sutcliffe
** It's also necessary to copy the current build deps to the staging repo. Once the above is done, you can use:
140
*** sudo -u freightstage -s /bin/bash
141
*** find /var/www/freight/apt/wheezy/1.7/ -iname "*deb" -exec freight-add -v -c /home/freightstage/freight.conf {} apt/wheezy/theforeman-1.7 \;
142
*** repeat for each OS
143 19 Dominic Cleal
*** freight cache -v -c ~/freight.conf
144 1 Greg Sutcliffe
* Update the changelog with a block announcing the release candidate, e.g:
145 6 Greg Sutcliffe
<pre>
146 1 Greg Sutcliffe
foreman (1.3.0~rc1-1) stable; urgency=low
147 6 Greg Sutcliffe
148 1 Greg Sutcliffe
  * Release 1.3.0 RC1
149
150
 -- Greg Sutcliffe <greg.sutcliffe@gmail.com>  Fri, 12 Sep 2013 16:31:00 +0100
151 6 Greg Sutcliffe
152 7 Greg Sutcliffe
</pre>
153 1 Greg Sutcliffe
** Note the Debian changelog is _extremely_ sensitive to format and syntax. If in doubt, do this on a debian box and use the "dch -i" tool to add the entry
154
** Use <release-in-three-parts>~rc<rc-version>-1<packaging-release> as the release to ensure the first stable release can use "-1"
155
* PR / Commit the git changes to foreman-packaging:deb/<version>
156
* Queue the Jenkins jobs!
157
* Test the resulting packages
158
159
h4. Stable release
160
161
The stable release is almost identical to the rc release, with good reason ;)
162 6 Greg Sutcliffe
163
* Foreman-packaging/deb/1.3 should already be up-to-date
164 1 Greg Sutcliffe
* Update the UPSTREAM variable in <package>/build_vars.sh to point to correct git-tag for the release (e.g. UPSTREAM=1.3.0)
165 8 Greg Sutcliffe
* Add any new dependencies from rc to stable (most should already be present, as RCs and Stable release share a numbered repo)
166
** Same process on server2, but also update the symlinks for "stable" to point to the new release, and delete the rc symlinks
167 1 Greg Sutcliffe
* Update the changelog with a block announcing the release, e.g:
168 6 Greg Sutcliffe
<pre>
169 1 Greg Sutcliffe
foreman (1.3.0-1) stable; urgency=low
170
(as above for the rest)
171 7 Greg Sutcliffe
</pre>
172 1 Greg Sutcliffe
* PR / Commit the git changes to foreman-packaging:deb/1.3
173
* Queue the Jenkins jobs!
174
* Test the resulting packages
175 6 Greg Sutcliffe
176 1 Greg Sutcliffe
Foreman-packaging/deb/<version> remains as a branch for point-releases on old versions
177 5 Greg Sutcliffe
178
h3. foreman-proxy
179
180
The exact same process above is used to release the proxy.
181
182
h3. foreman-installer
183 6 Greg Sutcliffe
184 5 Greg Sutcliffe
The exact same process above is used to release the proxy.
185 6 Greg Sutcliffe
186 1 Greg Sutcliffe
h3. Dependencies
187 6 Greg Sutcliffe
188 1 Greg Sutcliffe
Gem dependencies can be built from the same foreman-packaging/<version|develop> repo as the core packages, using the 'packaging_build_deb_dependency' job. Currently only gems can be built, using a modified gem2deb command which replaces the auto-generated debian/ dir with the one from our git repo during the build.
189 6 Greg Sutcliffe
190 1 Greg Sutcliffe
TODO(13/9/13): Non-gem dependencies cannot currently be built automatically, this will be fixed at some point.
191 6 Greg Sutcliffe
192 7 Greg Sutcliffe
If you need to manually build
193 1 Greg Sutcliffe
Foreman's Debian dependencies should all be present in the nightly repo in order to make the nightly builds work. Work is underway to store the appropriate packaging constructs in the foreman-packaging repo, such that they can be rebuilt if needed. The packages are built using pbuilder manually. Rough notes on performing this build are:
194 6 Greg Sutcliffe
195 7 Greg Sutcliffe
* Log in to server9 (the debian slave node)
196 6 Greg Sutcliffe
* Clone foreman-packaging:deb/develop
197 1 Greg Sutcliffe
* cd to the appropriate package and prepare the sources
198 6 Greg Sutcliffe
* run pdebuild-<os_arch> (eg. pdebuild-squeeze32) to generate a package for that OS/Arch
199 1 Greg Sutcliffe
* Upload to server2 and use freight to add to the repo
200
201 15 Greg Sutcliffe
h3. Plugins
202
203
Gem dependencies can be built from the same foreman-packaging/<version|develop> repo as the core packages, using the 'packaging_build_deb_plugin' job. This uses sources from the named user and packaging branch, and simply creates a bundler.d fine to be installed into the core Foreman package. It accepts a gitrelease flag if you wish to test changes
204
205 1 Greg Sutcliffe
TODO: Document this properly.
206
207
h2. Nightly Builds
208
209 6 Greg Sutcliffe
Nightly builds are fully automated. Next time they break I'll document what I did to fix them. Full automation is possible because of the fact that the Debian packages use gem vendoring to ship a cache of prebuilt gems to the user; thus, they are far less susceptible to changes in the Gemfile (downside: they'll never be accepted upstream)
210
211 1 Greg Sutcliffe
Work has begun to start building the foreman packages using proper system gems, see http://etherpad-domcleal.rhcloud.com/p/debian-packaging for how it's going. Currently we build the installer (based on kafo) using proper system gems. Foreman itself is targeted for 1.4