Project

General

Profile

Debian Packaging » History » Version 7

Greg Sutcliffe, 09/13/2013 10:21 AM

1 1 Greg Sutcliffe
h1. Debian Packaging
2
3
{{toc}}
4
5
h2. Artifacts
6
7 7 Greg Sutcliffe
The following artifacts are generated from the Deb build process and form part of any release:
8 1 Greg Sutcliffe
9
* foreman, including database and compute resource subpackages
10
* foreman-installer
11
* foreman-proxy
12 2 Greg Sutcliffe
* dependencies
13 1 Greg Sutcliffe
14 6 Greg Sutcliffe
h2. Jenkins
15 1 Greg Sutcliffe
16
Jenkins is used to automate the package builds, using the jobs:
17
18 6 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_dependency (for gems)
19
* http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_coreproject (for core/proxy/installer)
20
21
These jobs rely on matching the branching structure of the core projects to the branching in foreman-packaging.
22
23
In addition these matrix jobs build regular 'scratch' packages from the develop HEAD (13/9/13: currently disabled awaiting rewrite)
24 1 Greg Sutcliffe
* http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman_matrix
25
* http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman-proxy_matrix
26
27 6 Greg Sutcliffe
The matrix jobs only presently build the nightly packages, thus RC/Stable releases will be queued manually using the basic jobs. The configuration depends on the build you want:
28 1 Greg Sutcliffe
29 6 Greg Sutcliffe
h4. Scratch build parameters
30
Want to test your own branch?
31 1 Greg Sutcliffe
32 6 Greg Sutcliffe
Push your packaging updates to your fork of foreman-packaging:deb/develop
33
Change repoowner to your GitHub account
34
Select project (core/proxy/installer)
35
Select OS ('all', or a specific target)
36
branch should be 'develop'
37
gitrelease should be ticked
38
scratch must be ticked
39
40
The packages will be on stagingdeb.theforeman.org/<os>/<your github user>
41
42
h4. Release (RC or final) build parameters
43
This should only be done by the release nanny.
44
45
Make necessary merges from theforeman/foreman-packaging:deb/develop to deb/<release> (e.g. '1.3')
46
repoowner must be theforeman
47
Select project (core/proxy/installer)
48
Select OS ('all', or a specific target)
49
branch must be x.y (e.g 1.3) - this is both the branch of foreman-packaging _and_ the target component on deb.theforeman.org
50
gitrelease must be unticked
51
run once with scratch ticked before doing the final build (scratch will be at stagingdeb.theforeman.org/<os>/theforeman)
52
53
Final packages will be pushed to deb.theforeman.org/<os>/<branch>
54
55
h4. Nightly build parameters
56 7 Greg Sutcliffe
These are triggered normally by other jobs in Jenkins, e.g. the test_develop job triggers (TODO: 13/9/13: _will_ trigger, not working yet) the Deb matrix job with the following parameters.
57 6 Greg Sutcliffe
58
repoowner must be theforeman
59
branch must be develop
60
gitrelease must be ticked
61
scratch should be unticked
62
63
h2. Tooling
64
65 1 Greg Sutcliffe
h3. Pbuilder
66
67
"Pbuilder":http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html is the system used internally by jenkins to build the packages for each of the architectures. It should just work, but if there are problems, you can look at the pbuilder module in foreman-infra for more detail.
68
69 6 Greg Sutcliffe
h3. Jenkins
70
71
Jenkins is used to prepare the package sources, and the upstream sources, execute the appropriate pbuilder OS environment, and then upload the resulting packages to the appropriate deb repo.
72
73
h3. Freight
74
75
Freight is used to add individual debs to the overall repo. The following actions are performed:
76
77
* Packages are rsync'ed to (staging)deb.theforeman.org from the Jenkins package build
78
* Freight-add <debs> apt/<os>/<repo> is used to stage the debs for adding
79
* Freight-cache apt/<os> is used to rebuild the repo and publish it
80
81
There are cron jobs which clean out old nightly, and scratch debs after 7 days.
82
83
This should all happen automatically, this information is only provided for background. See foreman-infra/puppet/modules/freight for more code.
84
85 1 Greg Sutcliffe
h2. Project sources
86
87
h3. foreman
88
89 6 Greg Sutcliffe
Foreman{,proxy,installer} is built from the packaging constructs in the foreman-packaging repo:
90 1 Greg Sutcliffe
91 6 Greg Sutcliffe
* https://github.com/theforeman/foreman-packaging/tree/deb/develop (or 1.2/1.3/etc)
92 1 Greg Sutcliffe
93
This repo is pulled in by Jenkins, thefore build preparation consists of appropriately updating this repo before queueing the Jenkins jobs.
94
95
h4. Initial tests
96
97 6 Greg Sutcliffe
The nightly repo tracks the develop branch. At the point when a release candidate is forked, nightly will most likely be at the same commit, and will already be built (as it is triggered by a commit to develop). Thus, if the nightly packages work, one can conclude the packaging constructs in deb/nightly are probably sane. Minimal testing using a scratch build from your own repo is recommended, as above.
98 1 Greg Sutcliffe
99
h4. Release candidate
100
101
The following steps are used to step up a release candidate build from nightly
102 4 Greg Sutcliffe
103 6 Greg Sutcliffe
* Cherry-pick appropriate commits from deb/develop into deb/<version> (e.g. deb/1.3)
104
* Update the UPSTREAM variable in <package>/build_vars.sh to point to correct git-tag for the release (e.g. UPSTREAM=1.3.0-RC1) (TODO: make this a jenkins option like the rpms?)
105
* Add any new dependencies from nightly to <version> (these will have been built during the previous release cycle for nightly, so at release it's just a matter of importing the debs) (TODO: improve this, server login should not be needed)
106 1 Greg Sutcliffe
** Log in to server2
107 4 Greg Sutcliffe
** su -u freight -
108 1 Greg Sutcliffe
** Use freight-add / freight cache to add new deps to rc repo, e.g:
109
*** freight add staged/apt/wheezy/nightly/*sinatra* apt/wheezy/rc                    
110
*** freight cache -v
111
* Update the changelog with a block announcing the release candidate, e.g:
112
<pre>
113 6 Greg Sutcliffe
foreman (1.3.0~rc1-1) stable; urgency=low
114 1 Greg Sutcliffe
115 6 Greg Sutcliffe
  * Release 1.3.0 RC1
116 1 Greg Sutcliffe
117 6 Greg Sutcliffe
 -- Greg Sutcliffe <greg.sutcliffe@gmail.com>  Fri, 12 Sep 2013 16:31:00 +0100
118 1 Greg Sutcliffe
119
</pre>
120
** Note the Debian changelog is _extremely_ sensitive to format and syntax. If in doubt, do this on a debian box and use the "dch -i" tool to add the entry
121 6 Greg Sutcliffe
** Use <release-in-three-parts>~rc<rc-version>-1<packaging-release> as the release to ensure the first stable release can use "-1"
122 7 Greg Sutcliffe
* PR / Commit the git changes to foreman-packaging:deb/<version>
123 1 Greg Sutcliffe
* Queue the Jenkins jobs!
124
* Test the resulting packages
125
126
h4. Stable release
127
128
The stable release is almost identical to the rc release, with good reason ;)
129
130 6 Greg Sutcliffe
* Foreman-packaging/deb/1.3 should already be up-to-date
131
* Update the UPSTREAM variable in <package>/build_vars.sh to point to correct git-tag for the release (e.g. UPSTREAM=1.3.0)
132 1 Greg Sutcliffe
* Add any new dependencies from rc to stable
133
** Same process on server2
134
* Update the changelog with a block announcing the release, e.g:
135
<pre>
136 6 Greg Sutcliffe
foreman (1.3.0-1) stable; urgency=low
137 1 Greg Sutcliffe
(as above for the rest)
138
</pre>
139 7 Greg Sutcliffe
* PR / Commit the git changes to foreman-packaging:deb/1.3
140 1 Greg Sutcliffe
* Queue the Jenkins jobs!
141
* Test the resulting packages
142
143 6 Greg Sutcliffe
Foreman-packaging/deb/<version> remains as a branch for point-releases on old versions
144 1 Greg Sutcliffe
145 5 Greg Sutcliffe
h3. foreman-proxy
146
147
The exact same process above is used to release the proxy.
148
149
h3. foreman-installer
150
151 6 Greg Sutcliffe
The exact same process above is used to release the proxy.
152 5 Greg Sutcliffe
153 6 Greg Sutcliffe
h3. Dependencies
154 1 Greg Sutcliffe
155 6 Greg Sutcliffe
Gem dependencies can be built from the same foreman-packaging/<version|develop> repo as the core packages, using the 'packaging_build_deb_dependency' job. Currently only gems can be built, using a modified gem2deb command which replaces the auto-generated debian/ dir with the one from our git repo during the build.
156 1 Greg Sutcliffe
157 6 Greg Sutcliffe
TODO(13/9/13): Non-gem dependencies cannot currently be built automatically, this will be fixed at some point.
158 1 Greg Sutcliffe
159 6 Greg Sutcliffe
If you need to manually build
160 7 Greg Sutcliffe
Foreman's Debian dependencies should all be present in the nightly repo in order to make the nightly builds work. Work is underway to store the appropriate packaging constructs in the foreman-packaging repo, such that they can be rebuilt if needed. The packages are built using pbuilder manually. Rough notes on performing this build are:
161 1 Greg Sutcliffe
162 6 Greg Sutcliffe
* Log in to server9 (the debian slave node)
163 7 Greg Sutcliffe
* Clone foreman-packaging:deb/develop
164 6 Greg Sutcliffe
* cd to the appropriate package and prepare the sources
165 1 Greg Sutcliffe
* run pdebuild-<os_arch> (eg. pdebuild-squeeze32) to generate a package for that OS/Arch
166 6 Greg Sutcliffe
* Upload to server2 and use freight to add to the repo
167 1 Greg Sutcliffe
168
TODO: Document this properly.
169
170
h2. Nightly Builds
171
172
Nightly builds are fully automated. Next time they break I'll document what I did to fix them. Full automation is possible because of the fact that the Debian packages use gem vendoring to ship a cache of prebuilt gems to the user; thus, they are far less susceptible to changes in the Gemfile (downside: they'll never be accepted upstream)
173 6 Greg Sutcliffe
174
Work has begun to start building the foreman packages using proper system gems, see http://etherpad-domcleal.rhcloud.com/p/debian-packaging for how it's going. Currently we build the installer (based on kafo) using proper system gems. Foreman itself is targeted for 1.4