Debian Packaging » History » Revision 6
Revision 5 (Greg Sutcliffe, 05/24/2013 07:25 AM) → Revision 6/36 (Greg Sutcliffe, 09/13/2013 10:07 AM)
h1. Debian Packaging {{toc}} h2. Artifacts The following artifacts are generated from the RPM build process and form part of any release: * foreman, including database and compute resource subpackages * foreman-installer * foreman-proxy * dependencies h2. Tooling h3. Jenkins Jenkins is used to automate the package builds, using the jobs: * http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_dependency (for gems) http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman_individual-DEB * http://ci.theforeman.org/view/Packaging/job/packaging_build_deb_coreproject (for core/proxy/installer) These jobs rely on matching the branching structure of the core projects to the branching in foreman-packaging. In addition these matrix jobs build regular 'scratch' packages from the develop HEAD (13/9/13: currently disabled awaiting rewrite) http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman-proxy_individual-DEB * http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman_matrix * http://ci.theforeman.org/view/Packaging/job/packaging_build_foreman-proxy_matrix The matrix jobs only presently build the nightly packages, thus RC/Stable releases will be queued manually using the basic single jobs. The configuration depends on the build single jobs require you want: h4. Scratch build parameters Want to test your own branch? Push your packaging updates to your fork of foreman-packaging:deb/develop Change repoowner to your GitHub account Select project (core/proxy/installer) Select OS ('all', or a specific target) branch should be 'develop' gitrelease should be ticked scratch must be ticked sepcify: * The packages will be on stagingdeb.theforeman.org/<os>/<your github user> h4. Release (RC or final) build parameters This should only be done by the target release nanny. Make necessary merges from theforeman/foreman-packaging:deb/develop to deb/<release> (e.g. '1.3') repoowner must be theforeman Select project (core/proxy/installer) Select (nightly/rc/stable) * The target OS ('all', or a specific target) branch must be x.y (e.g 1.3) - this is both the branch of foreman-packaging _and_ the (wheezy/squeeze/precise) * The target component on deb.theforeman.org gitrelease must be unticked run once with scratch ticked before doing the final build (scratch will be at stagingdeb.theforeman.org/<os>/theforeman) Final packages will be pushed to deb.theforeman.org/<os>/<branch> h4. Nightly build parameters These are triggered normally by other jobs in Jenkins, e.g. the test_develop job triggers the RPM matrix job with the following parameters. repoowner must be theforeman branch must be develop gitrelease must be ticked scratch should be unticked h2. Tooling arch (32/64bit) h3. Pbuilder "Pbuilder":http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html is the system used internally by jenkins to build the packages for each of the architectures. It should just work, but if there are problems, you can look at the pbuilder module in foreman-infra for more detail. h3. Jenkins Jenkins is used to prepare the package sources, and the upstream sources, execute the appropriate pbuilder OS environment, and then upload the resulting packages to the appropriate deb repo. h3. Freight Freight is used to add individual debs to the overall repo. The following actions are performed: * Packages are rsync'ed to (staging)deb.theforeman.org from the Jenkins package build * Freight-add <debs> apt/<os>/<repo> is used to stage the debs for adding * Freight-cache apt/<os> is used to rebuild the repo and publish it There are cron jobs which clean out old nightly, and scratch debs after 7 days. This should all happen automatically, this information is only provided for background. See foreman-infra/puppet/modules/freight for more code. h2. Project sources h3. foreman Foreman{,proxy,installer} Foreman(-proxy) is built from the packaging constructs in the foreman-packaging foreman-rpms repo: * https://github.com/theforeman/foreman-packaging/tree/deb/develop https://github.com/theforeman/foreman-rpms/tree/deb/stable (or 1.2/1.3/etc) rc/nightly) This repo is pulled in by Jenkins, thefore build preparation consists of appropriately updating this repo before queueing the Jenkins jobs. h4. Initial tests The nightly repo tracks the develop branch. At the point when a release candidate is forked, nightly will most likely be at the same commit, and will already be built (as it is triggered by a commit to develop). Thus, if the nightly packages work, one can conclude the packaging constructs in deb/nightly are probably sane. Minimal testing using a scratch build from your own repo is recommended, as above. h4. Release candidate The following steps are used to step up a release candidate build from nightly * Cherry-pick appropriate Merge deb/nightly into deb/rc ** This _may_ require a force overwrite, but hopefully the merge will be clean as the only commits from deb/develop into deb/<version> (e.g. deb/1.3) direct to the rc repo should be the <package>/build.sh updates ** If a force overwrite is used, revert the changes to the <package>/build.sh files as they should only contain the SHAsum lines for the package * Update the UPSTREAM variable SHAsums in <package>/build_vars.sh <package>/build.sh to point to correct git-tag SHAsum for the release (e.g. UPSTREAM=1.3.0-RC1) (TODO: make this a jenkins option like the rpms?) * Add any new dependencies from nightly to <version> rc (these will have been built during the previous release cycle for nightly, so at release it's just a matter of importing the debs) (TODO: improve this, server login should not be needed) ** Log in to server2 ** su -u freight - ** Use freight-add / freight cache to add new deps to rc repo, e.g: *** freight add staged/apt/wheezy/nightly/*sinatra* apt/wheezy/rc *** freight cache -v * Update the changelog with a block announcing the release candidate, e.g: <pre> foreman (1.3.0~rc1-1) stable; (1.2.0+debian0.1) squeeze; urgency=low * Release 1.3.0 1.2.0 RC1 -- Greg Sutcliffe <greg.sutcliffe@gmail.com> Fri, 12 Sep Mon, 20 May 2013 16:31:00 +0100 </pre> ** Note the Debian changelog is _extremely_ sensitive to format and syntax. If in doubt, do this on a debian box and use the "dch -i" tool to add the entry ** Use <release-in-three-parts>~rc<rc-version>-1<packaging-release> <release-in-three-parts>+debian0.<rc-version> as the release to ensure the first stable release can use "-1" debian1 * PR / Commit the git changes to foreman-rpms:deb/<version> foreman-rpms:deb/rc * Queue the Jenkins jobs! * Test the resulting packages h4. Stable release The stable release is almost identical to the rc release, with good reason ;) * Foreman-packaging/deb/1.3 Merge deb/rc into deb/stable ** This is far more likely to be clean, very little should be merged here without haven't already be up-to-date * Update been through rc ** If required, update the UPSTREAM variable SHAsums in <package>/build_vars.sh to point to correct git-tag for the release (e.g. UPSTREAM=1.3.0) <package>/build.sh again * Add any new dependencies from rc to stable ** Same process on server2 * Update the changelog with a block announcing the release, e.g: <pre> foreman (1.3.0-1) stable; (1.2.0+debian1) squeeze; urgency=low (as above for the rest) </pre> * PR / Commit the git changes to foreman-rpms:deb/1.3 foreman-rpms:deb/stable * Queue the Jenkins jobs! * Test the resulting packages Foreman-packaging/deb/<version> remains as Assuming a branch for point-releases on old versions successful release, the changelog should be merged back to deb/nightly so that history is preserved during the next release cycle h3. foreman-proxy The exact same process above is used to release the proxy. h3. foreman-installer The exact same process above installer is used built using "fpm":https://github.com/jordansissel/fpm which provides a lightweight build mechanism that also supports Debian packages. A small script wraps this: * https://github.com/theforeman/foreman-rpms/blob/master/fpm/build_installer_pkgs.sh The script currently assumes you are able to release write to @/usr/share/foreman-installer@ rather than building in a temporary directory. Patches gladly accepted. A job on Jenkins can build this by checking out foreman-rpms and running the proxy. h3. Dependencies Gem dependencies script: * http://ci.theforeman.org/job/packaging_build_foreman-installer/ This can be built from triggered manually, where it takes the same foreman-packaging/<version|develop> repo as version number and the core packages, using build type (rpm/deb) to generate the 'packaging_build_deb_dependency' job. Currently only gems can be build artifact. Once built, using test the deb on at least one Debian OS to ensure it works (it's a modified gem2deb command which replaces very simple package, so it will install on all of them if it installs on one), then upload it to freight: ** Log in to server2 ** su -u freight - ** curl http://ci.theforeman.org/path/to/built/deb ** freight add ./installer.deb apt/wheezy/rc #(and the auto-generated debian/ dir other repos) ** freight cache -v (TODO: automate this properly along with the one from our git repo during the build. TODO(13/9/13): Non-gem dependencies cannot currently be built automatically, this will be fixed at some point. If you need to manually build other packages) h3. Dependencies Foreman's Debian dependencies should all be present in the nightly repo in order to make the nightly builds work. Work is underway to store the appropriate packaging constructs in the foreman-rpms repo, such that they can be rebuilt if needed. The packages are built using pbuilder manually. Rough notes on performing this build are: * Log in to server9 (the debian slave node) * Clone foreman-rpms:deb/develop foreman-rpms:deb/nightly * cd to the appropriate package and prepare the sources * run pdebuild-<os_arch> (eg. pdebuild-squeeze32) to generate a package for that OS/Arch * Upload to server2 serevr2 and use freight to add to the repo TODO: Document this properly. h2. Nightly Builds Nightly builds are fully automated. Next time they break I'll document what I did to fix them. Full automation is possible because of the fact that the Debian packages use gem vendoring to ship a cache of prebuilt gems to the user; thus, they are far less susceptible to changes in the Gemfile (downside: they'll never be accepted upstream) Work has begun to start building the foreman packages using proper system gems, see http://etherpad-domcleal.rhcloud.com/p/debian-packaging for how it's going. Currently we build the installer (based on kafo) using proper system gems. Foreman itself is targeted for 1.4