Project

General

Profile

History

RPM Packaging

Artifacts

The following artifacts are generated from the RPM build process and form part of any release:

  • foreman, including database and compute resource subpackages
  • foreman-installer
  • foreman-proxy
  • foreman-release
  • foreman-selinux
  • foreman-release-scl
  • rubygem-hammer_cli, rubygem-hammer_cli_foreman
  • tfm SCL package
  • SCL dependencies
  • non-SCL dependencies

Tooling

Koji

Koji is used to build packages in an automated manner. The instance we use is shared with the Katello project and is located at http://koji.katello.org/koji/, and configuration information is on the Koji wiki page.

tito

tito manages the building of Foreman from the source tree into Koji. The functionality we're using is limited, since we don't use it to tag each build in the upstream Foreman repository.

Software collections

Software collections is a packaging technique used for our EL builds to install our dependencies under /opt. We use the public collections for Ruby on Rails in combination with our own software collection for our extra packages.

Project sources

In our third major revision of RPM packaging, sources are to be united in the foreman-packaging repo for core projects and dependencies using tito for source management and building into Koji (perhaps Copr later). This mirrors the Debian packaging and allows us to work on packaging separately to core development which will make releasing easier.

The primary branch to work on in foreman-packaging is rpm/develop:

Documentation for working in this repository exists in its README file:

Building packages

Packages are usually built using tito on maintainer workstations directly to Koji. Precise commands are given in the README above.

Tito and Koji ensure that this process is as reproducible as possible. This can also be done using the following Jenkins jobs, but the underlying process is identical.

Test builds (scratch builds)

This works for either a Foreman (sub)project or dependency, and on every supported OS.

  1. Commit your changes locally
  2. Either:
    1. tito release koji-foreman --test --scratch
    2. Push your change to GitHub, then use packaging_build_rpm
      • project: foreman, rubygem-fog etc.
      • scratch: true
      • gitrelease: true
      • pr_git_url: URL to your GitHub fork
      • pr_git_ref: your git branch name

Releasing builds

When releasing either a Foreman (sub)project or dependency:

  1. Check the scratch build works
  2. Tag it, run tito tag --use-version=1.2.3-1 or tito tag if incrementing only the release (packaging-only change)
  3. Push the change as instructed
  4. Either:
    1. Use Jenkins job packaging_build_rpm
      • project: foreman, rubygem-fog etc.
      • scratch: false
    2. tito release koji-foreman

tfm software collection

Software collections provide Ruby and Ruby on Rails packages on EL 6 and 7, and we use these to run Foreman and Hammer CLI. Foreman ships a "tfm" (standing for "The ForeMan") collection containing many extra gems and that depends on the Ruby and Ruby on Rails software collections (ruby193 and v8314 at the time of writing).

Entrypoint into the collection is usually /usr/bin/tfm-ruby which is provided by tfm-runtime, and activates the tfm collection before running ruby. Activating tfm also enables dependent collections ruby193 and v8314. Passenger calls /usr/bin/tfm-ruby as its PassengerRuby interpreter and packages such as tfm-rubygem_hammer_cli use tfm-ruby in shebang lines.

The dependencies of tfm can be changed to switch Ruby on Rails versions, and packages inside the tfm collection rebuilt to match.

Foreman SCL buildroots (foreman-nightly-rhel*-build) all have tfm-build pre-installed via the Koji build/srpm-build groups (koji add-group-pkg). This ensures any package built in these roots will have %{scl} expanded to "tfm". They also have ruby193-scldevel and v8314-devel pre-installed to enable building of the tfm SCL meta-package itself.

Workflow

Nightly builds

Nightlies are built from the "develop" branch using tito release, which builds into the Koji tags:

  • foreman-nightly-rhel6
  • foreman-nightly-rhel7
  • foreman-nightly-fedora19

The proxy and installer are non-SCL and so get built into these tags:

  • foreman-nightly-nonscl-rhel6
  • foreman-nightly-nonscl-rhel7
  • foreman-nightly-fedora19

Builds occur when the test jobs on Jenkins complete successfully, so the test_develop job (which tests core Foreman's develop branch) runs tests and then uses "rake pkg:generate_source" which creates a tarball or .gem file of the project as appropriate. Each test job then has this file published as a build artifact.

The test job then triggers packaging_build_nightly_rpm with a project name parameter, which then triggers packaging_build_rpm with parameters to build a package into the nightly koji tags. Each subproject has a core test job which has SCM polling enabled (so it picks up on changes in git) and after tests succeed, triggers the RPM build job.

Builds are handled by the packaging_build_rpm job on Jenkins, which clones the packaging repo and runs tito to fetch the Jenkins artifacts (source files) and submits it to Koji for all supported OSes.

Scroll on down to the mashing and repo sync sections for info on how the resulting RPMs get published from koji.

Releases

At the start of a release cycle (using 1.2 as an example), the 1.2-stable branch is created from develop. All further changes are cherry-picked from develop to 1.2-stable.

The spec version at this point will reference the upcoming release, e.g.

Version: 1.2.0
Release: 0.develop%{?dist}

This must then be changed to 1.3.0 in develop immediately, and the release changed to RC1 in 1.2-stable when RC1's being released. This ensures that nightlies built from develop are correctly shown as newer than any builds from 1.2-stable, since all changes to 1.2 must be made via develop cherry-picking. Version numbers must be in a three-digit format, e.g. 1.2.0 rather than 1.2.

The release string must be changed to "0.1.RC1" for the first release candidate, then incremented for subsequent updates. The digit before the RC can be used for packaging-only changes within a single RC, e.g. "0.2.RC1". For the final release, the release string must be "1", then incremented only for packaging changes. This is as per the Fedora packaging guidelines: http://fedoraproject.org/wiki/Packaging:NamingGuidelines#Package_Versioning

An example version and release history:

  • 1.2.0-0.develop.el6 (develop nightlies, prior to 1.2-stable)
  • 1.2.0-0.1.RC1.el6
  • 1.2.0-0.2.RC1.el6
  • 1.2.0-0.3.RC2.el6
  • 1.2.0-1.el6
  • 1.2.0-2.el6
  • 1.2.1-0.1.RC1.el6
  • 1.2.1-1.el6
  • 1.3.0-0.develop.el6 (develop nightlies, post 1.2-stable)

Packages are built directly from the 1.2-stable branch into Koji using tito. Dedicated tags are used for release:

  • foreman-1.2-rhel6
  • foreman-1.2-rhel7
  • foreman-1.2-fedora19
  • foreman-1.2-nonscl-rhel6
  • foreman-1.2-nonscl-rhel7

These are created during the Release_Process using the helper script in rpm/develop to clone the existing nightly tags. This means we inherit a snapshot of the state of nightly as the basis for the release.

Repo signing

Any RC or final release must be GPG signed to ensure authenticity using our key. The signatures are written back to koji for each release and when the repo is "mashed", the signatures get used.

Make sure you have the key for the release being signed (use gpg_path to point to the GPG homedir) and in ~/.rpmmacros set:

%_gpg_path /home/dcleal/code/foreman/infra/gnupg/1.6
%_gpg_name packages@theforeman.org

First download all RPMs from the release tags into an empty directory. Update the VERSION and GPGKEY used for this release:

GPGKEY=7DFE6FC2
VERSION=1.13
OSES="rhel7 fedora24" 

GPGKEY=F06D8950
VERSION=1.14
OSES="rhel7 fedora24" 

for j in $OSES; do
  tags=foreman-$VERSION-$j
  [[ $j =~ rhel ]] && tags="$tags foreman-$VERSION-nonscl-$j" 
  for i in $tags; do
    koji -c ~/.koji/katello-config list-tagged --latest --quiet --inherit --sigs $i ; done \
    | sed 's!^!:!' \
    | perl -ane '$F[1] =~ s!\.src$!! or next; $R{$F[1]} = 1; $S{$F[1]} = 1 if lc($F[0]) eq lc(":'$GPGKEY'");
      END { print map "$_\n", grep { not exists $S{$_} } sort keys %R }' \
    | while read i ; do koji -c ~/.koji/katello-config download-build --debuginfo $i ; 
  done;
done;

Next, sign them all:

rpmsign --addsign *.rpm

Then upload the signatures:

kkoji import-sig *.rpm

Check that http://koji.katello.org/packages/foreman/1.5.1/2.el6/data/sigcache/1aa043b8/ exists, update the version and release numbers.

Now update the RPMs (this will take a while):

ls *.src.rpm | sed 's!\.src\.rpm$!!' | xargs -t -n20 -P2 koji -c ~/.koji/katello-config write-signed-rpm $(echo $GPGKEY | tr 'A-Z' 'a-z')

Check that http://koji.katello.org/packages/foreman/1.5.1/2.el6/data/signed/1aa043b8/ exists, updating the version again.

The Katello wiki is another good reference for this procedure. Note the tag names and key are different though.

Signing "extra" (centos-release-scl-*) RPMs

99% of our RPMs come from Koji, but we add a couple of centos-release-scl-* release RPMs from CentOS so we can configure SCL repositories quickly from our own. Since RPM signatures are normally stored in Koji and used as part of the mash process, this requires a different signing method.

The mash script downloads these files based on the files in https://github.com/theforeman/foreman-packaging/tree/rpm/develop/extras and combines them with the signed RPMs from Koji when generating repositories.

They are cached in /mnt/koji/releases/extras/ on the Koji server on a per-repo/release basis, so when starting a new release, they should be downloaded, signed manually and then the mash script will include the signed version alongside the signed RPMs from Koji.

  1. run mash script (e.g. /usr/local/bin/foreman-mash-split-1.7.py)
  2. download /mnt/koji/releases/extras/foreman-1.7-rhel*-dist/*/*.rpm
  3. rpmsign --addsign *.rpm
  4. upload and replace existing files
  5. re-run mash script

Repo mashing

A koji admin can then "mash" the repos, which creates a repo output (e.g. http://koji.katello.org/releases/yum/foreman-1.2/) from all the appropriate tags. This uses the comps file in the foreman-packaging repository (under rpm/develop or appropriate rpm/1.2-stable type branch) to list every package that will make it into the repo: https://github.com/theforeman/foreman-packaging/tree/rpm/develop/comps

This will overwrite the repo that's currently there.

The command depends on the release, e.g.
  • 1.2: /usr/local/bin/foreman-mash-split-1.2.py
  • nightly: /usr/local/bin/foreman-mash-split.py

The mash script can be triggered from Jenkins (which runs it on Koji over SSH) using the packaging_mash_rpms job. This job runs immediately after "packaging_build_nightly_rpm" to generate nightly repositories and triggers the next synchronisation step once complete.

Repo testing and synchronization

Repos are synchronized to the yum.theforeman.org virtual host on the main web server via capistrano, which is run on Jenkins (as with other remote deployments). The packaging_publish_rpm job handles the whole process of pulling repos from Koji, testing them and pushing to yum.tf.org.

It is a matrix job for each distro, so each distro works and can fail independently of the others. For each job (i.e. OS), it will run:

When releasing Foreman RCs or releases, run this packaging_publish_rpm job first with "dryrun" set to true, which will perform all test steps, but skips the final deployment step.

The packaging_publish_nightly_rpm job is triggered by the nightly mash cronjob using curl (and a token configured on the job), which calls the generic "packaging_publish_rpm" job with the necessary parameters set for nightly repo synchronisation.