Release Process

For each major release (i.e. not patch releases), the project selects a release manager who's responsible for taking the develop branch through to release. Their high level tasks and policies are described on Release_Management, while this document covers specific steps (down to individual commands) that need to be run at each step of the release.

Please amend these lists as you see free, and as you find what works and what doesn't work.

These examples are all based around a 1.2 series release, so any reference to "1.2" should be replaced by the current release, "1.1" by your previous release, and "1.3" by the next release.

Near-end of development phase

  1. Announce upcoming branching to foreman-dev (example) a month before
  2. Copy website manual content (theforeman.org repo) from previous version to this version (example)
    • add version number to documentation.md dropdown menu
  3. Change $latest and $next parameters on web class to point to the new version numbers (see foreman-infra/puppet/modules/web/manifests/init.pp)
  4. File a new release tracker on theforeman.org for "Foreman 1.2 manual" (example)
  5. Refresh unattended templates from the community-templates to Foreman core, by running script/sync_templates.sh
  6. Create a new branch in community-templates for upcoming Foreman release (e.g. 1.2-stable)
  7. Add new languages that are at a reasonable completion on Transifex to develop
  8. Generate a release GPG key using GPG_Keys
    • publish to keyserver
    • security.md on the website
    • create releases/1.2/RPM-GPG-KEY-foreman on yum.theforeman.org
  9. Add release to "Found in release" list in redmine: http://projects.theforeman.org/custom_fields/4/edit

Branching for series

When ready to branch for release candidates.

Package build systems

  1. Clone tags and create build targets in Koji using koji/copy-tags-commands.sh in foreman-packaging/rpm/develop
    • Make sure you have a kkoji section in ~/.koji/config , you can find instructions in http://projects.theforeman.org/projects/foreman/wiki/Koji
    • This generates commands that create foreman-1.2-rhel7 etc. as clones of nightly tags and foreman-plugins-1.2-rhel7 etc. with inheritance from the new Foreman tags
    • Untag all git/nightly builds: for t in foreman-1.2-rhel7 foreman-1.2-nonscl-rhel7 foreman-1.2-fedora24; do kkoji list-tagged $t | grep git | awk '{print $1}' | xargs kkoji untag-build $t; done
  2. Create mash scripts and configuration on Koji
    • copy /usr/local/bin/foreman-mash-split-1.*.py to foreman-mash-split-1.2.py, update the OSes and version numbers at the bottom
    • copy /etc/mash/foreman-1.*-*.mash and foreman-plugins-1.*-*.mash, update the version numbers, add/remove files for OS changes, set GPG key ID and ensure strict_keys=True
  3. Add new plugin tags (i.e. 1.2) to Koji plugins mash script (foreman-mash-split-plugins.py), remove old ones (keep three)
  4. Add version (1.2) to these jobs in axes and/or combination filters, remove old ones (keep three)
    1. release_plugins_push (note, it will be added to tests later)
    2. packaging_plugin_retag
    3. packaging_retag
    4. packaging_build_coreproject
    5. packaging_build_dep_dependency
    6. release_test
    7. release_push_deb
    8. release_push_rpm
  5. Clone Debian nightly repos to 1.2 using copy/freight instructions
  6. Edit foreman-packaging's PR template to add 1.2 and remove the old release

Main code repos

  1. Make releases of installer modules, usually new minor or major versions
  2. In foreman develop run make -C locale tx-update
  3. In foreman, smart-proxy, foreman-installer and foreman-selinux branch develop to 1.2-stable (from this point cherry-picks into this branch only from this point using: git cherry-pick -x SHA)
  4. In foreman-installer, branch 1.2-stable,
    1. change Puppetfile from git references to specific version ranges (e.g. >= 1.3.1 < 1.4.0)
    2. remove Puppetfile.lock from .gitignore
    3. run librarian-puppet install, commit Puppetfile, Puppetfile.lock and .gitignore changes
  5. In foreman-packaging for RPMs:
    1. branch rpm/develop to rpm/1.2
    2. on rpm/1.2
      1. in rel-eng/releasers.conf and rel-eng/tito.props, replace "nightly" with "1.2"
      2. change foreman/foreman.repo URLs from /nightly to /releases/1.2, change "nightly" in name to "1.2" and enable gpg checking
      3. change foreman/foreman-plugins.repo URLs from /plugins/nightly to /plugins/1.2, change "nightly" in name to "1.2" and DO NOT enable GPG checking
      4. update foreman/foreman.gpg with the release public key
    3. on rpm/develop
      1. change foreman/foreman-proxy/foreman-installer/foreman-selinux version to 1.3.0, add %changelog entry
      2. change tag_suffix to .fm1_3 in rel-eng/tito.props, and rel-eng/releasers.conf
  6. In foreman-packaging for debs:
    1. on deb/develop, update debian/*/*/changelog with entries from 1.1, commit with message "Sync 1.1.x releases into changelogs"
    2. branch deb/develop to deb/1.2
    3. on deb/develop, run scripts/changelog.rb -v 1.3.0-1 -m "Bump changelog to 1.3.0 to match VERSION" debian/*/*/changelog
  7. In foreman develop commit with message "Bump version to 1.3-develop":
    1. change VERSION to 1.3.0-develop
    2. change package.json version field to 1.3.0
    3. run extras/changelog
  8. In smart-proxy develop commit with message "Bump version to 1.3-develop":
    1. change VERSION to 1.3.0-develop
    2. run extra/changelog
  9. In foreman-selinux develop commit with message "Bump version to 1.3-develop":
    1. change VERSION to 1.3.0-develop
    2. run extras/changelog
  10. In foreman-installer develop commit with message "Bump version to 1.3-develop":
    1. change VERSION to 1.3.0-develop

Other systems

  1. Create release schedule page for next version (1.3) linked from Development_Resources
  2. Add next version number (1.3.0) to Redmine under Releases, sharing "With subprojects"
  3. Add first patch release (1.2.1) to Redmine in the same way
  4. Update foreman-dev with translations status to encourage 100% translations before release, announce string freeze date
  5. Change Transifex resource auto-update URL to point to the 1.2-stable branch
    • usually this happens after the .2 of the previous release (e.g. 1.1.2) has been released
  6. Set up test_1_2_stable job on Jenkins, copy from existing stable or test_develop (don't use dots in the name, it breaks)
  7. Set up test_proxy_1_2_stable job on Jenkins in the same way
  8. Ensure current Foreman deprecations for the next release are removed in develop

Preparing a release

Version numbers for release candidates start with "1.2.0-RC1" (always hyphenated). The first release after RC would be "1.2.0". Change examples below as appropriate.

Writing release notes

  1. Update manual if applicable for any additional installation steps
  2. Draft release notes in markdown (example), with these sections (and do not use personal pronouns):
    1. Headline features: half a dozen important features with a few sentences description each
    2. Upgrade notes: all important notices that users must be aware of before upgrading
    3. Release notes: bullet point list by category of most changes, excluding bug fixes for issues introduced during the release cycle, include link to bug numbers (helper vim search and replace command bellow)
    4. CLI release notes are taken from the hammer-cli and hammer-cli-foreman changelogs
    5. Link to installer changelogs and note versions being used

Helper vim command to replace all (#1234) bugs with links:

%s/(\(#\)\([0-9]\+\))/(\[\1\2]\(http:\/\/projects.theforeman.org\/issues\/\2))/g

Preparing code

  1. Request Hammer CLI releases from maintainers if desired
  2. Make patch releases of installer modules that have important changes
    • Branch to MAJ.MIN-stable if recent changes to the module aren't suitable for patch (x.y.z) release
  3. Compare tagged packages in nightly vs. release koji tag and re-tag any updated dependencies that are required
  4. Add a new Redmine release for the next minor, unless the series is EOL
  5. Remove/change release field for any open Redmine tickets assigned to the release still (next minor, unset it or reject)
  6. Change Redmine release state to Closed
  7. List all issues targeted at the release, order by Closed date ascending and use git cherry-pick -x to cherry pick from develop to 1.2-stable branch
  8. Clone tool_belt (foreman branch) and run:
    1. ./tools.rb setup-environment configs/foreman_12.yaml
    2. ./tools.rb cherry-picks --version 1.2.0 configs/foreman_12.yaml
    3. Verify tickets in the cherry_picks_1.2.0 file are accounted for or additional cherry pick them

Tagging a release

  1. In foreman 1.2-stable run:
    1. make -C locale tx-update (if Transifex has not switched to the next major release yet, usually after .2)
    2. script/sync_templates.sh
  2. In foreman 1.2-stable commit with message "Release 1.2.0":
    1. change VERSION to 1.2.0
    2. change package.json version field to 1.2.0
    3. run extras/changelog
  3. In smart-proxy 1.2-stable commit with message "Release 1.2.0":
    1. change VERSION to 1.2.0
    2. run extra/changelog
  4. In foreman-selinux 1.2-stable commit with message "Release 1.2.0":
    1. change VERSION to 1.2.0
    2. run extras/changelog
  5. In foreman-installer 1.2-stable commit with message "Release 1.2.0":
    1. change VERSION to 1.2.0
  6. Tag commits in foreman, smart-proxy, foreman-installer and foreman-selinux: git tag -m "Release 1.2.0" 1.2.0
  7. Push to the project repos: git push project && git push project 1.2.0
  8. Run the Jenkins Release Pipeline to create tarballs
  9. Verify tarballs are present on downloads.theforeman.org, download, sign and upload detached signatures
    1. gpg --homedir gnupg/1.2 -b -u packages@theforeman.org foreman-1.2.0.tar.bz2 (requires 1.2 release GPG key generated above)

Packaging a release

  1. In foreman-packaging rpm/1.2 branch, change foreman.spec, foreman-proxy.spec, foreman-selinux.spec, foreman-installer.spec:
    1. set version to "1.2.0"
    2. For RCs: set release to "0.1%{?dotalpha....", uncomment alphatag* (replace # with %), change to RC1
    3. For non-RCs: set release to "1%{?dotalpha....", comment alphatag* (replace % with #)
    4. in each package dir, run spectool -g *.spec and git annex add *.tar.bz2
    5. commit with message "Release 1.2.0"
    6. perform RPM scratch build of each project (see RPM_Packaging)
    7. tag each project with tito tag --keep-version --changelog "Release 1.2.0"
  2. Debian_Packaging: Update changelog files
    1. scripts/changelog.rb -v 1.2.0-1 -m "1.2.0 released" debian/*/*/changelog
  3. Trigger next step of release pipeline: release_packages
  4. Use RPM_Packaging to sign RPMs
  5. Trigger next step of release pipeline: release_mash
  6. Trigger next step of release pipeline: release_test
  7. Trigger next step of release pipeline: release_push_deb and release_push_rpm

Completion tasks

If releasing the first non-RC release (i.e. 1.2.0), start with:
  1. update $stable class parameter on web class to point to the new version number (see foreman-infra/puppet/modules/web/manifests/init.pp)
  2. deb.theforeman.org, changing stable:
    1. change web's /var/www/freight/apt/*/stable symlinks to the new version number
    2. rm -rf /var/www/vhosts/deb/htdocs/pool/*/stable/
    3. re-run freight cache
  3. Update theforeman.org:
    1. documentation.md change default version
    2. _config.yml foreman_version and quickstart link
    3. _layouts/manual.html latestversion
    4. plugins/index.md version
    5. add a blog post describing release highlights etc.

If releasing the first release candidate (i.e. RC1), start with:

  1. add version (1.2) to release_plugins_test job

For all releases:

  1. Add release to "Found in release" list in redmine: http://projects.theforeman.org/custom_fields/4/edit
  2. Update theforeman.org:
    1. release notes from earlier steps
    2. _includes/version.html
    3. _includes/latest_news.html entry
    4. security.md for any released CVE fixes
  3. Send announcement e-mail to foreman-announce, CC foreman-users, set reply-to to foreman-users
  4. Link to announcement e-mail on IRC, Twitter, Google+ and update IRC /topic
  5. Update the release schedule on Development_Resources to enter the date and for the following release(s)
  6. Complete Security Process for any released CVE fixes (oss-security list)
  7. Final only: Update Wikipedia Foreman entry at: https://en.wikipedia.org/wiki/Foreman_(software)