h1. Release Process

{{toc}}

For each major release (i.e. not patch releases), the project selects a release manager who's responsible for taking the develop branch through to release.  Their high level tasks and policies are described on [[Release_Management]], while this document covers specific steps (down to individual commands) that need to be run at each step of the release.

Please amend these lists as you see free, and as you find what works and what doesn't work.

These examples are all based around a 1.2 series release, so any reference to "1.2" should be replaced by the current release, "1.1" by your previous release, and "1.3" by the next release.

h2. Near-end of development phase

# Announce upcoming branching to discourse development category ("example":https://community.theforeman.org/t/1-19-branching-in-core-heads-up/10048) a month before

# Change @$latest@ and @$next@ parameters on @web@ class to point to the new version numbers (see @foreman-infra/puppet/modules/web/manifests/init.pp@)

# File a new release tracker on theforeman.org for "Foreman 1.2 manual" ("example":https://github.com/theforeman/theforeman.org/issues/494)

# Refresh unattended templates from the community-templates to Foreman core, by running @script/sync_templates.sh@

# Create a new branch in community-templates for upcoming Foreman release (e.g. 1.2-stable)

# Add new languages that are at a "reasonable completion on Transifex":https://www.transifex.com/foreman/foreman/foreman/ to *develop*

# Generate a release GPG key using [[GPG_Keys]]

#* publish to keyserver

#* security.md on the website, add public key fingerprint to static/keys

#* create @releases/1.2/RPM-GPG-KEY-foreman@ on yum.theforeman.org

# Ask plugin authors to start extracting i18n strings and pushing the changes into develop/master git branches so Transiflex can pick it up

h2. Branching for series

When ready to branch for release candidates.

h3. Package build systems

# Clone tags and create build targets in "Koji":http://koji.katello.org/koji using koji/copy-tags-commands.sh in foreman-packaging/rpm/develop or use the tool_belt

#* Make sure you have a kkoji section in ~/.koji/config , you can find instructions in [[Koji]]

#* This generates commands that create foreman-1.2-rhel7 etc. as clones of nightly tags and foreman-plugins-1.2-rhel7 etc. with inheritance from the new Foreman tags

#* Untag all git/nightly builds: @for t in foreman-1.2-rhel7 foreman-1.2-nonscl-rhel7 foreman-1.2-fedora24; do kkoji list-tagged $t  | grep git | awk '{print $1}' | xargs kkoji untag-build  $t; done@

# Create mash scripts and configuration on Koji

#* copy /usr/local/bin/foreman-mash-split-1.*.py to foreman-mash-split-1.2.py, update the OSes and version numbers at the bottom

#* copy /etc/mash/foreman-1.*-*.mash and foreman-plugins-1.*-*.mash, update the version numbers, add/remove files for OS changes, set GPG key ID and ensure strict_keys=True

# Add new plugin tags (i.e. 1.2) to Koji plugins mash script (foreman-mash-split-plugins.py), remove old ones (keep three)

# Add version (1.2) to these jobs in axes and/or combination filters, remove old ones (keep three)

## "foremanPluginsRelease":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/jenkins_job_builder/files/theforeman.org/pipelines/release/foremanPluginsRelease.groovy

## "packaging_build_deb_coreproject":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/packaging_build_deb_coreproject.yaml

## "packaging_build_deb_dependency":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/packaging_build_deb_dependency.yaml

## "release_test":https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/jenkins_job_builder/files/theforeman.org/yaml/jobs/release_test.yaml

## "release_push_deb":http://ci.theforeman.org/job/release_push_deb/

## "release_push_rpm":http://ci.theforeman.org/job/release_push_rpm/

# Clone Debian nightly repos to 1.2 using [[Debian_Packaging#Project-sources|copy/freight instructions]]

# Edit "foreman-packaging's PR template":https://github.com/theforeman/foreman-packaging/blob/master/.github/PULL_REQUEST_TEMPLATE.md to add 1.2 and remove the old release

h3. Main code repos

# Make releases of installer modules, usually new minor or major versions

## Currently puppet-puppet, puppet-foreman, puppet-dhcp, puppet-dns, puppet-foreman_proxy, puppet-tftp, puppet-git

# In foreman *develop* run @make -C locale tx-update@

# In foreman, smart-proxy, foreman-installer and foreman-selinux branch *develop* to *1.2-stable* (*from this point* cherry-picks into this branch only from this point using: @git cherry-pick -x SHA@)

# In foreman-installer, branch *1.2-stable*,

## run @bundle exec rake pin_modules@ to change Puppetfile from git references to specific version ranges (e.g. @>= 1.3.1 < 1.4.0@)

## run @sed -i '/Puppetfile.lock/d' .gitignore@ to remove Puppetfile.lock from .gitignore

## run @librarian-puppet install@

## commit Puppetfile, Puppetfile.lock and .gitignore changes

# In foreman-packaging for RPMs:

## branch *rpm/develop* to *rpm/1.2*

## on *rpm/1.2*

### in rel-eng/releasers.conf and rel-eng/tito.props, replace "nightly" with "1.2"

### change @foreman/foreman.repo@ URLs from @/nightly@ to @/releases/1.2@, change "nightly" in name to "1.2" and enable gpg checking

### change @foreman/foreman-plugins.repo@ URLs from @/plugins/nightly@ to @/plugins/1.2@, change "nightly" in name to "1.2" and DO NOT enable GPG checking

### update @foreman/foreman.gpg@ with the release public key

## on *rpm/develop*

### change foreman/foreman-proxy/foreman-installer/foreman-selinux version to 1.3.0, add %changelog entry

### change tag_suffix to @.fm1_3@ in rel-eng/tito.props, and rel-eng/releasers.conf

# In foreman-packaging for debs:

## on *deb/develop*, update debian/*/*/changelog with entries from 1.1, commit with message "Sync 1.1.x releases into changelogs"

## branch *deb/develop* to *deb/1.2*

## on *deb/develop*, run @scripts/changelog.rb -v 1.3.0-1 -m "Bump changelog to 1.3.0 to match VERSION" debian/*/*/changelog@ 

# In foreman *develop* commit with message "Bump version to 1.3-develop":

## change VERSION to 1.3.0-develop

## change package.json version field to 1.3.0

## run @extras/changelog@

# In smart-proxy *develop* commit with message "Bump version to 1.3-develop":

## change VERSION to 1.3.0-develop

## run @extra/changelog@

# In foreman-selinux *develop* commit with message "Bump version to 1.3-develop":

## change VERSION to 1.3.0-develop

## run @extras/changelog@

# In foreman-installer *develop* commit with message "Bump version to 1.3-develop":

## change VERSION to 1.3.0-develop

h3. Other systems

# Create release schedule page for next version (1.3) linked from [[Development_Resources]]

# Add next version number (1.3.0) to Redmine under "Versions":https://projects.theforeman.org/projects/foreman/settings/versions and all relevant subprojects (installer, proxy, and selinux)

# Add first patch release (1.2.1) to Redmine in the same way

# Update foreman-dev with "translations status":https://www.transifex.com/projects/p/foreman/ to encourage 100% translations before release

# Announce string freeze date on discourse and send announcement via https://www.transifex.com/foreman/foreman/announcements/

# Create test_1_2_stable.yaml and test_proxy_1_2_stable.yaml for JJB in foreman-infra, remove the oldest version to keep last 3

# Ensure current Foreman deprecations for the next release are removed in *develop*

h3. Rails for RPMs

The Rails stack is built as a separate SCL in Copr but then mirrored for a given Foreman version on yum.theforeman.org. When branching, a version should be created for the Foreman version being released:

<pre>

# ssh yum.theforeman.org

# cd /var/www/vhosts/yum/htdocs/rails/

# cp -rf foreman-nightly foreman-<version>

</pre>

The foreman-release-scl RPM then needs updating to point at this new repository in SCL repo file.

Koji needs to be updated to include a new external repository pointing to this new rails repository for builds.

  * ssh root@koji.katello.org

  * edit /etc/mrepo.conf and add foreman-<version> entry for new version

  * Now run: <pre>mrepo -ug -f -v foreman-rails-<version></pre>

  * Go back to developer machine

  * Add the new external repo: `koji add-external-repo foreman-rails-<version>-rhel-7 file:///mnt/koji/external-repos/www/foreman-rails-<version>-$arch/RPMS.ror51/`

h2. Preparing a release

Version numbers for release candidates start with "1.2.0-RC1" (always hyphenated).  The first release after RC would be "1.2.0".  Change examples below as appropriate.

h3. Writing release notes

# Copy website manual content ("theforeman.org repo":http://github.com/theforeman/theforeman.org) from previous version to this version ("example":https://github.com/theforeman/theforeman.org/pull/43)

#* add version number to documentation.md dropdown menu

# Update manual if applicable for any additional installation steps

# Draft release notes in markdown ("example":https://gist.github.com/domcleal/5567450), with these sections (and do not use personal pronouns):

## Headline features: half a dozen important features with a few sentences description each

## Upgrade notes: all important notices that users must be aware of before upgrading

## Release notes: bullet point list by category of most changes, excluding bug fixes for issues introduced during the release cycle, include link to bug numbers. You can auto-generate changes using a script from "theforeman.org":https://github.com/theforeman/theforeman.org/blob/gh-pages/scripts/release_notes.rb or use the 'changelog' command in "tool_belt":https://github.com/theforeman/tool_belt

## CLI release notes are taken from the hammer-cli and hammer-cli-foreman changelogs

## Link to installer changelogs and note versions being used

# Get the apipie doc and place it in api/$release_num

# Change links in _includes/version.html for 'latest' and 'latest stable'

h3. Preparing code

# Request Hammer CLI releases from maintainers if desired

# Make patch releases of installer modules that have important changes

#* Branch to MAJ.MIN-stable if recent changes to the module aren't suitable for patch (x.y.z) release

# Compare tagged packages in nightly vs. release koji tag and re-tag any updated dependencies that are required

# Add a new "Redmine version":https://projects.theforeman.org/projects/foreman/settings/versions for the next minor, unless the series is EOL

# Remove/change release field for any open Redmine tickets assigned to the release still (next minor, unset it or reject)

# Change Redmine version state to _Closed_

# List all issues targeted at the release, order by Closed date ascending and use @git cherry-pick -x@ to cherry pick from develop to *1.2-stable* branch

# Clone "tool_belt":https://github.com/theforeman/tool_belt (foreman branch) and run:

## @./tools.rb setup-environment configs/foreman_12.yaml@

## @./tools.rb cherry-picks --version 1.2.0 configs/foreman_12.yaml@

## Verify tickets in the @cherry_picks_1.2.0@ file are accounted for or additional cherry pick them

h2. Tagging a release

# In foreman *1.2-stable*:

## run @make -C locale tx-update@ (if Transifex has not switched to the next major release yet, usually after .2)

## run @script/sync_templates.sh@

## change VERSION to 1.2.0

## change package.json version field to 1.2.0

## run @extras/changelog@

## commit with message "Release 1.2.0"

# In smart-proxy *1.2-stable* commit with message "Release 1.2.0":

## change VERSION to 1.2.0

## run @extra/changelog@

# In foreman-selinux *1.2-stable* commit with message "Release 1.2.0":

## change VERSION to 1.2.0

## run extras/changelog

# In foreman-installer *1.2-stable* commit with message "Release 1.2.0":

## change VERSION to 1.2.0

# Before tagging, make sure the test_xxxx_stable versions of each of the projects are green in http://ci.theforeman.org/

# Tag commits in foreman, smart-proxy, foreman-installer and foreman-selinux: @git tag -m "Release 1.2.0" 1.2.0@

# Push to the project repos: @git push project && git push project 1.2.0@

# Run the Jenkins "Release Pipeline":http://ci.theforeman.org/view/Release%20pipeline/ to create tarballs

# Verify tarballs are present on downloads.theforeman.org, download, sign and upload detached signatures

## @gpg --homedir gnupg/1.2 -b -u packages@theforeman.org foreman-1.2.0.tar.bz2@ (requires 1.2 release GPG key generated above)

# If for some reason there was an issue with the tarballs that required uploading new tarballs, CDN cache should be invalidated so that the builders use the updated tarballs.

h2. Packaging a release

# In foreman-packaging rpm/1.2 branch, change foreman.spec, foreman-proxy.spec, foreman-selinux.spec, foreman-installer.spec:

## set version to "1.2.0"

## For RCs: set release to "0.1%{?dotalpha....", uncomment alphatag* (replace # with %), change to RC1

## For non-RCs: set release to "1%{?dotalpha....", comment alphatag* (replace % with #)

## in each package dir, remove the old tarball, run @spectool -g *.spec@ and @git annex add *.tar.bz2@

## commit with message "Release 1.2.0"

## perform RPM scratch build of each project (see [[RPM_Packaging]])

## tag each project with @tito tag --keep-version --changelog "Release 1.2.0"@

# Create new branches as described in [[Debian_Packaging#foreman]]

# Update changelog files for debs:

## @scripts/changelog.rb -v 1.2.0-1 -m "1.2.0 released" debian/*/*/changelog@

# Trigger next step of release pipeline: @release_packages@

# Use [[RPM_Packaging]] to sign RPMs

# Trigger next step of release pipeline: @release_mash@

# Trigger next step of release pipeline: @release_test@

# Trigger next step of release pipeline: @release_push_deb@ and @release_push_rpm@

h2. Completion tasks

If releasing the first non-RC release (i.e. 1.2.0), start with:

# update @$stable@ class parameter on @web@ class to point to the new version number (see @foreman-infra/puppet/modules/web/manifests/init.pp@)

# deb.theforeman.org, changing stable:

## change web's /var/www/freight/apt/*/stable symlinks to the new version number

## @rm -rf /var/www/vhosts/deb/htdocs/pool/*/stable/@

## re-run freight cache

# Update theforeman.org:

## @documentation.md@ change default version

## @_config.yml@ foreman_version and quickstart link

## @_layouts/manual.html@ latestversion

## @plugins/index.md@ version

## add a blog post describing release highlights etc.

# If there is any open version on https://projects.theforeman.org/projects/foreman/settings/versions for y-2 (e.g. 1.0.2 when releasing 1.2.0) that hasn't been released, close the version and move any issues assigned to it to the next y-1 release (e.g. 1.1.3).

For all releases:

# Update theforeman.org:

## release notes from earlier steps (use scripts/release_notes.rb in theforeman.org repository)

## @_includes/version.html@

## @_includes/latest_news.html@ entry

## @security.md@ for any released CVE fixes

# Publish announcement notice in Release Announcements: (https://community.theforeman.org/c/release-announcements)

# Link to announcement post on IRC, Twitter, Google+ and update IRC /topic

# Update the release schedule on [[Development_Resources]] to enter the date and for the following release(s)

# Complete [[Security Process]] for any released CVE fixes (oss-security list)

# Final only: Update Wikipedia Foreman entry at: https://en.wikipedia.org/wiki/Foreman_(software)