Koji Setup

Our intance is running at AWS EC2 (us-east-1) as i3.xlarge instance (4 CPUs, 32 GB RAM, 900 GB SSD NVMe). It is running CentOS 7 from EBS volume (8 GB). The account is managed by Bryan Kearney, access to the instance has few people including Lukas Zapletal, Eric Helms and Mike McCune. If you need to be there, contact them.

Volumes and mounts

The instance has two EBS volumes attached:

  • /dev/sda1 - root
  • /dev/sdx - data volume (/mnt/koji available as /dev/xvdx1)

The instance must be running in a security group with ports 22, 80, 443, 873 (rsyncd), 44323 (read only monitoring microsite) allowed (all IPv4 TCP).

Root EBS volume is mounted via UUID in fstab:

UUID=29342a0b-e20f-4676-9ecf-dfdf02ef6683 / xfs defaults 0 0

Note that other volumes are not present in fstab, this is to prevent booting into emergency mode when the VM is respinned on a different hypervisor with different or empty ephemeral or EBS storage configuration. All the rest is mounted in /etc/rc.local:

swapon /dev/nvme0n1p1
mount /dev/nvme0n1p2 /mnt/tmp -o defaults,noatime,nodiratime
mount /dev/xvdx1 /mnt/koji -o defaults,noatime,nodiratime
hostnamectl set-hostname koji.katello.org
systemctl restart pmcd pmlogger pmwebd
mount | grep /mnt/koji && systemctl restart rsyncd
mount | grep /mnt/koji && systemctl start postgresql
systemctl start httpd
mount | grep /mnt/koji && mount | grep /mnt/tmp && systemctl start kojid
mount | grep /mnt/koji && mount | grep /mnt/tmp && systemctl start kojira

On our current VM flavour there is a local SSD NVMe storage (/dev/nvme0n1) with two partitions created (50/50). The first one is swap and the second one is mounted as /mnt/tmp where koji does all the work. This volume needs to be fast, it grows over the time and contains temporary files (built packages, build logs, support files).

The main data folder where PostgreSQL database and koji generated repositories and external repositories are present is on EBS volume mounted as /mnt/koji. Note this was created as ext4 which can sometimes lead to mkfs, perhaps xfs would be better fit for our use case.

Services required for koji (postgresql, httpd, kojid, kojira, rsyncd) are only started if required volumes are mounted.

Hostname

The instance has a floating IP, in /etc/hosts we have an entry for that:

34.224.159.44 koji.katello.org kojihub.katello.org koji kojihub

When the IP changes, make sure this does change as well.

When new instance is booted via AWS, it will have a random hostname assigned. In the rc.local we set the hostname to koji.katello.org on every boot.

Backups

There is a cron job (/etc/cron.weekly/koji-backup) that performs two actions every week:

Full PostgreSQL database dump into /mnt/koji/backups/postgres.

File system backup of /mnt/tmp (ephemeral storage) into /mnt/koji/backups/ephemeral. This backup skips all files named RPM (these are not needed), duplicity tool is used, no encryption is done. The main purpose of this backup is to store required filesystem structure so koji can be quickly brought up after crash. Since the backup mostly contains directories and build logs, it is not big. To restore that, use:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force --no-encryption

Both backups does not have any rotation and need to be deleted every year. The full backup script looks like:

#!/bin/bash
/usr/bin/duplicity --full-if-older-than 1M --no-encryption -vWARNING --exclude '/mnt/tmp/**/*rpm' /mnt/tmp file:///mnt/koji/backups/ephemeral
date=`date +"%Y%m%d"`
filename="/mnt/koji/backups/postgres/koji_${date}.dump" 
pg_dump -Fc -f "$filename" -U koji koji

Updates

We are running CentOS 7 with Koji (1.11) installed from EPEL7 and mrepo package installed from Fedora. Since koji was bumped in the EPEL to non-compatible version, we have disabled EPEL7 repository for now.

Update procedure:

  • Shutdown the instance.
  • Make a root EBS volume snapshot.
  • Start the instance and check koji is operating properly.
  • Perform the yum upgrade in screen session. Do not let koji to upgrade.
  • Reboot and check koji services.
  • Send announcement and edit the history section down here.

If update fails, shutdown the failed VM, create new AMI from the snapshot, mount the data EBS volume and spawn a new instance.

Monitoring

Root email is redirected via /etc/aliases to lzap. Logwatch is configured to send emails daily coming to emails specified in /etc/logwatch/conf/logwatch.conf to the same e-mail. One additional extra free space check is installed in /etc/cron.daily/check-disk-space to send extra email via cron when any of volumes is consuming more than 90% space.

There is a PCP daemon (pmcd) running on the instance and pmlogger active creating archives of performance data in /var/log/pcp/pmlogger (30 days rotation). It is possible to connect and see live data at http://koji.katello.org:44323/grafana/ or http://koji.katello.org:44323/graphite/ (both are JS only applications with read-only API into PCP archives). PCP data can be exported into external Graphite, InfluxDB or Zabbix applications, but none of that has been configured.

Local postfix instance is not configured for relaying so it delivers directly, which can be problem for some freemails/gmail. But it works just fine with redhat.com SMTP server, check delivery first with sendmail if configuring new email.

Adding external repository

Edit /etc/mrepo.conf and add required entry. Warning: If you delete any repository from here, mrepo will delete all packages from disk and resyncing can take a while. Do not run mrepo with dry-run option as it does perform the changes.

Then run this script in a screen session (takes several hours depending on content) which will resync ALL external repositories. Inform packagers about fresh content which may break future builds as we do not run sync regularly.

[root@koji ~]# cat /usr/local/bin/koji-sync-external-repos 
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}' koji regen-repo --nowait '{}'

Verify that repodata was regenerated for external repos:

find /mnt/koji/external-repos/ -name repomd.xml

Finally, add external repositories to koji database.

koji add-external-repo fedora27-external-repo file:///mnt/koji/external-repos/www/fedora27-\$arch/RPMS.os/
Created external repo 58
koji add-external-repo fedora27-updates-external-repo file:///mnt/koji/external-repos/www/fedora27-\$arch/RPMS.updates/
Created external repo 59

History

  • Summer 2017 - Instance was installed according to Koji wiki guide.
  • 2017/10/27 - Hardware failure. Instance was relaunched on a different hypervisor. Edited fstab and this page was created.