Plugins » Katello

h1. Pulp 3 Integration

This page is meant to capture links to various resources around pulp3 integration

* Initial Integration Document: https://docs.google.com/document/d/1eFM4H9BteA_nTDbUkRJgEiqSttXfAtB9-51a_OuwVJw/edit?usp=sharing

* Initial Community Post: https://community.theforeman.org/t/pulp-3-migration/10474

* Youtube Videos: https://www.youtube.com/playlist?list=PL59WSIBtIpv7tbjnsEAWNcdiOKAmjjgqE

* Tasks List: https://docs.google.com/spreadsheets/d/1PFx-VAEXlivlqDYR9INMW26AUWtuMS_XtEnwhFVP5Rc/edit?usp=sharing

* Main Tracker: https://projects.theforeman.org/issues/26987

h2. Katello Dev box with pulp3:

Dev boxes are deployed with pulp3 by default, via an rpm installation

* vagrant up katello-dev-box

* vagrant ssh katello-dev-box

h2. Katello Dev box with pulp3 on master:

Dev boxes come installed and configured to use yum, docker and files on pulp3. 

To enable ansible collection content type, you need to perform the following steps on the devel box:

#  <pre>

#  sudo pip3 install pulp-ansible==0.7.0

#  sudo mv /usr/local/lib/python3.6/site-packages/pulp_ansibl* /usr/lib/python3.6/site-packages/

#  sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' python3-django-admin migrate --no-input

#  </pre>

#  In file ~/foreman/config/settings.plugins.d/katello.yaml, 

#  Add :ansible_collection: true to the dict of content types.

#  Restart foreman server and refresh smart proxy features.

h2. Updating to a New Pulp Version in master & nightly:

1.  Request build of new version via the discourse Development discussion unless its already built: http://yum.theforeman.org/pulpcore/  (https://github.com/theforeman/pulpcore-packaging).  List the plugin versions you would like (usually the latest of them all) and request a time frame (try to give ~2 weeks). Example: https://community.theforeman.org/t/request-for-pulpcore-3-13-builds/23855      This command can help you get the list:

<pre>

for i in pulpcore pulp-file pulp-rpm pulp-cli pulp-container pulp-ansible pulp-certguard pulp-deb pulp-python; do echo $i `curl -s https://pypi.org/pypi/$i/json |jq  .info.version`; done

</pre>

1.  Send an email to satellite-delivery@redhat.com and link to the discourse thread.

2.  Once the new yum repo is available, update your pulp packages locally on your devel environment.  If you are wanting to jump ahead, try to pip3 install the new versions and pray.

3. Add the new Pulp version to puppet-pulpcore if it isn't there already

* Example commit: https://github.com/theforeman/puppet-pulpcore/commit/6f09c2ff328fac9fcaf1388d9112a5130db3e2d7

4.  Identify any installer changes, open installer PRs

5.  Identify katello updates:

* Updating pulp client gem bindings in katello.gemspec 

* re-record vcr cassettes

* Identify any katello code changes needed to fix test cases

* Open katello PR with changes but do not merge yet

6. Update katello-repos in https://github.com/theforeman/foreman-packaging to point to the new version

* If the new bindings are required for whatever reason, get them merged first.

7. Update forklift to deploy with the new version: https://github.com/theforeman/forklift/blob/master/vagrant/config/versions.yaml

8. Open a PR to update the client binding RPMs at foreman-packaging (but don't merge yet)

9. Merge the Katello gemspec update PR first, and then the client bindings packaging PR

10. Update rubygem-katello at foreman-packaging to have the new bindings requirements:

<pre>

# First, build a gem of the Katello version in question

# Then, cd into the foreman-packaging directory

gem2rpm -t gem2rpm/foreman_plugin.spec.erb /path/to/katello.gem | ./update-requirements specfile - packages/katello/rubygem-katello/rubygem-katello.spec

# Now, open up a PR with the rubygem-katello.spec updates

</pre>

h2. Using a 2nd server as a pulp server, for use in development.

If you have a devel environment and want to spin up a 2nd dev box via pulp_installer to test with, for example to dev or test against a pulp version that doesn't have rpm builds yet.

1.  Spinup a fresh dev environment, or reset your old one with 'bundle exec rake katello:reset'

2.  On your hypervisor, setup pulp-installer:

<pre>

git clone https://github.com/pulp/pulp_installer

cd pulp_installer

git clone  https://github.com/theforeman/forklift.git

</pre>

3.  edit pulp_installer/example.user-config.yml  uncomment plugins to install

4.  Spin up PULP3 Box:

<pre>

vagrant up pulp3-sandbox-centos7

vagant ssh pulp3-sandbox-centos7

</pre>

5.  grab the ip address and hostname of the PULP3 box:

<pre>

sudo yum install net-tools

hostname

ifconfig

</pre>

6.  Add hostname/ipaddress to /etc/hosts on the KATELLO server, as root:

<pre>

   echo "192.168.0.8  pulp3-sandbox-centos7.example.com" >> /etc/hosts

</pre> 

7.  Reset the admin password on the PULP3 Box (as root, run):

<pre>

DJANGO_SETTINGS_MODULE='pulpcore.app.settings' PULP_SETTINGS='/etc/pulp/settings.py'  /usr/local/lib/pulp/bin/pulpcore-manager  reset-admin-password --password password

</pre>

8.  copy contents of  /etc/pulp/certs/root.crt on PULP box to the KATELLO box in file: /etc/pki/ca-trust/source/anchors/pulpcore.crt

9.  Update ca trust:

<pre>

update-ca-trust

</pre>

10. On the KATELLO box, reconfigure the smart proxy plugin, edit  /etc/foreman-proxy/settings.d/pulpcore.yml  replace contents with (Substituting correct hostname):

<pre>

---

:enabled: https

:pulp_url: https://pulp3-sandbox-centos7.balmora.example.com

:content_app_url: https://pulp3-sandbox-centos7.balmora.example.com/pulp/content

:mirror: false

:username: admin

:password: password

</pre>

11.  restart the foreman-proxy:

<pre>

systemctl restart foreman-proxy

</pre>

12.  Navigate to Infrastructure > Smart proxies and refresh the smart proxy features

h3. To pip install a pulp plugin on the new pulp server:

Make sure to use the pip executable found in the same bin directory that the pulp service executables are found. For example, '/usr/local/lib/pulp/bin/pip'.

To find this location run: 

<pre>

systemctl status "pulp*"

</pre> 

Then look under the CGroup of one of the pulp services.

h2. Example for querying the Pulp 3 API in a development environment:

<pre>

curl https://`hostname`/pulp/api/v3/status/   --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key

</pre>

Recommendation: try the pulp-cli: https://github.com/pulp/pulp-cli#katello

h3. Other Resources

* pulp3 gap analysis wiki https://projects.theforeman.org/projects/katello/wiki/PulpV3GapAnalysis

* pulp3 gap analysis etherpad https://etherpad.net/p/pulp3_container

* pulp3 docker roadmap https://pulp.plan.io/projects/pulp/wiki/Pulp_Docker_Roadmap

h1. OSTree Content (Centos8)

h2. Centos8 required! The foreman-installer option "--foreman-proxy-content-enable-ostree" must be used.

h2. Adding ostree content

h3. Testing the content after upload or sync

There are 2 primary scenarios for adding ostree content to Katello - uploading ostree archives or syncing a remote ostree repository.

Katello v4.3 is required for either of these scenarios.

Verifying Content Units

In both scenarios you may verify content units by querying via hammer-cli or through the UI.

<pre>

$ hammer content-units list --content-type ostree_refs

---|---------|------------------------------------------------------------------|---------

ID | NAME    | VERSION                                                          | FILENAME

---|---------|------------------------------------------------------------------|---------

14 | rawhide | b8a587c1a2aa3db2993101e2f8cd1eee48d130032a21d9493b89be1272eecdd0 |         

13 | stable  | 7dfa2caf98286074e6e79eb7aa2f0e18978c233a95a908118138b7b4f18752a7 |         

---|---------|------------------------------------------------------------------|---------

</pre>

Navigating to the repository info details URL, such as:

http://<katello host>/products/>product iod>/repositories/<repository id>/content/ostree_refs?page=1&per_page=20

The commit refs associated with the ostree repository will be shown:

h3. Syncing Remote OStree Repository

A user creates a new repository of type “ostree”. The “Upstream URL”  field may be left blank or it may be populated before sync actions with a remote repository URL.

Example for testing purposes: https://fixtures.pulpproject.org/ostree/small/

If using hammer-cli (with the hammer-cli-katello plugin enabled):

hammer repository upload-content --id <repostory_id> --content-type ostree_ref --path <path to fixtures_small_repo.tar file> --ostree-repository-name small

Not that the “--ostree-repository-name” is required for ostree content uploads.

<pre>

$ hammer repository upload-content --path ~/projects/2efad613-fb5e-4485-b67c-10f8ebed27e8-commit.tar  --id 1

Could not upload the content:

  OSTree commit ref uploads require a repository name.

</pre>

This repository name refers to the repository within the uploaded archive itself. Creating archive images is detailed below.

If using the UI, invoke the sync action on the repository. The UI presents a sync action progress indicator showing the sync action summary, updated as it completes.

h3. Creating An Image Builder Archive for Testing

A user may create a test image in a couple of ways.

From an existing pulp fixture URL:

<pre>

wget --no-parent -r https://fixtures.pulpproject.org/ostree/small/

tar --exclude="index.html" -cvf "fixtures_small_repo.tar" -C fixtures.pulpproject.org/ostree "small"\

</pre>

This produces the archive “fixtures_small_repo.tar”.

Alternately, a user may use the os-builder command line tools, as outlined in

https://www.osbuild.org/guides/user-guide/building-ostree-images.html.

The images produced through the osbuild command line tools are typically quite large compared to the pulp fixture archive mentioned above (nearly 1GB vs 30KB). The osbuild images are much closer to realistic images and should be used to test handling of large files.