Foreman » Installer

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=2062189

Description of problem: satellite-installer gets failed with "Could not open SSL root certificate file /root/.postgresql/root.crt"

Version-Release number of selected component (if applicable):
candlepin 4.1.11

How reproducible:
always

Steps to Reproduce:
1. Install the unconfigured(only packages) satellite
2. Generate the SSL certificate.
3. Install the external Postgresql database and update all the required configurations for SSL.
4. Run the installer on the satellite to integrate the external database but the command gets failed with following error

satellite-installer --scenario satellite --foreman-initial-admin-password --foreman-db-manage false --foreman-db-host postgres.com --foreman-db-database foreman1db --foreman-db-username foreman1! --foreman-db-password --foreman-db-port 5432 --katello-candlepin-manage-db false --katello-candlepin-db-host postgres.com --katello-candlepin-db-name candlepin1db --katello-candlepin-db-user candlepin1! --katello-candlepin-db-password --katello-candlepin-db-port 5432 --foreman-proxy-content-pulpcore-manage-postgresql false --foreman-proxy-content-pulpcore-postgresql-host postgres.com --foreman-proxy-content-pulpcore-postgresql-db-name pulpcore1db --foreman-proxy-content-pulpcore-postgresql-user pulp1! --foreman-proxy-content-pulpcore-postgresql-password --foreman-proxy-content-pulpcore-postgresql-port 5432 --foreman-db-sslmode verify-full --foreman-db-root-cert /usr/share/foreman/root.crt --katello-candlepin-db-ssl true --katello-candlepin-db-ssl-verify true --foreman-proxy-content-pulpcore-postgresql-ssl true

2022-03-09 04:26:29 [NOTICE] [configure] 500 configuration steps out of 1443 steps complete.
2022-03-09 04:26:34 [ERROR ] [configure] Execution of '/usr/share/candlepin/cpdb --create --schema-only --dbhost=postgres.com --dbport=5432 --database=candlepin1db?ssl=true --user=candlepin1! --password=****' returned 1: ########## ERROR ############
2022-03-09 04:26:34 [ERROR ] [configure] Error running command: /usr/share/candlepin/liquibase.sh --driver=org.postgresql.Driver --classpath=/var/lib/tomcat/webapps/candlepin/WEB-INF/lib/postgresql-42.3.2.jar:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ --changeLogFile=db/changelog/changelog-create.xml --url="jdbc:postgresql://postgres.com:5432/candlepin1db?ssl=true" --username=$DBUSERNAME --password=$DBPASSWORD --logLevel=severe migrate -Dcommunity=False
2022-03-09 04:26:34 [ERROR ] [configure] Status code: 65280
2022-03-09 04:26:34 [ERROR ] [configure] Command output: Liquibase update Failed: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.
2022-03-09 04:26:34 [ERROR ] [configure] SEVERE 3/9/22, 4:26 AM:liquibase: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: Could not open SSL root certificate file /root/.postgresql/root.crt.

Actual results:
Installer gets failed with "Could not open SSL root certificate file /root/.postgresql/root.crt" file error.

Expected results:
Installer should be completed successfully.

Additional info: Didn't see this problem with candlepin 4.1.1

This leads me to this commit: https://github.com/candlepin/candlepin/commit/5123d749532b4fd1c1a73e7039778a5529fa787a

And thus to the upstream changes: https://github.com/pgjdbc/pgjdbc/compare/REL42.2.2...REL42.3.2

Quick grep for "cert" leads to https://github.com/pgjdbc/pgjdbc/commit/cdeeaca47dc3bc6f727c79a582c9e4123099526e and especially https://github.com/pgjdbc/pgjdbc/blob/cdeeaca47dc3bc6f727c79a582c9e4123099526e/pgjdbc/src/main/java/org/postgresql/ssl/LibPQFactory.java#L57-L73 which is building exactly the path that is not found here.

So, TL;DR: the postgresql module used by Candlepin changed behaviour (to be more secure) and something in the chain (installer → cpdb → …) doesn't pass the cert paths correctly, thus triggering the fallback which doesn't exist.

The exception we're seeing is this one, btw: https://github.com/pgjdbc/pgjdbc/blob/REL42.3.2/pgjdbc/src/main/java/org/postgresql/ssl/LibPQFactory.java#L150