Foreman Maintain

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1847189

Description of problem:
foreman-maintain restore fails on large postgresql databases.

Version-Release number of selected component (if applicable):
Satellite 6.7.0

How reproducible:

Consistently

Steps to Reproduce:
1.Attempt to restore a Satellite with a postgresql database so large, it requires more than 30 minutes to restore.
2.
3.

Actual results:
Restore fails with postgresql errors, and when you go to the satellite Web UI, you receive a Web Appliation error.

Expected results:
Restore is successful and Satellite Web UI functions.

Additional info:
This is due to 2 foreman-rake commands running via cron on the satellite every 30 minutes. If they run before the postgresql restores certain tables in the foreman database, the commands reinitialze the tables with default values. Then when the restore process restores those tables, it creates duplicate entries with duplicate id field values. This prevents the primary key constraint from applying correctly, resulting in postgresql errors. As the tables do not have a primary key, the Web Application fails.

The problematic sql errors are:
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 5163; 2606 21956 CONSTRAINT remote_execution_features_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "remote_execution_features_pkey"
DETAIL: Key (id)=(7) is duplicated.
Command was: ALTER TABLE ONLY remote_execution_features
ADD CONSTRAINT remote_execution_features_pkey PRIMARY KEY (id);

pg_restore: [archiver (db)] Error from TOC entry 5181; 2606 21976 CONSTRAINT settings_pkey foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "settings_pkey"
DETAIL: Key (id)=(58) is duplicated.
Command was: ALTER TABLE ONLY settings
ADD CONSTRAINT settings_pkey PRIMARY KEY (id);

pg_restore: [archiver (db)] Error from TOC entry 5179; 1259 22325 INDEX index_settings_on_name foreman
pg_restore: [archiver (db)] could not execute query: ERROR: could not create unique index "index_settings_on_name"
DETAIL: Key (name)=(katello_default_provision) is duplicated.
Command was: CREATE UNIQUE INDEX index_settings_on_name ON settings USING btree (name);

pg_restore: [archiver (db)] Error from TOC entry 5438; 2606 22652 FK CONSTRAINT fk_rails_8642ef3b5e foreman
pg_restore: [archiver (db)] could not execute query: ERROR: there is no unique constraint matching given keys for referenced table "remote_execution_features"
Command was: ALTER TABLE ONLY job_invocations
ADD CONSTRAINT fk_rails_8642ef3b5e FOREIGN KEY (remote_execution_feature_id) REFERENCES...
pg_restore: [archiver (db)] Error from TOC entry 6205; 0 0 ACL ansible_roles foreman

Successful Workaround: have foreman-maintain stop the crond.service on Satelltue while the restore is running.