Foreman Maintain

During restore tests with Satellite 6.12 and Satellite 6.13 I noticed that foreman-maintain backup snapshot creates broken pgsql_data.tar.gz archives that cannot be restored with foreman-maintain restore. This situation can also be reproduced with the latest nightly version of Foreman and Katello.

Basically there are two problems when creating a snapshot backup with foreman-maintain backup snapshot:

1.) An archive with broken paths (missing / after data) will be created:

[root@foreman ~]# tar tfv /var/backup/katello-backup-2023-06-06-13-17-36/pgsql_data.tar.gz | head -n 20
-rw------- postgres/postgres 3 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/dataPG_VERSION
drwx------ postgres/postgres 0 2023-06-03 22:19 var/snap/pgsql/var/lib/pgsql/database/
drwx------ postgres/postgres 0 2023-06-06 13:10 var/snap/pgsql/var/lib/pgsql/database/1/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/database/13448/
drwx------ postgres/postgres 0 2023-06-06 13:10 var/snap/pgsql/var/lib/pgsql/database/13449/
drwx------ postgres/postgres 0 2023-06-06 13:10 var/snap/pgsql/var/lib/pgsql/database/159342/
drwx------ postgres/postgres 0 2023-06-06 13:10 var/snap/pgsql/var/lib/pgsql/database/161204/
drwx------ postgres/postgres 0 2023-06-06 13:09 var/snap/pgsql/var/lib/pgsql/database/166949/
drwx------ postgres/postgres 0 2023-04-13 10:54 var/snap/pgsql/var/lib/pgsql/database/pgsql_tmp/
-rw------- postgres/postgres 30 2023-06-06 13:09 var/snap/pgsql/var/lib/pgsql/datacurrent_logfiles
drwx------ postgres/postgres 0 2023-06-06 13:10 var/snap/pgsql/var/lib/pgsql/dataglobal/
drwx------ postgres/postgres 0 2023-06-06 13:07 var/snap/pgsql/var/lib/pgsql/datalog/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/datapg_commit_ts/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/datapg_dynshmem/
-rw-r----- postgres/postgres 697 2023-06-02 20:44 var/snap/pgsql/var/lib/pgsql/datapg_hba.conf
-rw-r----- postgres/postgres 47 2023-04-05 21:05 var/snap/pgsql/var/lib/pgsql/datapg_ident.conf
drwx------ postgres/postgres 0 2023-06-06 13:19 var/snap/pgsql/var/lib/pgsql/datapg_logical/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/datapg_logical/mappings/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/datapg_logical/snapshots/
drwx------ postgres/postgres 0 2023-04-05 21:04 var/snap/pgsql/var/lib/pgsql/datapg_multixact/

2. Even if you fix 1.), the contents of the archive will be unpacked to the wrong place during the restore, which results in the database remaining completely empty after the restore - as if Foreman/Satellite had been freshly installed.

# create backup
foreman-maintain backup snapshot /var/backup

# Fix first bug (missing '/')
cd /var/backup/katello-backup-2023-06-06-13-17-36/
tar xfz pgsql_data.tar.gz 
cd var/snap/pgsql/var/lib/pgsql
for file in $(ls); do new=$(echo $file | sed s/data//g); mv $file $new; done
mkdir -p data && chown postgres: data && chmod 700 data
mv * data
cd /var/backup/katello-backup-2023-06-06-13-17-36/
tar cfz pgsql_data-fixed.tar.gz var/snap/pgsql/var/lib/pgsql/data/
rm -f pgsql_data.tar.gz 
mv pgsql_data-fixed.tar.gz pgsql_data.tar.gz

# size of /var/lib/pgsql/data before restore
[root@foreman ~]# du -hs /var/lib/pgsql/data
3.9G /var/lib/pgsql/data

# restore 
foreman-maintain restore /var/backup/katello-backup-2023-06-06-14-06-10/

# size of /var/lib/pgsql/data after restore
[root@foreman ~]# du -hs /var/lib/pgsql/data
598M /var/lib/pgsql/data

# satellite/foreman is now completly empty

# actual data is in the wrong directory: 
du -hs /var/snap/pgsql/var/lib/pgsql/data/
3.9G /var/snap/pgsql/var/lib/pgsql/data/

The cause is, if I understand it correctly, that there is no distinction between source and destination in the backup_local method
For an offline backup, this works because the backup is created from the real postgres data directory (feature(:foreman_database).data_dir) and is unpacked there on restore.
But for a snapshot backup the source directory is the snapshot directory (@mount_dir) and the destination directory should be the postgres data directory "feature(:foreman_database).data_dir). But on restore it will be unpacked to @mount_dir instead.

A possible solution would be to have the backup_local method accept an additional restore_dir option, which must be set to the postgres data directory during a backup:

diff --git a/definitions/procedures/backup/offline/candlepin_db.rb b/definitions/procedures/backup/offline/candlepin_db.rb
index f00a641..5d2c943 100644
--- a/definitions/procedures/backup/offline/candlepin_db.rb
+++ b/definitions/procedures/backup/offline/candlepin_db.rb
@@ -36,7 +36,8 @@ module Procedures::Backup
             pg_backup_file,
             :listed_incremental => File.join(@backup_dir, '.postgres.snar'),
             :volume_size => @tar_volume_size,
-            :data_dir => pg_data_dir
+            :data_dir => pg_data_dir,
+            :restore_dir => feature(:candlepin_database).data_dir
           )
         end
       end
diff --git a/definitions/procedures/backup/offline/foreman_db.rb b/definitions/procedures/backup/offline/foreman_db.rb
index cd73c64..4285bb2 100644
--- a/definitions/procedures/backup/offline/foreman_db.rb
+++ b/definitions/procedures/backup/offline/foreman_db.rb
@@ -44,6 +44,7 @@ module Procedures::Backup
           :listed_incremental => File.join(@backup_dir, '.postgres.snar'),
           :volume_size => @tar_volume_size,
           :data_dir => pg_dir,
+          :restore_dir => feature(:foreman_database).data_dir,
           :command => cmd
         )
       end
diff --git a/definitions/procedures/backup/offline/pulpcore_db.rb b/definitions/procedures/backup/offline/pulpcore_db.rb
index 5aba2ef..94a79a7 100644
--- a/definitions/procedures/backup/offline/pulpcore_db.rb
+++ b/definitions/procedures/backup/offline/pulpcore_db.rb
@@ -36,7 +36,8 @@ module Procedures::Backup
             pg_backup_file,
             :listed_incremental => File.join(@backup_dir, '.postgres.snar'),
             :volume_size => @tar_volume_size,
-            :data_dir => pg_data_dir
+            :data_dir => pg_data_dir,
+            :restore_dir => feature(:pulpcore_database).data_dir
           )
         end
       end
diff --git a/lib/foreman_maintain/concerns/base_database.rb b/lib/foreman_maintain/concerns/base_database.rb
index 1fea0d5..33c4446 100644
--- a/lib/foreman_maintain/concerns/base_database.rb
+++ b/lib/foreman_maintain/concerns/base_database.rb
@@ -101,13 +101,14 @@ module ForemanMaintain

       def backup_local(backup_file, extra_tar_options = {})
         dir = extra_tar_options.fetch(:data_dir, data_dir)
+        restore_dir = extra_tar_options.fetch(:restore_dir, data_dir)
         command = extra_tar_options.fetch(:command, 'create')

         FileUtils.cd(dir) do
           tar_options = {
             :archive => backup_file,
             :command => command,
-            :transform => "s,^,#{dir[1..]},S",
+            :transform => "s,^,#{restore_dir[1..]},S",
             :files => '*',
           }.merge(extra_tar_options)
           feature(:tar).run(tar_options)