Project

General

Profile

Bug #9101

After dropping Foreman database, cannot re-run installer successfully

Added by Eric Helms almost 6 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Foreman modules
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Scenario:

1. Install Foreman 1.7
2. Decide you want to re-install or reset data
3. run 'foreman-rake db:drop'
4. run 'foreman-installer'
5. Get error

This also manifests itself within the katello-installer and is blocking our ability to reset all data. The user ends up in an entirely broken state if they try to use our --reset flag that includes the above scenario. From my investigations the error appears to revolve around the fact that after dropping the database, when the installer hits this line:

https://github.com/theforeman/puppet-foreman/blob/master/manifests/database.pp#L24

The following error is thrown by running the command:

foreman_bootdisk: skipping engine hook (PGError: ERROR: relation "template_kinds" does not exist
LINE 4: WHERE a.attrelid = '"template_kinds"'::regclass
^
: SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '"template_kinds"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
)

The puppet function ends up returning a nil value which Puppet interprets as not needing to propogate further and the migrate and seed are never run. After an unsuccessful run of the above scenario, you will get the same error if you try to run 'foreman-rake console'.

I had a few different ideas on how to solve this but ultimately was unsure if this is an issue with something bootdisk is doing or how the puppet function is interpreting the return when running this command.

Associated revisions

Revision a32b6639 (diff)
Added by Dominic Cleal almost 6 years ago

fixes #9101 - refresh db:migrate if DB class changes or settings are missing

When the DB is missing, the foreman::database::${type} class will refresh as
it recreates it, and now triggers the migration. On a clean install, this was
working mostly by luck as foreman::config refreshed everything.

foreman_config_entry didn't cause a refresh due to ignore_missing, which meant
it would only help when Foreman settings had been initialised, but before the
migration had completed. With an empty DB, it wouldn't work.

Removing ignore_missing means foreman_config_entry will also continue to
refresh the db:migrate until the setting is populated, and the migration
completes.

Only compatible with Foreman 1.7+ as it'll cause continuous refreshes on older
versions without the setting.

History

#1 Updated by Ohad Levy almost 6 years ago

I assume this is because db:migrate / seed only happens during package post?

#2 Updated by Dominic Cleal almost 6 years ago

  • Project changed from Foreman to Installer
  • Status changed from New to Need more information
  • Priority changed from Urgent to Normal

Eric Helms wrote:

The following error is thrown by running the command:

foreman_bootdisk: skipping engine hook (PGError: ERROR: relation "template_kinds" does not exist
LINE 4: WHERE a.attrelid = '"template_kinds"'::regclass
^
: SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '"template_kinds"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
)

This isn't an error, it's only logging from foreman_bootdisk to say that the tables aren't yet present.

What matters from the foreman-config command is only the exit code. Could you check $? please?

The puppet function ends up returning a nil value ...

I'm not sure I fully understand. foreman_config_entry is a provider, not a function: https://github.com/theforeman/puppet-foreman/blob/master/lib/puppet/provider/foreman_config_entry/cli.rb so it depends on whether this thinks it's "in sync" or not.

If you could provide a full debug log from the installer, that would be appreciated, plus the exit code of the foreman-config command.

#3 Updated by Dominic Cleal almost 6 years ago

Ohad Levy wrote:

I assume this is because db:migrate / seed only happens during package post?

That's only the case in a package-only installation. When using the installer, DB configuration happens after package installation, so the installer runs migrations/seeding itself.

#4 Updated by Eric Helms almost 6 years ago

Here is the output of running foreman-config manually and a full foreman-debug was uploaded (foreman-debug-bhzqA.tar.xz):

[root@katello-foreman share]# su - foreman -s /bin/bash -c "/usr/share/foreman/script/foreman-config -k 'db_pending_migration'" 
foreman-config script is deprecated. Please consider using `foreman-rake config` instead
foreman_bootdisk: skipping engine hook (PGError: ERROR:  relation "template_kinds" does not exist
LINE 4:              WHERE a.attrelid = '"template_kinds"'::regclass
                                        ^
:             SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
              FROM pg_attribute a LEFT JOIN pg_attrdef d
                ON a.attrelid = d.adrelid AND a.attnum = d.adnum
             WHERE a.attrelid = '"template_kinds"'::regclass
               AND a.attnum > 0 AND NOT a.attisdropped
             ORDER BY a.attnum
)
rake aborted!
PGError: ERROR:  relation "settings" does not exist
LINE 4:              WHERE a.attrelid = '"settings"'::regclass
                                        ^
:             SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
              FROM pg_attribute a LEFT JOIN pg_attrdef d
                ON a.attrelid = d.adrelid AND a.attnum = d.adnum
             WHERE a.attrelid = '"settings"'::regclass
               AND a.attnum > 0 AND NOT a.attisdropped
             ORDER BY a.attnum

Tasks: TOP => config
(See full trace by running task with --trace)
[root@katello-foreman share]# echo $?
1

#5 Updated by Dominic Cleal almost 6 years ago

  • Category set to Foreman modules
  • Status changed from Need more information to New

Okay, so it's the ignore_missing that's doing it. foreman::db::postgresql correctly refreshes as it recreates the database, but it's not propagating from that class through the chain to the foreman::rake resource as the foreman_config_entry is evaluated with no changes.

https://github.com/theforeman/puppet-foreman/blob/master/manifests/database.pp#L23-L29

I guess since #4611 is available in a stable branch, we can now drop ignore_missing from puppet-foreman, which only now needs to support 1.7 and develop (at a minimum, anyway).

And/or we could add extra notifies between foreman::db::postgresql (etc) and the foreman-rake db:migrate resource, so if the DB changes or is written, we can be sure of a notification to migrate and seed it (without the notify passing through foreman_config_entry for pending migration), which might skip it).

#6 Updated by Dominic Cleal almost 6 years ago

Something like this is what I thought for my last comment:

    class { $db_class: } ->
    foreman_config_entry { 'db_pending_migration':
      value          => false,
      dry            => true,
      ignore_missing => true,
    } ~>
    foreman::rake { 'db:migrate':
      subscribe => Class[$db_class],      ## added this line
    } ->
    foreman_config_entry { 'db_pending_seed':
      value          => false,
      dry            => true,
      ignore_missing => true,
      # to address #7353: settings initialization race condition
      before         => $foreman_service,
    } ~>
    foreman::rake { 'db:seed':
# .....

That retains 1.6 compatibility - although 'master' doesn't require it, then any change to the "db_class" (foreman::db::postgresql) will always trigger a migration, just in case.

#7 Updated by Dominic Cleal almost 6 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Dominic Cleal
  • Pull request https://github.com/theforeman/puppet-foreman/pull/277 added
  • Pull request deleted ()

#8 Updated by Dominic Cleal almost 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#9 Updated by Dominic Cleal almost 6 years ago

  • Legacy Backlogs Release (now unused) set to 28

Also available in: Atom PDF