Project

General

Profile

Actions

Bug #29689

closed

Katello Scenario with external database fails during migration

Added by Manuel Laug almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
-
Target version:
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

When defining an external database for Foreman, Candlepin and Pulpcore via custom-hiera.yaml the installer fails during its initial and every following run with errors regarding missing PostgresDB relations.

This is a compacted output of a second installer run (after its initial one). I've attached the full output of the installer with stripped hostnames.

[...]
[ INFO 2020-05-04T14:09:44 verbose]  Applying configuration version '1588601378'
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "http_proxies" does not exist
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LINE 8:                WHERE a.attrelid = '"http_proxies"'::regclass
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                                           ^
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: :               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                      pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                      c.collname, col_description(a.attrelid, a.attnum) AS comment
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                 FROM pg_attribute a
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                 LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                 LEFT JOIN pg_type t ON a.atttypid = t.oid
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                 LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                WHERE a.attrelid = '"http_proxies"'::regclass
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                  AND a.attnum > 0 AND NOT a.attisdropped
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                ORDER BY a.attnum
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `async_exec'
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `block (2 levels) in query'
[...]
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Caused by:
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::UndefinedTable: ERROR:  relation "http_proxies" does not exist
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LINE 8:                WHERE a.attrelid = '"http_proxies"'::regclass
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:                                           ^
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `async_exec'
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `block (2 levels) in query'
[...]
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Tasks: TOP => db:migrate => db:load_config => environment
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
[ WARN 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
[ERROR 2020-05-04T14:10:13 verbose]  '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[...]
[ERROR 2020-05-04T14:10:13 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
[...]

The content of custom-hiera.yaml is:

---
foreman::db_manage: false
foreman::db_host: <db host>
foreman::db_database: foreman
foreman::db_username: <db user>
foreman::db_password: <db password>
foreman::db_port: 5432

katello::candlepin_manage_db: false
katello::candlepin_db_host: <db host>
katello::candlepin_db_name: candlepin
katello::candlepin_db_user: <db user>
katello::candlepin_db_password: <db password>
katello::candlepin_db_port: 5432

pulpcore::postgresql_manage_db: false
pulpcore::postgresql_db_host: <db host>
pulpcore::postgresql_db_name: pulpcore
pulpcore::postgresql_db_user: <db user>
pulpcore::postgresql_db_password: <db password>
pulpcore::postgresql_db_port: 5432

Using the installer in its default configuration (with local PostgresDB deployment), everything seems to work flawlessly.

Versions:

[root@foreman foreman-installer]# rpm -qa | grep -i foreman-installer
foreman-installer-2.0.0-1.el7.noarch
foreman-installer-katello-2.0.0-1.el7.noarch

[root@foreman foreman-installer]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * centos-sclo-rh: mirror.infonline.de
 * epel: mirror.de.leaseweb.net
repo id                                                                         repo name                                                                                                                    status
base/7/x86_64                                                                   CentOS-7 - Base                                                                                                              10,070
centos-sclo-rh/x86_64                                                           CentOS-7 - SCLo rh                                                                                                            6,509
epel/x86_64                                                                     Extra Packages for Enterprise Linux 7 - x86_64                                                                               13,250
extras/7/x86_64                                                                 CentOS-7 - Extras                                                                                                               392
foreman/x86_64                                                                  Foreman 2.0                                                                                                                     665
foreman-plugins/x86_64                                                          Foreman plugins 2.0                                                                                                             315
katello/x86_64                                                                  Katello 3.15                                                                                                                     52
katello-candlepin/x86_64                                                        Candlepin: an open source entitlement management system.                                                                          2
katello-pulpcore/x86_64                                                         pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.                                                             66
openlogic/7/x86_64                                                              CentOS-7 - openlogic packages for x86_64                                                                                         22
pulp/7/x86_64                                                                   Pulp Community Release                                                                                                           67
puppet6/x86_64                                                                  Puppet 6 Repository el 7 - x86_64                                                                                               207
updates/7/x86_64                                                                CentOS-7 - Updates                                                                                                              240
repolist: 31,857


Files

foreman_installer_db_error foreman_installer_db_error 71.5 KB Manuel Laug, 05/04/2020 02:17 PM
Actions #1

Updated by Tomer Brisker almost 4 years ago

  • Project changed from Installer to Katello

The migration is failing on this line:
[ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.1/app/models/setting/content.rb:171:in `content_default_http_proxy'

moving to katello

Actions #2

Updated by James Jeffers almost 4 years ago

  • Target version set to Katello 3.15.1
  • Triaged changed from No to Yes

Investigate and determine if it should go into 3.15.1 or later.

Actions #3

Updated by Manuel Laug almost 4 years ago

Just a small note: This behaviour only happens when the latency to the database is relatively high (about ~80ms rtt).

Actions #4

Updated by James Jeffers almost 4 years ago

  • Target version changed from Katello 3.15.1 to Katello 3.15.2
Actions #5

Updated by James Jeffers almost 4 years ago

  • Target version changed from Katello 3.15.2 to Katello 3.15.3
Actions #6

Updated by James Jeffers almost 4 years ago

We've reproduced this issue locally using a 400ms latency into the network.

Actions #7

Updated by The Foreman Bot almost 4 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to James Jeffers
  • Pull request https://github.com/Katello/katello/pull/8792 added
Actions #8

Updated by The Foreman Bot almost 4 years ago

  • Fixed in Releases Katello 4.0.0 added
Actions #9

Updated by James Jeffers almost 4 years ago

  • Status changed from Ready For Testing to Closed
Actions

Also available in: Atom PDF