Bug #29689
closedKatello Scenario with external database fails during migration
Description
When defining an external database for Foreman, Candlepin and Pulpcore via custom-hiera.yaml the installer fails during its initial and every following run with errors regarding missing PostgresDB relations.
This is a compacted output of a second installer run (after its initial one). I've attached the full output of the installer with stripped hostnames.
[...] [ INFO 2020-05-04T14:09:44 verbose] Applying configuration version '1588601378' [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted! [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "http_proxies" does not exist [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LINE 8: WHERE a.attrelid = '"http_proxies"'::regclass [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: ^ [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: : SELECT a.attname, format_type(a.atttypid, a.atttypmod), [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: c.collname, col_description(a.attrelid, a.attnum) AS comment [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: FROM pg_attribute a [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LEFT JOIN pg_type t ON a.atttypid = t.oid [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: WHERE a.attrelid = '"http_proxies"'::regclass [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: AND a.attnum > 0 AND NOT a.attisdropped [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: ORDER BY a.attnum [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `async_exec' [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `block (2 levels) in query' [...] [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Caused by: [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::UndefinedTable: ERROR: relation "http_proxies" does not exist [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: LINE 8: WHERE a.attrelid = '"http_proxies"'::regclass [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: ^ [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `async_exec' [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `block (2 levels) in query' [...] [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Tasks: TOP => db:migrate => db:load_config => environment [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace) [ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls. [ERROR 2020-05-04T14:10:13 verbose] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0] [...] [ERROR 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0] [...]
The content of custom-hiera.yaml is:
---
foreman::db_manage: false
foreman::db_host: <db host>
foreman::db_database: foreman
foreman::db_username: <db user>
foreman::db_password: <db password>
foreman::db_port: 5432
katello::candlepin_manage_db: false
katello::candlepin_db_host: <db host>
katello::candlepin_db_name: candlepin
katello::candlepin_db_user: <db user>
katello::candlepin_db_password: <db password>
katello::candlepin_db_port: 5432
pulpcore::postgresql_manage_db: false
pulpcore::postgresql_db_host: <db host>
pulpcore::postgresql_db_name: pulpcore
pulpcore::postgresql_db_user: <db user>
pulpcore::postgresql_db_password: <db password>
pulpcore::postgresql_db_port: 5432
Using the installer in its default configuration (with local PostgresDB deployment), everything seems to work flawlessly.
Versions:
[root@foreman foreman-installer]# rpm -qa | grep -i foreman-installer
foreman-installer-2.0.0-1.el7.noarch
foreman-installer-katello-2.0.0-1.el7.noarch
[root@foreman foreman-installer]# yum repolist
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
* centos-sclo-rh: mirror.infonline.de
* epel: mirror.de.leaseweb.net
repo id repo name status
base/7/x86_64 CentOS-7 - Base 10,070
centos-sclo-rh/x86_64 CentOS-7 - SCLo rh 6,509
epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13,250
extras/7/x86_64 CentOS-7 - Extras 392
foreman/x86_64 Foreman 2.0 665
foreman-plugins/x86_64 Foreman plugins 2.0 315
katello/x86_64 Katello 3.15 52
katello-candlepin/x86_64 Candlepin: an open source entitlement management system. 2
katello-pulpcore/x86_64 pulpcore: Fetch, Upload, Organize, and Distribute Software Packages. 66
openlogic/7/x86_64 CentOS-7 - openlogic packages for x86_64 22
pulp/7/x86_64 Pulp Community Release 67
puppet6/x86_64 Puppet 6 Repository el 7 - x86_64 207
updates/7/x86_64 CentOS-7 - Updates 240
repolist: 31,857
Files
Updated by Tomer Brisker over 4 years ago
- Project changed from Installer to Katello
The migration is failing on this line:
[ WARN 2020-05-04T14:10:13 verbose] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.1/app/models/setting/content.rb:171:in `content_default_http_proxy'
moving to katello
Updated by James Jeffers over 4 years ago
- Target version set to Katello 3.15.1
- Triaged changed from No to Yes
Investigate and determine if it should go into 3.15.1 or later.
Updated by Manuel Laug over 4 years ago
Just a small note: This behaviour only happens when the latency to the database is relatively high (about ~80ms rtt).
Updated by James Jeffers over 4 years ago
- Target version changed from Katello 3.15.1 to Katello 3.15.2
Updated by James Jeffers over 4 years ago
- Target version changed from Katello 3.15.2 to Katello 3.15.3
Updated by James Jeffers over 4 years ago
We've reproduced this issue locally using a 400ms latency into the network.
Updated by The Foreman Bot over 4 years ago
- Status changed from New to Ready For Testing
- Assignee set to James Jeffers
- Pull request https://github.com/Katello/katello/pull/8792 added
Updated by The Foreman Bot over 4 years ago
- Fixed in Releases Katello 4.0.0 added
Updated by James Jeffers over 4 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset katello|41711a2d9397c42d5961eba03cd236679e208075.