Is this a clean install on 1.15-RC1 or an upgrade from 1.14?

When do you see this error / where do you see it? Is it after restarting httpd (systemctl restart httpd), in /var/log/foreman/production.log?

Also would it be possible to list plugins you have installed and their versions?

We have foreman installed via package and upgraded from 1.14 to 1.15 so no clean install of 1.15.

We get this error when restarting httpd and we cannot access foreman Web Client.

I think this is an Issue with the DB if we create a new DB and restart Foreman everything works somehow, i'm currently working on searching the query that lets the application crash..

Last Query before foreman "crashes" :

 
SELECT `taxonomies`.id FROM `taxonomies` INNER JOIN `taxable_taxonomies` 
ON `taxonomies`.`id` = `taxable_taxonomies`.`taxonomy_id` 
WHERE `taxable_taxonomies`.`taxable_id` = 32 AND `taxable_taxonomies`.`taxable_type` = 'Filter' AND `taxonomies`.`type` = 'Location'  
ORDER BY `taxonomies`.`title` ASC;

ROLLBACK;

Could you maybe help us out with some more detail on the database architecture ?
Like why is the Table 'foreman.taxonomies' and 'foreman.taxable_taxonomies' empty ? As those ar the last SELECT's before foreman crashes..

These two tables are only used if you enable organizations and/or locations in settings.yaml. So they are empty because you probably didn't allow it.

So we were able to fix this issue for now by downgrading again to 1.14 after we checked that the db seems to work again on 1.14 we upgraded again to 1.15 and it seems to work at the moment. If you want i can provide some more details..

Thanks
Sandro

I've got the same problem while upgrading from 1.14.1 to 1.15. Running foreman-installer --scenario katello --upgrade produces the same backtrace.

I've got the same problem while upgrading from 1.14.1 to 1.15. Running foreman-installer --scenario katello --upgrade produces similar backtrace:

Upgrade Step: db_seed...
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Filters.role is locked for user modifications.
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:79:in `raise_record_invalid'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:43:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `block in save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:220:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `save!'
/usr/share/foreman/app/models/role.rb:154:in `add_permissions!'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_salt-8.0.2/db/seeds.d/75-salt_seeds.rb:9:in `block in <top (required)>'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_salt-8.0.2/db/seeds.d/75-salt_seeds.rb:8:in `<top (required)>'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `load'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `block in load'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `load'
/usr/share/foreman/db/seeds.rb:38:in `block in <top (required)>'
/usr/share/foreman/db/seeds.rb:36:in `each'
/usr/share/foreman/db/seeds.rb:36:in `<top (required)>'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `load'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `block in load'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/dependencies.rb:268:in `load'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/engine.rb:547:in `load_seed'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/tasks/database_tasks.rb:250:in `load_seed'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/railties/databases.rake:183:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:seed
(See full trace by running task with --trace)
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Seeding /usr/share/foreman/db/seeds.d/02-permissions_list.rb
Seeding /usr/share/foreman/db/seeds.d/02-roles_list.rb
Seeding /usr/share/foreman/db/seeds.d/03-auth_sources.rb
Seeding /usr/share/foreman/db/seeds.d/03-permissions.rb
Seeding /usr/share/foreman/db/seeds.d/03-roles.rb
Seeding /usr/share/foreman/db/seeds.d/04-admin.rb
Seeding /usr/share/foreman/db/seeds.d/05-taxonomies.rb
Seeding /usr/share/foreman/db/seeds.d/06-architectures.rb
Seeding /usr/share/foreman/db/seeds.d/07-data.rb
Seeding /usr/share/foreman/db/seeds.d/07-provisioning_templates.rb
Seeding /usr/share/foreman/db/seeds.d/08-data.rb
Seeding /usr/share/foreman/db/seeds.d/08-partition_tables.rb
Seeding /usr/share/foreman/db/seeds.d/10-installation_media.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/101-locations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/102-organizations.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/103-provisioning_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/104-proxy.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/106-mail_notifications.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/107-enable_dynflow.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/108-subcription-bookmarks.rb
Seeding /usr/share/foreman/db/seeds.d/11-smart_proxy_features.rb
Seeding /usr/share/foreman/db/seeds.d/13-compute_profiles.rb
Seeding /usr/share/foreman/db/seeds.d/15-bookmarks.rb
Seeding /usr/share/foreman/db/seeds.d/16-mail_notifications.rb
Seeding /usr/share/foreman/db/seeds.d/17-notification_blueprints.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/db/seeds.d/20-foreman_tasks_permissions.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/db/seeds.d/60-dynflow_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.1/db/seeds.d/60-ssh_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.9.1/db/seeds.d/61-foreman_tasks_bookmarks.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_ansible-1.4.5/db/seeds.d/62_ansible_proxy_feature.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure-1.3.1/db/seeds.d/70-cloudapp-domain.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.3.1/db/seeds.d/70-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/db/seeds.d/75-job_templates.rb
Seeding /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_salt-8.0.2/db/seeds.d/75-salt_seeds.rb
foreman-rake db:seed failed! Check the output for error!
Upgrade step db_seed failed. Check logs for more information.

The problem Rob Sanders reports seems to be caused by foreman_salt not using the plugin DSL for seeding roles but rather doing it its own way - https://github.com/theforeman/foreman_salt/blob/master/db/seeds.d/75-salt_seeds.rb

This fix for #19039 - PR https://github.com/theforeman/foreman/pull/4415 contains the fix. If we need to fix it for 1.15.z and #19039 would not be acceptable for stable branch, here's the patch that should help.

diff --git a/app/registries/foreman/plugin.rb b/app/registries/foreman/plugin.rb
index dd0e343..4e2ffc3 100644
--- a/app/registries/foreman/plugin.rb
+++ b/app/registries/foreman/plugin.rb
@@ -245,7 +245,9 @@ module Foreman #:nodoc:
       return false if pending_migrations || Rails.env.test?
       Role.transaction do
         role = Role.where(:name => name).first_or_create
-        role.add_permissions!(permissions) if role.permissions.empty?
+        role.ignore_locking do
+          role.add_permissions!(permissions) if role.permissions.empty?
+        end
         rbac_registry.role_ids << role.id
       end
     rescue PermissionMissingException => e

Ido, the patch mentioned here only works for the trace that Sandro encountered. Your (and Robs) trace is caused by the foreman_salt plugin that tries to seed permissions in non-standard way https://github.com/theforeman/foreman_salt/blob/master/db/seeds.d/75-salt_seeds.rb#L9

It should use this Foreman DSL http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin#Adding-roles

EDIT: I suggest you open issue at http://projects.theforeman.org/projects/salt

OK, Thanks, I have opened a ticket.

Can you please help me to understand why this error occurs since the upgrade and what is the impact of this issue?