Bug #21158
closed
foreman-rake fix_db_cache fails with error ActiveRecord::RecordInvalid: Validation failed: Permissions Permissions must be of same resource type
Description
With katello 3.3 (Foreman 1.14), with the last packages, after checking the fix_db_cache option in the UI and restarting, Foreman does not start anymore.
When accessing to the UI we can see the following error:
ActiveRecord::RecordInvalid: Validation failed: Permissions Permissions must be of same resource type
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:79:in `raise_record_invalid'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:43:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `block in save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:220:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
/usr/share/foreman/app/services/cache_manager.rb:14:in `block in create_new_filter_cache'
/usr/share/foreman/app/models/role.rb:83:in `ignore_locking'
/usr/share/foreman/app/services/cache_manager.rb:13:in `create_new_filter_cache'
/usr/share/foreman/app/services/cache_manager.rb:22:in `recache!'
/usr/share/foreman/lib/tasks/fix_cache.rake:4:in `block in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
If we try to execute manually the task with "foreman-rake fix_db_cache" we get the same result.
The only way I have found to workaround the problem is to comment the file /usr/share/foreman/app/services/cache_manager.rb, line 20
#create_new_filter_cache
Although is similar to the problem described here: http://projects.theforeman.org/issues/19266 the solution provided does not work, as is built around fixing the role permission, while in my error it seems that in the database there is some broken record that fails to be cached when executing create_new_filter_cache. Could you provide information about where I should look in the database to find the inconsistency?
Regards.
Files
Updated by Marek Hulán about 7 years ago
- Status changed from New to Need more information
Is it still the issue? There were several improvements in recent version regarding this. It could also indicate DB inconsistency from previous versions.
Updated by Javier Vela about 7 years ago
Hi, The issue still persists but is because we are still in the release katello 3.3 (Foreman 1.14), that has not received any update since I opened this bug report. We have not upgraded katello to the last major version as we are do not want to start a migration until we asses if we have inconsistencies our database.
If you could share some insight about where to look in the database to find the inconsistencies raising the error:
ActiveRecord::RecordInvalid: Validation failed: Permissions Permissions must be of same resource type
We could then plan a migration to the last version.
regards.
Updated by Marek Hulán about 7 years ago
Just check records in permissions table, group the by filter (join through filtering table). Each permission for such group should have the sam7e resource type. This looks like #20190 that has patch attached. Mind to test if it fixes your env? Don't forget to backup your db first. After applying the patch, you need to run foreman-rake db:migrate
Updated by Javier Vela about 7 years ago
- Status changed from Need more information to Resolved
Hi,
I have tested the script in staging and production, and in both environments has worked without problems. It has fixed the inconsistencies and now the command foreman-rake fix_db_cache works perfect.
Many thanks for the support, I am marking the bug reported as resolved.
regards.
Updated by Marek Hulán about 7 years ago
Great, thanks for letting us know. Would you mind also adding a note to https://github.com/theforeman/foreman/pull/4638 so that the reviewer know it works and helps?
Updated by Javier Vela about 7 years ago
Hi,
Apologies for letting you know that it was solved, but the command foreman-rake fix_db_cache only worked because I forgot to uncomment the #create_new_filter_cache line. I have uncommented it and now it fails again:
foreman-rake fix_db_cache
Successfully encrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully decrypted field for Setting::Auth oauth_consumer_key
Successfully encrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Successfully decrypted field for Setting::Auth oauth_consumer_secret
Recreating cache
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Permissions Permissions must be of same resource type
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:79:in `raise_record_invalid'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/validations.rb:43:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/attribute_methods/dirty.rb:29:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `block in save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:220:in `transaction'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:291:in `save!'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
/opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `each'
/usr/share/foreman/app/services/cache_manager.rb:13:in `create_new_filter_cache'
/usr/share/foreman/app/services/cache_manager.rb:20:in `recache!'
/usr/share/foreman/lib/tasks/fix_cache.rake:4:in `block in <top (required)>'
Tasks: TOP => fix_db_cache
(See full trace by running task with --trace)
I am going to investigate to see if I found any inconsistency in the permissions table.
Regards.
Updated by Javier Vela about 7 years ago
I executed by hand the initial query using foreman-rake console, and it returns 0 rows:
irb(main):007:0> Filter.unscoped.joins(:permissions).group('filters.id', :resource_type).having('min(resource_type) <> max(resource_type)').pluck(:id)
=> []
irb(main):008:0>
So, it is not fixing anything as it is not able to detect any problem. One doubt I have is, the query seems to be relating the table permissions and filters, but in my DB I have a intermediate table, filterings, that is needed to relate filters and permissions.
I have executed this query on postgresql to try to get all the necessary information:
select permissions.name, permissions.resource_type, filters.role_id from permissions inner join filterings on filterings.permission_id = permissions.id inner join filters on filters.id = filterings.filter_id;
Attached in the dump (350 rows)