Project

General

Profile

Actions
Copy link

Bug #8968

closed

Race condition during facts importing

Added by Romain Vrignaud almost 10 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Marek Hulán
Category:
foreman_chef
Target version:
foreman_chef 0.1.3
Difficulty:
Triaged:
Bugzilla link:
Pull request:
https://github.com/theforeman/foreman_chef/pull/14
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Hello,
On a ~100 hosts deployment with foreman_chef and foreman-tasks, we see about 30% error rate of fact uploading.
We see lots of "ForemanTasks::Lock::LockConflict" caused by "ActiveRecord::RecordInvalid: Validation failed: Name has already been taken".

The backtrace is:

/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/validations.rb:56:in `save!'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/transactions.rb:264:in `block in save!'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/transactions.rb:208:in `transaction'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/transactions.rb:264:in `save!'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/activerecord-3.2.18/lib/active_record/validations.rb:41:in `create!'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:29:in `block in add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `each'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:36:in `block in add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `each'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:36:in `block in add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `each'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:22:in `add_missing_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman_chef-0.0.4/app/models/foreman_chef/fact_importer.rb:17:in `add_new_facts'
/opt/foreman/app/services/fact_importer.rb:26:in `import!'
/opt/foreman/app/models/host/base.rb:57:in `import_facts'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman-tasks-0.6.10/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/opt/foreman/app/models/concerns/foreman/thread_session.rb:73:in `as'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/foreman-tasks-0.6.10/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:456:in `block (3 levels) in execute_run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/middleware/stack.rb:26:in `call'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/middleware.rb:16:in `pass'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action/progress.rb:16:in `run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:455:in `block (2 levels) in execute_run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:454:in `catch'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:454:in `block in execute_run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:371:in `call'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:371:in `block in with_error_handling'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:371:in `catch'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:371:in `with_error_handling'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:449:in `execute_run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/action.rb:236:in `execute'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract.rb:152:in `call'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract.rb:152:in `with_meta_calculation'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/executors/parallel/worker.rb:17:in `on_message'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:82:in `on_envelope'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:72:in `receive'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:99:in `loop'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:99:in `block in run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:99:in `catch'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:99:in `run'
/opt/foreman/vendor/bundle/ruby/2.1.0/gems/dynflow-0.7.5/lib/dynflow/micro_actor.rb:13:in `block in initialize'

Actions
Copy link
 #1

Updated by Romain Vrignaud almost 10 years ago

  • Category set to foreman_chef
Actions
Copy link
 #2

Updated by Marek Hulán almost 10 years ago

  • Subject changed from Lots of error with fact uploading to Race condition during facts importing

Caused by race condition in facts importing, locks are just consequence. We should fix ForemanChef::FactImporter so it does not fail at https://github.com/theforeman/foreman_chef/blob/v0.1.0/app/models/foreman_chef/fact_importer.rb#L33

if another task create fact_name meanwhile, we should ignore the error, reload it and associate it with new fact_value

Actions
Copy link
 #3

Updated by Marek Hulán almost 10 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Marek Hulán
  • Pull request https://github.com/theforeman/foreman_chef/pull/14 added
  • Pull request deleted ()
Actions
Copy link
 #4

Updated by Marek Hulán almost 10 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
  • Translation missing: en.field_release set to 37
Actions
Copy link

Also available in: Atom PDF