Project

General

Profile

Actions
Copy link

Bug #11232

closed

Occassional error in tasks when importing facts from foreman-chef

Added by Alex Balk over 9 years ago. Updated over 9 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Tomer Brisker
Category:
-
Target version:
-
Difficulty:
Triaged:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Foreman 1.9RC2 on Ubuntu 12.04.5, foreman-chef 0.1.5-1 and foreman-tasks 0.7.1-1, running 5 executors.

Sometimes fact uploads pause in state "error" with the message below. When clicking "resume" they complete successfully.

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

This is the error from the "Errors" tab:


{"host"=>{"id"=>3003, "name"=>"foo.bar.com"},
 "facts"=>
  {"dmi::dmidecode_version"=>"2.11",
   "dmi::smbios_version"=>"2.7",

[...snip...] 

   "operatingsystem"=>"Ubuntu",
   "operatingsystemrelease"=>"12.04",
   "_timestamp"=>"2015-07-27 06:39:53 -0400"}}

Output:

{}

Exception:

ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

Backtrace:

/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:41:in `block in reset_counters'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `reset_counters'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:19:in `block in update_counter_caches'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `each'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `update_counter_caches'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:416:in `_run__4048862386791947774__update__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:272:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:348:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:689:in `_run__4048862386791947774__save__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/managed.rb:461:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:126:in `import_facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware.rb:16:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:16:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/world.rb:30:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `block in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:480:in `execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:262:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/context.rb:47:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/actor.rb:26:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:21:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:161:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `block in synchronize'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:97:in `work'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:78:in `block in call_job'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `run_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'

Related issues 1 (0 open — 1 closed)

Is duplicate of Foreman - Bug #10133: Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rbClosedTomer Brisker04/14/2015Actions
Actions
Copy link

Also available in: Atom PDF