Project

General

Profile

Actions

Bug #11232

closed

Occassional error in tasks when importing facts from foreman-chef

Added by Alex Balk over 8 years ago. Updated over 8 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

Foreman 1.9RC2 on Ubuntu 12.04.5, foreman-chef 0.1.5-1 and foreman-tasks 0.7.1-1, running 5 executors.

Sometimes fact uploads pause in state "error" with the message below. When clicking "resume" they complete successfully.

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

This is the error from the "Errors" tab:


{"host"=>{"id"=>3003, "name"=>"foo.bar.com"},
 "facts"=>
  {"dmi::dmidecode_version"=>"2.11",
   "dmi::smbios_version"=>"2.7",

[...snip...] 

   "operatingsystem"=>"Ubuntu",
   "operatingsystemrelease"=>"12.04",
   "_timestamp"=>"2015-07-27 06:39:53 -0400"}}

Output:

{}

Exception:

ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

Backtrace:

/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:41:in `block in reset_counters'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `reset_counters'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:19:in `block in update_counter_caches'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `each'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `update_counter_caches'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:416:in `_run__4048862386791947774__update__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:272:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:348:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:689:in `_run__4048862386791947774__save__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/managed.rb:461:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:126:in `import_facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware.rb:16:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:16:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/world.rb:30:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `block in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:480:in `execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:262:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/context.rb:47:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/actor.rb:26:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:21:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:161:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `block in synchronize'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:97:in `work'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:78:in `block in call_job'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `run_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'


Related issues 1 (0 open1 closed)

Is duplicate of Foreman - Bug #10133: Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rbClosedTomer Brisker04/14/2015Actions
Actions #1

Updated by Alex Balk over 8 years ago

Issues is not isolated to update on "operatingsystems":


PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95774 waits for ShareLock on transaction 11982799; blocked by process 1053.
Process 1053 waits for ExclusiveLock on tuple (0,47) of relation 16565 of database 16385; blocked by process 95774.
HINT:  See server log for query details.
: UPDATE "environments" SET "hosts_count" = 2242 WHERE "environments"."id" = 2

/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:41:in `block in reset_counters'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `reset_counters'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:19:in `block in update_counter_caches'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `each'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `update_counter_caches'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:416:in `_run__3506879273860728703__update__1282167615730828315__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:272:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:348:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:689:in `_run__3506879273860728703__save__1282167615730828315__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/managed.rb:461:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:126:in `import_facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware.rb:16:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:16:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/world.rb:30:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `block in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:480:in `execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:262:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/context.rb:47:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/actor.rb:26:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:21:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:161:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `block in synchronize'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:97:in `work'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:78:in `block in call_job'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `run_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'

Actions #2

Updated by Alex Balk over 8 years ago

A few more (minus trace):


PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95745 waits for ExclusiveLock on tuple (0,77) of relation 16565 of database 16385; blocked by process 95773.
Process 95773 waits for ShareLock on transaction 12746545; blocked by process 98646.
Process 98646 waits for ExclusiveLock on tuple (0,77) of relation 16565 of database 16385; blocked by process 95745.
HINT:  See server log for query details.
: UPDATE "environments" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "environments"."id" IN (SELECT "environments"."id" FROM "environments"  WHERE "environments"."id" = 2 ORDER BY environments.name)

G::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95774 waits for ShareLock on transaction 10396489; blocked by process 95773.
Process 95773 waits for ShareLock on transaction 10396502; blocked by process 95774.
HINT:  See server log for query details.
CONTEXT:  SQL statement "SELECT 1 FROM ONLY "public"."models" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x" 
: UPDATE "hosts" SET "model_id" = 2, "operatingsystem_id" = 7, "environment_id" = 2, "updated_at" = '2015-07-27 10:51:21.142286' WHERE "hosts"."type" IN ('Host::Managed') AND "hosts"."id" = 2500

Actions #3

Updated by Tomer Brisker over 8 years ago

  • Is duplicate of Bug #10133: Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rb added
Actions #4

Updated by Tomer Brisker over 8 years ago

  • Status changed from New to Duplicate

This looks like a duplicate of #10133.
Please try the patch (https://github.com/theforeman/foreman/pull/2362) that was proposed for it, that PR was closed as we could not reproduce the error.
If the patch fixes this problem let us know so we can merge it.

Actions #5

Updated by Tomer Brisker over 8 years ago

  • Assignee set to Tomer Brisker
Actions #6

Updated by Marek Hulán over 8 years ago

Patch was applied and tested, no deadlock appeared! Will update the PR with this finding. Thanks Ohad Levy for spotting the relation with #10133

Actions

Also available in: Atom PDF