Bug #11232: Occassional error in tasks when importing facts from foreman-chef - Chef - Foreman

Actions

Copy link

Bug #11232

closed

Occassional error in tasks when importing facts from foreman-chef

Added by Alex Balk over 9 years ago. Updated over 9 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Tomer Brisker

Category:

Target version:

Difficulty:

Triaged:

Bugzilla link:

Pull request:

Fixed in Releases:

Found in Releases:

Red Hat JIRA:

Description

Foreman 1.9RC2 on Ubuntu 12.04.5, foreman-chef 0.1.5-1 and foreman-tasks 0.7.1-1, running 5 executors.

Sometimes fact uploads pause in state "error" with the message below. When clicking "resume" they complete successfully.

PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

This is the error from the "Errors" tab:


{"host"=>{"id"=>3003, "name"=>"foo.bar.com"},
 "facts"=>
  {"dmi::dmidecode_version"=>"2.11",
   "dmi::smbios_version"=>"2.7",

[...snip...] 

   "operatingsystem"=>"Ubuntu",
   "operatingsystemrelease"=>"12.04",
   "_timestamp"=>"2015-07-27 06:39:53 -0400"}}

Output:

{}

Exception:

ActiveRecord::StatementInvalid: PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 646 waits for ShareLock on transaction 11925197; blocked by process 647.
Process 647 waits for ExclusiveLock on tuple (0,7) of relation 16545 of database 16385; blocked by process 646.
HINT:  See server log for query details.
: UPDATE "operatingsystems" SET "hosts_count" = 759 WHERE "operatingsystems"."id" = 2

Backtrace:

/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:41:in `block in reset_counters'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `reset_counters'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:19:in `block in update_counter_caches'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `each'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `update_counter_caches'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:416:in `_run__4048862386791947774__update__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:272:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:348:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:689:in `_run__4048862386791947774__save__2111300948237819981__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/managed.rb:461:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:126:in `import_facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware.rb:16:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:16:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/world.rb:30:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `block in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:480:in `execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:262:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/context.rb:47:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/actor.rb:26:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:21:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:161:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `block in synchronize'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:97:in `work'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:78:in `block in call_job'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `run_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Alex Balk over 9 years ago

Issues is not isolated to update on "operatingsystems":


PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95774 waits for ShareLock on transaction 11982799; blocked by process 1053.
Process 1053 waits for ExclusiveLock on tuple (0,47) of relation 16565 of database 16385; blocked by process 95774.
HINT:  See server log for query details.
: UPDATE "environments" SET "hosts_count" = 2242 WHERE "environments"."id" = 2

/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `async_exec'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:1163:in `exec_no_cache'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_delete'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:41:in `block in reset_counters'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/counter_cache.rb:19:in `reset_counters'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:19:in `block in update_counter_caches'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `each'
/usr/share/foreman/app/models/concerns/counter_cache_fix.rb:13:in `update_counter_caches'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:416:in `_run__3506879273860728703__update__1282167615730828315__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:272:in `update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:348:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:689:in `_run__3506879273860728703__save__1282167615730828315__callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activesupport-3.2.21/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block (2 levels) in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:259:in `block in save'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.9.1/gems/activerecord-3.2.21/lib/active_record/transactions.rb:258:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/models/host/managed.rb:461:in `populate_fields_from_facts'
/usr/share/foreman/app/models/host/base.rb:126:in `import_facts'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:20:in `block in run'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
/usr/share/foreman/vendor/ruby/1.9.1/gems/foreman-tasks-0.7.1/app/lib/actions/foreman/host/import_facts.rb:18:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:487:in `block (3 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware.rb:16:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action/progress.rb:16:in `run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/stack.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/middleware/world.rb:30:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:486:in `block (2 levels) in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:485:in `block in execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `block in with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:402:in `with_error_handling'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:480:in `execute_run'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/action.rb:262:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/context.rb:47:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/dynflow-0.8.1/lib/dynflow/actor.rb:26:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:21:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/abstract.rb:26:in `pass'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:161:in `process_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `block in synchronize'
/usr/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/synchronization/monitor_object.rb:15:in `synchronize'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-edge-0.1.0.pre3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:19:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:97:in `work'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/serialized_execution.rb:78:in `block in call_job'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:332:in `run_task'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (3 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `loop'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:303:in `block (2 levels) in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `catch'
/usr/share/foreman/vendor/ruby/1.9.1/gems/concurrent-ruby-0.9.0.pre3/lib/concurrent/executor/ruby_thread_pool_executor.rb:302:in `block in create_worker'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `call'
/usr/share/foreman/vendor/ruby/1.9.1/gems/logging-2.0.0/lib/logging/diagnostic_context.rb:448:in `block in create_with_logging_context'

Actions

Copy link

Updated by Alex Balk over 9 years ago

A few more (minus trace):


PG::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95745 waits for ExclusiveLock on tuple (0,77) of relation 16565 of database 16385; blocked by process 95773.
Process 95773 waits for ShareLock on transaction 12746545; blocked by process 98646.
Process 98646 waits for ExclusiveLock on tuple (0,77) of relation 16565 of database 16385; blocked by process 95745.
HINT:  See server log for query details.
: UPDATE "environments" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "environments"."id" IN (SELECT "environments"."id" FROM "environments"  WHERE "environments"."id" = 2 ORDER BY environments.name)

G::TRDeadlockDetected: ERROR:  deadlock detected
DETAIL:  Process 95774 waits for ShareLock on transaction 10396489; blocked by process 95773.
Process 95773 waits for ShareLock on transaction 10396502; blocked by process 95774.
HINT:  See server log for query details.
CONTEXT:  SQL statement "SELECT 1 FROM ONLY "public"."models" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x" 
: UPDATE "hosts" SET "model_id" = 2, "operatingsystem_id" = 7, "environment_id" = 2, "updated_at" = '2015-07-27 10:51:21.142286' WHERE "hosts"."type" IN ('Host::Managed') AND "hosts"."id" = 2500

Actions

Copy link

Updated by Tomer Brisker over 9 years ago

Is duplicate of Bug #10133: Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rb added

Actions

Copy link

Updated by Tomer Brisker over 9 years ago

Status changed from New to Duplicate

This looks like a duplicate of #10133.
Please try the patch (https://github.com/theforeman/foreman/pull/2362) that was proposed for it, that PR was closed as we could not reproduce the error.
If the patch fixes this problem let us know so we can merge it.

Actions

Copy link