Bug #5990: multiple calls to create or update domain throws deadlock error - Foreman

Custom queries

3.11.4 release DONE
3.11.4 release TODO
3.12.1 release DONE
3.12.1 release TODO
3.13.0 release DONE
3.13.0 release TODO
All bugs by votes
All issues closed in the last 3 weeks
Easy and Trivial Issues
Easy Issues (only)
Foreman - new open issues in last 10 days
Good first issues - Foreman core
high prio open bugs
Host registration [open]
Mine
My issues closed in the last 3 weeks
old bugs
Open with target version set (aka blockers)
registration
taken-none
Team Atlas untriaged bugs
templates
Trivial Issues
Untriaged and opened in the past two weeks

Actions

Copy link

Bug #5990

closed

multiple calls to create or update domain throws deadlock error

Added by Thomas McKay over 10 years ago. Updated about 9 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Category:

Database

Target version:

Difficulty:

Triaged:

Bugzilla link:

1102871

Pull request:

Fixed in Releases:

Found in Releases:

Red Hat JIRA:

Description

Running hammer-cli-csv w/ 6 threads against a 7 CPU server

```
E, [2014-05-29T13:48:16.720000 #2214] ERROR -- : 500 Internal Server Error {
"error" => {
"message" => "PGError: ERROR: deadlock detected\nDETAIL: Process 13511 waits for ShareLock on transaction 25358; blocked by process 13424.\nProcess 13424 waits for ExclusiveLock on tuple (0,31) of relation 18370 of database 17586; blocked by process 13511.\nHINT: See server log for query details.\n: UPDATE \"taxonomies\" SET \"updated_at\" = '2014-05-29 21:46:53.703727', \"ignore_types\" = '--- []\n', \"default_info\" = '---\nsystem: []\ndistributor: []\n' WHERE \"taxonomies\".\"type\" IN ('Organization') AND \"taxonomies\".\"id\" = 3"
}
}
```

Files

log.txt	log.txt	19.5 KB	Thomas McKay, 05/29/2014 05:52 PM
create_psauxww.txt	create_psauxww.txt	15.6 KB	Johan Sunnerstig, 07/04/2014 03:39 AM
create_trace.txt	create_trace.txt	18.2 KB	Johan Sunnerstig, 07/04/2014 03:39 AM
delete_psauxww.txt	delete_psauxww.txt	19.7 KB	Johan Sunnerstig, 07/04/2014 03:39 AM
delete_trace.txt	delete_trace.txt	16.1 KB	Johan Sunnerstig, 07/04/2014 03:39 AM

Related issues 1 (0 open — 1 closed)

Is duplicate of Foreman - Bug #10133: Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rb

Closed

Tomer Brisker

04/14/2015

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Thomas McKay over 10 years ago

Bugzilla link set to https://bugzilla.redhat.com/show_bug.cgi?id=1102871

Actions

Copy link

Updated by Thomas McKay over 10 years ago

Similar errors for creating hosts

```
E, [2014-05-29T14:26:02.345000 #5199] ERROR -- : 500 Internal Server Error {
"error" => {
"message" => "PGError: ERROR: deadlock detected\nDETAIL: Process 17187 waits for ShareLock on transaction 30354; blocked by process 17212.\nProcess 17212 waits for ExclusiveLock on tuple (0,55) of relation 17753 of database 17586; blocked by process 17187.\nHINT: See server log for query details.\n: UPDATE \"architectures\" SET \"hosts_count\" = COALESCE + 1 WHERE \"architectures\".\"id\" = 1"
}
}
```

Actions

Copy link

Updated by Lukas Zapletal over 10 years ago

Tom, can you attach ps auxwww output while the test was running?

Actions

Copy link Download all files

Updated by Johan Sunnerstig over 10 years ago

File create_psauxww.txt create_psauxww.txt added
File create_trace.txt create_trace.txt added
File delete_psauxww.txt delete_psauxww.txt added
File delete_trace.txt delete_trace.txt added

Lukas Zapletal wrote:

Tom, can you attach ps auxwww output while the test was running?

Hi,
I'm running into what seems to be the same problem, only we're using MySQL. Here are some logs and a ps auxww from both deletion and creation of servers in batches of 50.
We have two Foreman hosts, I've tried running both against their load balanced address and individually and it doesn't seem to make a difference. Neither one has much load to speak of.
Foreman 1.5.1, Debian 7.5, ruby 1.9.3 running behind Passenger 3.

Oh and the MySQL database in one of the ps outputs is not the one the foreman servers are using, I don't think it's used at all, I'm just not sure enough to just remove it yet.

Actions

Copy link

Updated by Filip Pytloun over 9 years ago

Hello,

we are facing the same issue when deploying new hosts.
Unfortunately I don't have a reproducer yet and it happens only occasionaly :-/

Foreman API returns following:

 "error": {"message":"PGError: ERROR:  deadlock detected\nDETAIL:  Process 19775 waits for ShareLock on transaction 65552980; blocked by process 23414.\nProcess 23414 waits for ExclusiveLock on tuple (0,25) of relation 16503 of database 16414; blocked by process 19775.\nHINT:  See server log for query details.\n: UPDATE \"architectures\" SET \"hosts_count\" = COALESCE(\"hosts_count\", 0) + 1 WHERE \"architectures\".\"id\" = 1"}

This can be found in postgres logs:

Mar 23 13:14:36 foreman postgres[19775]: [6] (2015-03-23 12:14:36 GMT foreman [local] [unknown]) ERROR:  deadlock detected||(2015-03-23 12:14:36 GMT foreman [local] [unknown]) DETAIL:  Process 19775 waits for ShareLock on transaction 65552980; blocked by process 23414.\nProcess 23414 waits for ExclusiveLock on tuple (0,25) of relation 16503 of database 16414; blocked by process 19775.\nProcess 19775: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1\nProcess 23414: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1||(2015-03-23 12:14:36 GMT foreman [local] [unknown]) HINT:  See server log for query details.||STATEMENT:  UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1 severity=warning

Slow query log for mentioned processes above:

Mar 23 13:07:47 foreman postgres[23414]: slow_query database=foreman client=local appname= severity=emerg
Mar 23 13:07:47 foreman postgres[23414]: #011 duration: 25299.736 (locks objects=0.000 pages=0.000 others=25296.784) ms  execute <unnamed>: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) - 1 WHERE "architectures"."id" = 1 severity=emerg
Mar 23 13:14:35 foreman postgres[19775]: slow_query database=foreman client=local appname= severity=emerg
Mar 23 13:14:35 foreman postgres[19775]: #011 duration: 7590.179 (locks objects=0.000 pages=0.000 others=7588.383) ms  execute a11: INSERT INTO "hosts" ("architecture_id", "build", "certname", "comment", "compute_profile_id", "compute_resource_id", "created_at", "disk", "domain_id", "enabled", "environment_id", "hostgroup_id", "image_file", "image_id", "installed_at", "ip", "last_compile", "last_freshcheck", "last_report", "location_id", "mac", "managed", "medium_id", "model_id", "name", "operatingsystem_id", "organization_id", "otp", "owner_id", "owner_type", "provision_method", "ptable_id", "puppet_ca_proxy_id", "puppet_proxy_id", "puppet_status", "realm_id", "root_pass", "serial", "source_file_id", "subnet_id", "type", "updated_at", "use_image", "uuid") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44) RETURNING "id" severity=emerg
Mar 23 13:14:35 foreman postgres[23414]: slow_query database=foreman client=local appname= severity=emerg
Mar 23 13:14:35 foreman postgres[23414]: #011 duration: 8119.303 (locks objects=0.000 pages=0.000 others=8115.033) ms  execute a15: INSERT INTO "hosts" ("architecture_id", "build", "certname", "comment", "compute_profile_id", "compute_resource_id", "created_at", "disk", "domain_id", "enabled", "environment_id", "hostgroup_id", "image_file", "image_id", "installed_at", "ip", "last_compile", "last_freshcheck", "last_report", "location_id", "mac", "managed", "medium_id", "model_id", "name", "operatingsystem_id", "organization_id", "otp", "owner_id", "owner_type", "provision_method", "ptable_id", "puppet_ca_proxy_id", "puppet_proxy_id", "puppet_status", "realm_id", "root_pass", "serial", "source_file_id", "subnet_id", "type", "updated_at", "use_image", "uuid") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41, $42, $43, $44) RETURNING "id" severity=emerg

Full Foreman stack trace:

PGError: ERROR:  deadlock detected
DETAIL:  Process 19775 waits for ShareLock on transaction 65552980; blocked by process 23414.
Process 23414 waits for ExclusiveLock on tuple (0,25) of relation 16503 of database 16414; blocked by process 19775.
HINT:  See server log for query details.
: UPDATE "architectures" SET "hosts_count" = COALESCE("hosts_count", 0) + 1 WHERE "architectures"."id" = 1 (ActiveRecord::StatementInvalid)
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:1158:in `async_exec'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:1158:in `exec_no_cache'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block in exec_delete'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:674:in `exec_delete'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:96:in `update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation.rb:294:in `update_all'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation.rb:278:in `update_all'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/querying.rb:7:in `update_all'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/counter_cache.rb:80:in `update_counters'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/locking/optimistic.rb:166:in `update_counters'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/counter_cache.rb:99:in `increment_counter'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/builder/belongs_to.rb:30:in `block in add_counter_cache_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:449:in `_run__1526810562777591602__create__4438396813344648884__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:268:in `create'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:347:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `block in create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:667:in `_run__1526810562777591602__save__4438396813344648884__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/persistence.rb:84:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:50:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block (2 levels) in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:241:in `block in save'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:252:in `rollback_active_record_state!'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:240:in `save'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save_with_type'
/usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:77:in `create'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:528:in `block (4 levels) in _run__1986487176532718764__process_action__2724997943193415624__callbacks'share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_6921'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:93:in `disable_json_root'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6921'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:516:in `block (3 levels) in _run__1986487176532718764__process_action__2724997943193415624__callbacks'share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_6920'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_2107'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6920'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:493:in `block (2 levels) in _run__1986487176532718764__process_action__2724997943193415624__callbacks'share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_6919'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6919'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:426:in `block in _run__1986487176532718764__process_action__2724997943193415624__callbacks'3/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_6918'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_6918'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in `_run__1986487176532718764__process_action__2724997943193415624__callbacks'rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/mapper.rb:42:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/extractor/recorder.rb:97:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:35:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/query_cache.rb:64:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__4158480795765478168__call__4438396813344648884__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:26:in `call_app'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/rack/logger.rb:16:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/builder.rb:134:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:64:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.5/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.5/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.5/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.5/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'

Actions

Copy link