Bug #21599
closedduplicate key value violates unique constraint "index_fact_names_on_name_and_type" on simulatenous host register
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1474293
Description of problem:
Our automation hit yet another race condition on registering the hosts simultaneously (2 hosts).
See attached foreman-debug:
production.log - see requests: 4c6a15a5 and 64762ef4 (which failed) (2017-07-20 21:05:54)
2017-07-20 21:05:51 64762ef4 [app] [I] Started POST "/rhsm/consumers?owner=Ag72NG&activation_keys=IsRwhiODa7" for 10.8.30.229 at 2017-07-20 21:05:51 -0400
2017-07-20 21:05:51 64762ef4 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
...
...
2017-07-20 21:05:54 [audit] [I] [140318963445328.lab.eng.rdu2.redhat.com] added 153 (3963.0ms)
2017-07-20 21:05:54 [app] [I] Import facts for '140318963445328.lab.eng.rdu2.redhat.com' completed. Added: 153, Updated: 0, Deleted 0 facts
2017-07-20 21:05:54 [app] [E] Fact dmi::system::uuid could not be imported because of PG::Error: ERROR: duplicate key value violates unique constraint "index_fact_names_on_name_and_type"
| : INSERT INTO "fact_names" ("compose", "created_at", "name", "short_name", "type", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
2017-07-20 21:05:54 [app] [E] Fact distribution::name could not be imported because of PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block
| : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'distribution' AND "fact_names"."type" = 'Katello::RhsmFactName' ORDER BY fact_names.name LIMIT 1
...
...
2017-07-20 21:05:55 [audit] [I] [140599713026320.lab.eng.rdu2.redhat.com] added 166 (1966.6ms)
2017-07-20 21:05:55 [app] [W] Error during fact import for 140599713026320.lab.eng.rdu2.redhat.com
| ActiveRecord::StatementInvalid: PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block
| : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'memor
| /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `ex
| /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `bl
...
...
2017-07-20 21:05:55 [foreman-tasks/action] [E] ERF50-2486 [Foreman::WrappedException]: Import of facts failed for host 14059971302632
| /usr/share/foreman/app/services/fact_importer.rb:48:in `import!'
...
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `blo
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_c
2017-07-20 21:05:56 64762ef4 [app] [I] Completed 500 Internal Server Error in 5045ms (Views: 0.6ms | ActiveRecord: 92.5ms)
2017-07-20 21:05:57 4c6a15a5 [app] [I] Completed 200 OK in 9308ms (Views: 2.9ms | ActiveRecord: 45.2ms)
Version-Release number of selected component (if applicable):
6.2.11-1
How reproducible:
sometimes
Steps to Reproduce:
hard to reproduce, however:
1. get 2 different hosts register to the same satellite at the same time, with the same org, env and activation key
2. making katello more busy might help, in our case, there were some POST requests to Environments and Content Views made just after the register requests.
Actual results:
duplicate key value violates unique constraint "index_fact_names_on_name_and_type"
Updated by The Foreman Bot about 7 years ago
- Status changed from New to Ready For Testing
- Assignee set to Shimon Shtein
- Pull request https://github.com/theforeman/foreman/pull/4996 added
Updated by Marek Hulán about 7 years ago
- Related to Bug #9016: Fact import code consumes lot of memory added
Updated by The Foreman Bot about 7 years ago
- Pull request https://github.com/theforeman/foreman/pull/5007 added
Updated by Lukas Zapletal about 7 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset b5256c3a3cd23c50ef6d90e44ea0c45c2500cc00.
Updated by Lukas Zapletal about 7 years ago
- Status changed from Closed to Assigned
THERE ARE TWO PATCHES ASSOCIATED.
One is quick and dirty patch of removing explicit SQL transactions, this should be easily backportable in case you are hit by this bug:
https://github.com/theforeman/foreman/pull/5007
The other one is much more complex, it keeps transactions and only pulls fact name creation out:
https://github.com/theforeman/foreman/pull/5007
Updated by Marek Hulán about 7 years ago
- Status changed from Assigned to Closed
- Assignee changed from Shimon Shtein to Lukas Zapletal
- Translation missing: en.field_release set to 296
- Pull request deleted (
https://github.com/theforeman/foreman/pull/5007)
Marking as 1.17.0 as the transaction was dropped, the second PR is more complicated and it's not clear whether it makes it into 1.17 so I'm removing it for now. If it will make it in before branching, it can be readded, otherwise a new redmine issue should be created.
Updated by Shimon Shtein about 7 years ago
- Pull request https://github.com/theforeman/foreman/pull/5007 added
- Pull request deleted (
https://github.com/theforeman/foreman/pull/4996)
Marek, I think you meant to remove https://github.com/theforeman/foreman/pull/4996 and leave the transaction dropping (https://github.com/theforeman/foreman/pull/5007)
Updated by Lukas Zapletal about 7 years ago
- Related to Bug #21776: Fix fact names transaction handling added