Project

General

Profile

Bug #21599

duplicate key value violates unique constraint "index_fact_names_on_name_and_type" on simulatenous host register

Added by Shimon Shtein almost 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Category:
Facts
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1474293

Description of problem:
Our automation hit yet another race condition on registering the hosts simultaneously (2 hosts).
See attached foreman-debug:
production.log - see requests: 4c6a15a5 and 64762ef4 (which failed) (2017-07-20 21:05:54)

2017-07-20 21:05:51 64762ef4 [app] [I] Started POST "/rhsm/consumers?owner=Ag72NG&activation_keys=IsRwhiODa7" for 10.8.30.229 at 2017-07-20 21:05:51 -0400
2017-07-20 21:05:51 64762ef4 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
...
...
2017-07-20 21:05:54 [audit] [I] [140318963445328.lab.eng.rdu2.redhat.com] added 153 (3963.0ms)
2017-07-20 21:05:54 [app] [I] Import facts for '140318963445328.lab.eng.rdu2.redhat.com' completed. Added: 153, Updated: 0, Deleted 0 facts
2017-07-20 21:05:54 [app] [E] Fact dmi::system::uuid could not be imported because of PG::Error: ERROR: duplicate key value violates unique constraint "index_fact_names_on_name_and_type" | : INSERT INTO "fact_names" ("compose", "created_at", "name", "short_name", "type", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
2017-07-20 21:05:54 [app] [E] Fact distribution::name could not be imported because of PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block | : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'distribution' AND "fact_names"."type" = 'Katello::RhsmFactName' ORDER BY fact_names.name LIMIT 1
...
...
2017-07-20 21:05:55 [audit] [I] [140599713026320.lab.eng.rdu2.redhat.com] added 166 (1966.6ms)
2017-07-20 21:05:55 [app] [W] Error during fact import for 140599713026320.lab.eng.rdu2.redhat.com | ActiveRecord::StatementInvalid: PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block | : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'memor | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `ex | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `bl
...
...
2017-07-20 21:05:55 [foreman-tasks/action] [E] ERF50-2486 [Foreman::WrappedException]: Import of facts failed for host 14059971302632 | /usr/share/foreman/app/services/fact_importer.rb:48:in `import!'
... | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `blo | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_c
2017-07-20 21:05:56 64762ef4 [app] [I] Completed 500 Internal Server Error in 5045ms (Views: 0.6ms | ActiveRecord: 92.5ms)
2017-07-20 21:05:57 4c6a15a5 [app] [I] Completed 200 OK in 9308ms (Views: 2.9ms | ActiveRecord: 45.2ms)

Version-Release number of selected component (if applicable):
6.2.11-1

How reproducible:
sometimes

Steps to Reproduce:
hard to reproduce, however:
1. get 2 different hosts register to the same satellite at the same time, with the same org, env and activation key
2. making katello more busy might help, in our case, there were some POST requests to Environments and Content Views made just after the register requests.

Actual results:
duplicate key value violates unique constraint "index_fact_names_on_name_and_type"


Related issues

Related to Foreman - Bug #9016: Fact import code consumes lot of memoryClosed2015-01-18
Related to Foreman - Bug #21776: Fix fact names transaction handlingClosed2017-11-27

Associated revisions

Revision b5256c3a (diff)
Added by Lukas Zapletal almost 3 years ago

Fixes #21599 - explicit transaction for import dropped

History

#1 Updated by The Foreman Bot almost 3 years ago

  • Assignee set to Shimon Shtein
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/4996 added

#2 Updated by Marek Hulán almost 3 years ago

  • Related to Bug #9016: Fact import code consumes lot of memory added

#3 Updated by The Foreman Bot almost 3 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5007 added

#4 Updated by Ivan Necas almost 3 years ago

  • Category set to Facts

#5 Updated by Lukas Zapletal almost 3 years ago

  • % Done changed from 0 to 100
  • Status changed from Ready For Testing to Closed

#6 Updated by Lukas Zapletal almost 3 years ago

  • Status changed from Closed to Assigned

THERE ARE TWO PATCHES ASSOCIATED.

One is quick and dirty patch of removing explicit SQL transactions, this should be easily backportable in case you are hit by this bug:
https://github.com/theforeman/foreman/pull/5007

The other one is much more complex, it keeps transactions and only pulls fact name creation out:
https://github.com/theforeman/foreman/pull/5007

#7 Updated by Marek Hulán almost 3 years ago

  • Legacy Backlogs Release (now unused) set to 296
  • Assignee changed from Shimon Shtein to Lukas Zapletal
  • Status changed from Assigned to Closed
  • Pull request deleted (https://github.com/theforeman/foreman/pull/5007)

Marking as 1.17.0 as the transaction was dropped, the second PR is more complicated and it's not clear whether it makes it into 1.17 so I'm removing it for now. If it will make it in before branching, it can be readded, otherwise a new redmine issue should be created.

#8 Updated by Shimon Shtein almost 3 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5007 added
  • Pull request deleted (https://github.com/theforeman/foreman/pull/4996)

Marek, I think you meant to remove https://github.com/theforeman/foreman/pull/4996 and leave the transaction dropping (https://github.com/theforeman/foreman/pull/5007)

#9 Updated by Lukas Zapletal almost 3 years ago

  • Related to Bug #21776: Fix fact names transaction handling added

Also available in: Atom PDF