Project

General

Profile

Bug #35485

While running concurrent registrations, foreman fails with 500 ISE on index_operatingsystems_on_title unique index violation

Added by Pavel Moravec 3 months ago. Updated 21 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Facts
Target version:
-
Difficulty:
easy
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

High level reproducer:
- concurrently register systems with the same OS title (like "RedHat 8.6")
- .. while that OS was not listed among OSes in foreman

Particular reproducer:

1) Have 5 RHEL8.6 systems named "my-rhel86-[1-5]", with katello-ca-latest installed / prepared to run sub-man register
2) Have some AK (optionally)
3) Have no RHEL8.6 system on that foreman instance
3) Run below script:

#!/bin/bash

SYSTEMS=5

while true; do
echo "$(date): registering all $SYSTEMS systems"
for i in $(seq 1 $SYSTEMS); do
ssh my-rhel86-${i} "subscription-manager register --activationkey=ak_library --org=myOrg" &
done
wait
sleep 5
echo "$(date): registered all systems:"
hammer host list --search os_minor=6
echo "$(date): deleting all $SYSTEMS systems"
for i in $(seq 1 $SYSTEMS); do
ssh my-rhel86-${i} "subscription-manager unregister" &
done
wait
echo "$(date): unregstered, deleting hosts.."
for i in $(seq 1 $SYSTEMS); do
echo "host delete --name my-rhel86-${i}"
done | hammer shell
hammer host list --search os_minor=6
hammer os delete --title "RedHat 8.6"
done

Within a few iterations, below error happens (stack trace is from Satellite 6.11, not upstream foreman):

2022-09-03T20:27:22 [I|app|a0e6beaa] Started POST "/rhsm/consumers?owner=RedHat&activation_keys=ak_library" for 1.2.3.4 at 2022-09-03 20:27:22 +0200
2022-09-03T20:27:22 [I|app|a0e6beaa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2022-09-03T20:27:22 [I|app|a0e6beaa] Parameters: {"type"=>"system", "name"=>"my-rhel86-3.example.com", "facts"=>"[FILTERED]", "installedProducts"=>[{"productId"=>"479", "productName"=>"Red Hat Enterprise Linux for x86_64", "version"=>"8.6", "arch"=>"x86_64"}], "contentTags"=>["rhel-8", "rhel-8-x86_64"], "role"=>"", "addOns"=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"RedHat", "activation_keys"=>"ak_library"}
2022-09-03T20:27:22 [I|aud|a0e6beaa] Nic::Managed (147) create event on mac
..
2022-09-03T20:27:25 [I|aud|a0e6beaa] Host::Base (147) destroy event on ansible_role_ids []
2022-09-03T20:27:25 [I|app|a0e6beaa] ForemanWebhooks::EventSubscriber: host_destroyed.event.foreman event received
2022-09-03T20:27:25 [E|kat|a0e6beaa] ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_operatingsystems_on_title"
a0e6beaa | DETAIL: Key (title)=(RedHat 8.6) already exists.
a0e6beaa |
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
..
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/suppressor.rb:48:in `save!'
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/persistence.rb:55:in `create!'
a0e6beaa | /usr/share/foreman/app/services/katello/rhsm_fact_parser.rb:71:in `operatingsystem'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:166:in `block in set_non_empty_values'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:165:in `each'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:165:in `set_non_empty_values'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:159:in `populate_fields_from_facts'
a0e6beaa | /usr/share/foreman/app/models/host/managed.rb:490:in `populate_fields_from_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
a0e6beaa | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:28:in `telemetry_duration_histogram'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
a0e6beaa | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:143:in `import_facts'
..

Evident cause:

rhsm_fact_parser.rb#L79 :

::Operatingsystem.find_by(os_attributes) || ::Operatingsystem.create!(os_attributes)

should be:

::Operatingsystem.find_or_create_by(os_attributes)

Associated revisions

Revision a11d509b (diff)
Added by Pavel Moravec 2 months ago

Fixes #35485 - concurrent safe create OS when parsing facts (#9403)

Concurrent execution of this line can attempt to create just created
::Operatingsystem. Use proper find_or_create_by method for that.

Signed-off-by: Pavel Moravec <>

Signed-off-by: Pavel Moravec <>

History

#1 Updated by The Foreman Bot 3 months ago

  • Assignee set to Pavel Moravec
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/9403 added

#2 Updated by The Foreman Bot 2 months ago

  • Fixed in Releases 3.5.0 added

#3 Updated by Pavel Moravec 2 months ago

  • Status changed from Ready For Testing to Closed

#4 Updated by Ewoud Kohl van Wijngaarden 21 days ago

  • Triaged changed from No to Yes
  • Category set to Facts

Also available in: Atom PDF