Bug #35485
closedWhile running concurrent registrations, foreman fails with 500 ISE on index_operatingsystems_on_title unique index violation
Description
High level reproducer:
- concurrently register systems with the same OS title (like "RedHat 8.6")
- .. while that OS was not listed among OSes in foreman
Particular reproducer:
1) Have 5 RHEL8.6 systems named "my-rhel86-[1-5]", with katello-ca-latest installed / prepared to run sub-man register
2) Have some AK (optionally)
3) Have no RHEL8.6 system on that foreman instance
3) Run below script:
#!/bin/bash
SYSTEMS=5
while true; do
echo "$(date): registering all $SYSTEMS systems"
for i in $(seq 1 $SYSTEMS); do
ssh my-rhel86-${i} "subscription-manager register --activationkey=ak_library --org=myOrg" &
done
wait
sleep 5
echo "$(date): registered all systems:"
hammer host list --search os_minor=6
echo "$(date): deleting all $SYSTEMS systems"
for i in $(seq 1 $SYSTEMS); do
ssh my-rhel86-${i} "subscription-manager unregister" &
done
wait
echo "$(date): unregstered, deleting hosts.."
for i in $(seq 1 $SYSTEMS); do
echo "host delete --name my-rhel86-${i}"
done | hammer shell
hammer host list --search os_minor=6
hammer os delete --title "RedHat 8.6"
done
Within a few iterations, below error happens (stack trace is from Satellite 6.11, not upstream foreman):
2022-09-03T20:27:22 [I|app|a0e6beaa] Started POST "/rhsm/consumers?owner=RedHat&activation_keys=ak_library" for 1.2.3.4 at 2022-09-03 20:27:22 +0200
2022-09-03T20:27:22 [I|app|a0e6beaa] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
2022-09-03T20:27:22 [I|app|a0e6beaa] Parameters: {"type"=>"system", "name"=>"my-rhel86-3.example.com", "facts"=>"[FILTERED]", "installedProducts"=>[{"productId"=>"479", "productName"=>"Red Hat Enterprise Linux for x86_64", "version"=>"8.6", "arch"=>"x86_64"}], "contentTags"=>["rhel-8", "rhel-8-x86_64"], "role"=>"", "addOns"=>[], "usage"=>"", "serviceLevel"=>"", "owner"=>"RedHat", "activation_keys"=>"ak_library"}
2022-09-03T20:27:22 [I|aud|a0e6beaa] Nic::Managed (147) create event on mac
..
2022-09-03T20:27:25 [I|aud|a0e6beaa] Host::Base (147) destroy event on ansible_role_ids []
2022-09-03T20:27:25 [I|app|a0e6beaa] ForemanWebhooks::EventSubscriber: host_destroyed.event.foreman event received
2022-09-03T20:27:25 [E|kat|a0e6beaa] ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_operatingsystems_on_title"
a0e6beaa | DETAIL: Key (title)=(RedHat 8.6) already exists.
a0e6beaa |
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `exec_params'
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
..
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/suppressor.rb:48:in `save!'
a0e6beaa | /usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/persistence.rb:55:in `create!'
a0e6beaa | /usr/share/foreman/app/services/katello/rhsm_fact_parser.rb:71:in `operatingsystem'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:166:in `block in set_non_empty_values'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:165:in `each'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:165:in `set_non_empty_values'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:159:in `populate_fields_from_facts'
a0e6beaa | /usr/share/foreman/app/models/host/managed.rb:490:in `populate_fields_from_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:50:in `block (2 levels) in parse_facts'
a0e6beaa | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:28:in `telemetry_duration_histogram'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:49:in `block in parse_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:90:in `block in skipping_orchestration'
a0e6beaa | /usr/share/foreman/app/models/concerns/orchestration.rb:124:in `without_orchestration'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:89:in `skipping_orchestration'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:45:in `parse_facts'
a0e6beaa | /usr/share/foreman/app/services/host_fact_importer.rb:34:in `import_facts'
a0e6beaa | /usr/share/foreman/app/models/host/base.rb:143:in `import_facts'
..
Evident cause:
rhsm_fact_parser.rb#L79 :
::Operatingsystem.find_by(os_attributes) || ::Operatingsystem.create!(os_attributes)
should be:
::Operatingsystem.find_or_create_by(os_attributes)
Updated by The Foreman Bot about 2 years ago
- Status changed from New to Ready For Testing
- Assignee set to Pavel Moravec
- Pull request https://github.com/theforeman/foreman/pull/9403 added
Updated by Pavel Moravec about 2 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset foreman|a11d509b993bef859acaeaf9f4c7764b0c90fde3.
Updated by Ewoud Kohl van Wijngaarden about 2 years ago
- Category set to Facts
- Triaged changed from No to Yes