Project

General

Profile

Bug #21569

clean backend object takes a long time to run on a foreman instance with thousands of hosts

Added by Partha Aji almost 4 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Upgrades
Target version:
Difficulty:
Triaged:
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

During the upgrade of a database with a large # of consumers the clean_backend_objects step ran for several hours and then eventually died:

foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
Upgrade Step: clean_backend_objects (this may take a while) ...

[DEBUG 2017-10-27 13:28:32 main] foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
[ INFO 2017-10-27 13:28:32 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-10-27 18:25:40 main] rake aborted!
[DEBUG 2017-10-27 18:25:40 main] Errno::ECONNRESET: Connection reset by peer - SSL_connect

during the execution there was a lot of load on the Foreman as we made many API calls into Candlepin to check for missing data.

We need to improve the performance of this task so it doesn't cause undue upgrade pain.


Related issues

Blocked by Runcible - Bug #21661: Make the consumer resource provide a retrieve_all Closed2017-11-14

Associated revisions

Revision a0aeddee (diff)
Added by Partha Aji almost 4 years ago

Fixes #21569 - Speed up clean backend objects

The clean backend objects rake task would loop through every host and
figure out if the candlepin and pulp ids were valid. This meant for
every host there would be 2 rest calls for this check. On machines where
the number of hosts were in the 10 s of thousands this would mean 20K
rest calls.
On top of that the final clean "orphaned" systems, i.e.'find all systems
in candlepin that are not in katello' would potentially load up or
serialize all candlepin consumers in 1 very big json (very expensive
task) to determine if there are orphaned consumers.

This commit tries to address this load by
1) Reducing the json foot print in candlepin (just get the UUIDs and
nothing else)
2) Page the calls. (Fetch 5000 uuids at a time, dont get everything)
3) Common sense arithmetic to calculate/fetch hosts that are in Katello
but not in CP/Pulp and vice versa
4) Hopefully more modular code for readability.

History

#1 Updated by Partha Aji almost 4 years ago

  • Bugzilla link set to 1507498

#2 Updated by The Foreman Bot almost 4 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/7051 added

#3 Updated by Eric Helms almost 4 years ago

  • Legacy Backlogs Release (now unused) set to 250

#4 Updated by Partha Aji almost 4 years ago

  • Related to Bug #21661: Make the consumer resource provide a retrieve_all added

#5 Updated by Partha Aji almost 4 years ago

  • Related to deleted (Bug #21661: Make the consumer resource provide a retrieve_all )

#6 Updated by Partha Aji almost 4 years ago

  • Blocked by Bug #21661: Make the consumer resource provide a retrieve_all added

#7 Updated by Partha Aji almost 4 years ago

  • % Done changed from 0 to 100
  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF