clean backend object takes a long time to run on a foreman instance with thousands of hosts
During the upgrade of a database with a large # of consumers the clean_backend_objects step ran for several hours and then eventually died:
foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-10-27 13:28:32 main] foreman-rake katello:correct_puppet_environments COMMIT=true finished successfully!
[ INFO 2017-10-27 13:28:32 main] Upgrade Step: clean_backend_objects (this may take a while) ...
[DEBUG 2017-10-27 18:25:40 main] rake aborted!
[DEBUG 2017-10-27 18:25:40 main] Errno::ECONNRESET: Connection reset by peer - SSL_connect
during the execution there was a lot of load on the Foreman as we made many API calls into Candlepin to check for missing data.
We need to improve the performance of this task so it doesn't cause undue upgrade pain.
Fixes #21569 - Speed up clean backend objects
The clean backend objects rake task would loop through every host and
figure out if the candlepin and pulp ids were valid. This meant for
every host there would be 2 rest calls for this check. On machines where
the number of hosts were in the 10 s of thousands this would mean 20K
On top of that the final clean "orphaned" systems, i.e.'find all systems
in candlepin that are not in katello' would potentially load up or
serialize all candlepin consumers in 1 very big json (very expensive
task) to determine if there are orphaned consumers.
This commit tries to address this load by
1) Reducing the json foot print in candlepin (just get the UUIDs and
2) Page the calls. (Fetch 5000 uuids at a time, dont get everything)
3) Common sense arithmetic to calculate/fetch hosts that are in Katello
but not in CP/Pulp and vice versa
4) Hopefully more modular code for readability.