Plugins » Katello

Description of problem:
When katello:clean_backend_objects rake script is running on a system with >1500 candlepin consumers at a time a consumer updates its facts, the rake script can wrongly detect a system is missing in candlepin, like:

Host 16080 hostname.example.com ff0a6edf-311c-4924-a7b9-b72707931c7b is partially missing subscription information. Un-registering

The reason is katello within ::Katello::Resources::Candlepin::Consumer::all_uuids call queries candlepin in pages:

2023-03-23 07:46:34,701 [thread=https-jsse-nio-127.0.0.1-23443-exec-9] [req=849f90be-9f62-40e9-9541-341e3055916b, org=, csid=] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/?owner=MyOrganization&include=uuid&per_page=1500&page=1
2023-03-23 07:48:50,123 [thread=https-jsse-nio-127.0.0.1-23443-exec-75] [req=a5d01c04-bf19-45e2-a42a-9b23493598c3, org=, csid=] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/?owner=MyOrganization&include=uuid&per_page=1500&page=2
2023-03-23 07:50:21,081 [thread=https-jsse-nio-127.0.0.1-23443-exec-26] [req=40f37098-fbed-40a6-972e-f70510000ced, org=, csid=391be221-c85d-4224-953b-d7df9143650b] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=PUT, uri=/candlepin/consumers/c8a22c7e-d959-42ef-b6a2-1eecbffd2459
2023-03-23 07:50:41,851 [thread=https-jsse-nio-127.0.0.1-23443-exec-20] [req=944317ac-69f3-45c5-9467-92b78734b276, org=, csid=e68d1403-cc68-4c79-a6a3-b8bd329fe9da] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=PUT, uri=/candlepin/consumers/8484b642-fbb1-43ff-800a-26e792c37dfc
2023-03-23 07:51:11,228 [thread=https-jsse-nio-127.0.0.1-23443-exec-63] [req=6b6a49be-eaff-4a16-9626-2ecc58bfd97e, org=, csid=] INFO org.candlepin.servlet.filter.logging.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/?owner=MyOrganization&include=uuid&per_page=1500&page=3

The problem occurs when a client updates consumer facts between the "get me next 1.5k consumers" requests - like the PUT requests above. This shuffles ordering of the consumers in a response, causing some UUID is skipped and some is present twice in the overall all_uuids call (we can easily demonstrate this).

Those skipped consumers are then wrongly marked as "partially missing subscription information".

Version-Release number of selected component (if applicable):
Sat 6.12.2 / any older as well

How reproducible:
very reliably

Steps to Reproduce:
1. Have a Satellite with >1.5k Content Hosts
2. invoke "foreman-rake katello:clean_backend_objects"
3. while it is running, run "subscription-manager facts --update" on a few hosts
4. check the rake script output

Actual results:
4. with some (high) probability, the rake script will wrongly detect a missing candlepin consumer

Expected results:
4. no such false alarms

Additional info:
We can easily demonstrate the "PUTing consumer facts shuffles consumers ordeing" on >10 hosts system as well. Just enable candlepin API from cmdline per https://access.redhat.com/solutions/2955931 , and replace "get me clients in 1.5k batches" requests by ".. in batches of, say, 7"

for i in $(seq 1 4); do curl -sk -H "Content-Type:application/json" -u admin:admin "https://localhost:23443/candlepin/consumers/?owner=YOUR_ORGANIZATION&include=uuid&per_page=7&page=${i}" | json_reformat; sleep 1; echo "uuid"; done | grep uuid

Normally, it will print something like:

"uuid": "08513930-d0f7-4882-8129-77673dc721c2" 
        "uuid": "cbc30fb8-9939-467d-9273-052eaf791eb4" 
        "uuid": "7ba62210-3c30-4a4c-ae0a-fb31df3bc33d" 
        "uuid": "f83e11cc-c0c3-479d-83ae-573fad2090bc" 
        "uuid": "9c8bbc55-ac77-4e2c-853d-a2a6fc24862a" 
        "uuid": "f72296a5-0b56-48b3-8712-f2ff4813e778" 
        "uuid": "762a8c10-2793-489e-a53f-a3ffb2e3acb0" 
uuid
        "uuid": "aac2882d-0bf6-4336-9d41-b30c94c452de" 
        "uuid": "60787875-b416-4aaf-bf1e-bd365639f4a8" 
        "uuid": "adb8a530-03a9-4c90-95b6-845b742adeb4" 
        "uuid": "f8bffcba-cd62-4a0f-b86d-f02a2f27e14f" 
        "uuid": "b8787ab1-58aa-4a14-b1f6-68eae436954e" 
        "uuid": "8f7d4e4e-6b7f-4498-9019-30dad9b8a62a" 
        "uuid": "f8677b63-d1b3-4495-a86e-00d415995bfd" 
uuid
        "uuid": "62d3e041-00c2-4adf-8135-055815954203" 
        "uuid": "f72296a5-0b56-48b3-8712-f2ff4813e778" 
        "uuid": "0e7893a7-fc9c-4b5b-97d7-b44c8b120071" 
        "uuid": "8f7d4e4e-6b7f-4498-9019-30dad9b8a62a" 
        "uuid": "663af0ad-4206-4294-a947-83fd5cfa612d" 
        "uuid": "adb8a530-03a9-4c90-95b6-845b742adeb4" 
        "uuid": "f8677b63-d1b3-4495-a86e-00d415995bfd" 
uuid
        "uuid": "7ba62210-3c30-4a4c-ae0a-fb31df3bc33d" 
        "uuid": "b8787ab1-58aa-4a14-b1f6-68eae436954e" 
        "uuid": "aac2882d-0bf6-4336-9d41-b30c94c452de" 
        "uuid": "82480bc1-3737-429c-8947-d1de745e9e27" 
        "uuid": "762a8c10-2793-489e-a53f-a3ffb2e3acb0" 
uuid

Every time the same sequence. Now, during executing the script (play with the sleep time there), run "subscription-manager facts --update" on a client. The output will be e.g.:
"uuid": "08513930-d0f7-4882-8129-77673dc721c2"
"uuid": "cbc30fb8-9939-467d-9273-052eaf791eb4"
"uuid": "7ba62210-3c30-4a4c-ae0a-fb31df3bc33d"
"uuid": "f83e11cc-c0c3-479d-83ae-573fad2090bc"
"uuid": "9c8bbc55-ac77-4e2c-853d-a2a6fc24862a"
"uuid": "f72296a5-0b56-48b3-8712-f2ff4813e778"
"uuid": "762a8c10-2793-489e-a53f-a3ffb2e3acb0"
uuid
"uuid": "aac2882d-0bf6-4336-9d41-b30c94c452de"
"uuid": "60787875-b416-4aaf-bf1e-bd365639f4a8"
"uuid": "adb8a530-03a9-4c90-95b6-845b742adeb4"
"uuid": "f8bffcba-cd62-4a0f-b86d-f02a2f27e14f"
"uuid": "b8787ab1-58aa-4a14-b1f6-68eae436954e"
"uuid": "8f7d4e4e-6b7f-4498-9019-30dad9b8a62a"
"uuid": "f8677b63-d1b3-4495-a86e-00d415995bfd"
uuid
"uuid": "f83e11cc-c0c3-479d-83ae-573fad2090bc"
"uuid": "f72296a5-0b56-48b3-8712-f2ff4813e778"
"uuid": "0e7893a7-fc9c-4b5b-97d7-b44c8b120071"
"uuid": "8f7d4e4e-6b7f-4498-9019-30dad9b8a62a"
"uuid": "663af0ad-4206-4294-a947-83fd5cfa612d"
"uuid": "adb8a530-03a9-4c90-95b6-845b742adeb4"
"uuid": "f8677b63-d1b3-4495-a86e-00d415995bfd"
uuid
"uuid": "7ba62210-3c30-4a4c-ae0a-fb31df3bc33d"
"uuid": "b8787ab1-58aa-4a14-b1f6-68eae436954e"
"uuid": "aac2882d-0bf6-4336-9d41-b30c94c452de"
"uuid": "82480bc1-3737-429c-8947-d1de745e9e27"
"uuid": "762a8c10-2793-489e-a53f-a3ffb2e3acb0"
uuid

See the 62d3e041-.. is replaced by another f83e11cc-.., so the rake script won't get UUID 62d3e041 any more, during this run.