Bug #31829
openThe proxies cached list of DHCP reservations gets out of sync with the DHCP server
Description
We have a foreman deployment that currently manages 20k physical hosts spread across a few dozen data centers (smart proxies). We often run into DHCP conflict errors when rebuilding a host's configuration (e.g. Building a host). The master asks the proxy if there is already a DHCP entry for this host, the proxy responds that their isn't, so the master tries to create one. The proxy then sets up the OMAPI calls to create the reservation, but DHCPd rejects the request because the DHCP record does actually exist. If I call the proxy's DHCP endpoint and ask it for that reservation, it returns None. If I then restart the foreman-proxy service and again ask the proxy to get the same reservation, it's returned as expected.
So it seems that the proxy's cached view of the DHCP leases/reservations gets out of sync with reality until we reload the service.
We're using stock ISC DHCP with proxy version 1.21.3.
It's also worth noting that we're using the same DHCP pool for both static and dynamic leases. This was a mistake that we've fixed in the next generation of our deployment, but for the most part sharing the pool between static and dynamic has worked okay. After looking through the proxy code, I suspect this problem is being caused by the fact that the dynamic and static ranges are the same. Our provisioning flow goes like this:
1. New host PXEs and gets a dynamic IP and boots into Discovery
2. We convert the Discovered host into Provisioned, reserving the dynamic IP the host got when it first booted.
So because of this, there is a time period when both static and dynamic leases exist for the same host. I suspect that when the dynamic lease does finally expire, the proxy's DHCP observer treats this as the reservation being removed, so it removes the reservation from its internal cache. This is just a guess and could be a total red herring, but I haven't been able to find reports of anyone else running into this conflict issue.