Foreman » Smart Proxy

Hello, please use our forum for support requests, I don't see a particular bug isolated yet.

Proxy DHCP module uses inotify to detect when a lease file was modified, then it loads it into memory. If you use e.g. NFS inotify will not work at all. If you have 20k records it might take a while to load and parse the whole file. This is growing file it can grow quite a bit until it's squashed. ISC always squash the file during restart. I suggest you to measure how long it takes to parse that file, it's the subnets_hosts_and_leases method. I think it would be wise to add some timing into the debug log for everybody.

Yes, proxy should correctly treat expired leases, our parser sees that and marks those as released.

In regard to discovery lease, Foreman has a special code that should treat lease as non-conflict because we assume it could have been a discovered host.

I don't know what exactly do you mean by static reservations, Foreman only performs OMAPI calls and all these entries are dynamic only. Static reservations are done via configuration edit and Foreman cannot update those, it does not simply have permissions to do that.

Thanks for the response! By 'static reservation' I actually meant 'dynamic reservation' (as opposed to just a basic lease).

I haven't been able to trigger this problem in a test lab, it only shows up in our production deployments long after all the hosts have been provisioned. I only mentioned the size of our deployment to try to capture the fact that the problem is somewhat rare, but shows up regularly once the number of managed hosts gets large enough. So the most succinct way I can describe this bug is:

1. Setup a bunch of Hosts in Foreman, including dynamic DHCP reservations for their primary and IPMI interfaces via foreman
2. Wait a few weeks...
3. Trigger a 'Rebuild Host Configuration' action on hosts until one of them fails with a DHCP conflict (conflicting reservation already exists)
4. Query the Smart Proxy API directly for the MAC or IP of the conflicting interface, observe that the smart proxy says there are no such reservations.
5. Restart the Foreman Proxy service
6. Again query the Smart Proxy API for the MAC or IP of the conflicting interface, observe that after the restart the proxy returns the reservation as expected.

So the reason I think this is a bug is that ONLY restarting the proxy service changes the behavior of the smart proxy API. However, I totally understand that this bug description isn't very actionable.

When I run into this again, I'll try touching the leases db file to force inotify to run and see if it helps. I'll also try timing the file parsing like you suggested.

Thanks again for the input. Any other suggestions for debugging further would be greatly appreciated!