Foreman » Smart Proxy

I don't think filtering on networks would help. We need to parse individual records before making a decision whether to keep or discard them, and it looks to me that the speed of parsing is the real issue here.

The stack trace from when the USR1 signal arrived is quite telling:

"/usr/share/foreman-proxy/lib/proxy/memory_store.rb:50:in `block in get_all_values'" 
"/usr/share/foreman-proxy/lib/proxy/memory_store.rb:49:in `each'" 
"/usr/share/foreman-proxy/lib/proxy/memory_store.rb:49:in `inject'" 
"/usr/share/foreman-proxy/lib/proxy/memory_store.rb:49:in `get_all_values'" 
"/usr/share/foreman-proxy/lib/proxy/memory_store.rb:21:in `values'" 
"/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:48:in `block in find_subnet'" 
"/usr/share/ruby/monitor.rb:211:in `mon_synchronize'" 
"/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:43:in `find_subnet'" 
"/usr/share/foreman-proxy/modules/dhcp_isc/isc_file_parser.rb:38:in `block in parse_config_and_leases_for_records'" 
"/usr/share/foreman-proxy/modules/dhcp_isc/isc_file_parser.rb:24:in `scan'" 
"/usr/share/foreman-proxy/modules/dhcp_isc/isc_file_parser.rb:24:in `parse_config_and_leases_for_records'" 
"/usr/share/foreman-proxy/modules/dhcp_isc/leases_file.rb:13:in `hosts_and_leases'" 

The find_subnet code for non-subnet addresses is known to be slow (it has a comment stating this), it isn't written to be terribly efficient.

It's unclear to me why there aren't any log messages between 2016-11-15T23:57:52.075717 and 2016-11-16T03:25:01.797869, unless the code is in an infinite loop or deadlocked somehow (not that requests should be accepted at this point), or if it is parsing a lot of data that isn't causing any reservations to be added. Has the log been snipped at this point too, but without a comment? (It was above.)

Regarding the USR1 signal (from log rotation), this could be addressed separately to install signal handlers earlier, and also perhaps to change the dhcp_isc module to parse the files after startup (similarly to the Puppet v3 module).

Konstantin, I suspect that the pauses are caused by large numbers of host { } entries being processed. Could you indicate over the 16k subnets roughly how many host entries you'd have in total in the config and leases files?

We should be seeing debug output if leases/reservations are being added/removed, I'd say the thread is blocked/hanging. I'm not going to insist it's impossible for a deadlock to be there (I will check though), but it doesn't appear to be likely -- service initialization/initial config/lease file parsing occurs in a single thread/under a single lock. I wonder if at this point we are trashing the disk? I don't have any data, but the lease file that is kept in memory (and copied around) and all host/subnet records might trigger heavy swapping.

I'd like to see the whole snipped bit -- the time it took to parse subnet and reservation records does not add up to to 4 hours (the initialization should be done in about 80 seconds, still a lot, but a problem of entirely different magnitude) if projected to all possible records (I assumed 16523 networks and 1444018 ips in those networks, based on the breakdown of networks above).

Agree re: find_subnet and handling of SIGUSR1.

Ideally, I'd like to get my hands on the config and leases files, as it would make troubleshooting this easier.

Dmitri Dolguikh wrote:

We should be seeing debug output if leases/reservations are being added/removed, I'd say the thread is blocked/hanging.

I think the logging of new leases/reservations will only happen after parse_config_and_leases_for_records has parsed the file and returned a list of reservations to add. If this method was taking hours, I wouldn't expect to see any logging until it completed (the ISC state change observer calls the parser method and adds the return value to the subnet service in bulk).

I'd like to see the whole snipped bit -- the time it took to parse subnet and reservation records does not add up to to 4 hours (the initialization should be done in about 80 seconds, still a lot, but a problem of entirely different magnitude) if projected to all possible records (I assumed 16523 networks and 1444018 ips in those networks, based on the breakdown of networks above).

I think it could take this long for 16k networks. I've been benchmarking SubnetService#add_subnet and the performance degradation really does appear to be quite severe.

Running add_subnet for (number of subnets):

      add_subnet (1)     33.404k (±19.7%) i/s -    300.425k in   9.791448s
      add_subnet (5)      3.005k (±12.1%) i/s -     29.171k in   9.974092s
     add_subnet (50)     46.702  (± 4.3%) i/s -    466.000  in  10.007726s
    add_subnet (500)      0.496  (± 0.0%) i/s -      5.000  in  10.079957s

Edit: benchmark source code: https://github.com/domcleal/smart-proxy/tree/17373-subnetsvc-benchmark

The last result shows it takes half a second to call add_subnet only 500 times. The bulk of the time is taken up in find_subnet to check for overlapping subnets. I'm running it for 16,500 subnets, but it's still going after eleven minutes. I'm unsure how far it's got.

Compounding this problem, adding host { } reservations is as bad because each calls find_subnet to find the containing subnet. If the number of reservations is a few orders of magnitude more than the number of subnets, it's clear that the total wall time will be quite incredible.

To answer some of your questions, guys:

1. This particular instance has 30K host records and ~82K lease records in its dhcpd.leases.

2. I did not remove any log entries other than those <snip> sections, but I made sure I did not remove any relevant data.

3. I can't share my exact dhcpd.conf or dhcpd.leases, but my original ticket has a large-dataset files attached (still much smaller than what I have in production now - 1580 subnets and ~41K leases). I just re-used that set on 1.13.1 and even though smart-proxy started, it still took 46 minutes to load on my VM (16 CPUs & 64GB RAM). A complete unedited log available here - https://gist.github.com/korekhov/4e2214df78796e6d7a7dd27162ac6074. There are also gaps in the log, most noticeable ones:

D, [2016-11-17T12:01:34.153661 #10957] DEBUG -- : Added a subnet: 43.145.54.0
D, [2016-11-17T12:01:34.203004 #10957] DEBUG -- : Added a subnet: 43.145.50.0
D, [2016-11-17T12:08:06.231786 #10957] DEBUG -- : Added a reservation: 43.168.225.222:11:22:33:72:20:a9:7b02c-719435-sp
D, [2016-11-17T12:08:06.232501 #10957] DEBUG -- : Added a reservation: 43.168.227.205:11:22:33:a9:61:09:7b02c-715857-sp

and even larger here

D, [2016-11-17T12:08:06.592150 #10957] DEBUG -- : Added a reservation: 43.212.113.253:11:22:33:09:ed:8f:8b03c-0560-sp
D, [2016-11-17T12:08:06.592197 #10957] DEBUG -- : Added a reservation: 43.158.157.235:11:22:33:03:40:37:1b01c-c4f7-sp
D, [2016-11-17T12:47:30.355166 #10957] DEBUG -- : Added a reservation: 169.254.1.203:11:22:33:39:ac:40:blah
D, [2016-11-17T12:47:30.355593 #10957] DEBUG -- : Deleted a reservation: 169.254.1.203:11:22:33:39:ac:40:blah

I'm sure you can use the same dataset to do your testing.

4. I removed those /30s and /31s from dhcpd.conf for my another test on the same VM, and results are much better - smart-proxy loaded in a bit less than 3 hours (compared to 8 hours followed by the failure last night). But I still see a gap of no log messages for a long time:

D, [2016-11-17T12:40:56.137785 #20231] DEBUG -- : Added a reservation: 10.90.66.243:3c:4a:92:f4:96:bc:miloproxy03
D, [2016-11-17T12:40:56.137837 #20231] DEBUG -- : Added a reservation: 10.90.174.58:00:50:56:99:00:05:cache009
D, [2016-11-17T15:47:06.130065 #20231] DEBUG -- : Added a reservation: 10.53.117.237:84:8f:69:ff:59:2c:4b03c-ffb6-sp
D, [2016-11-17T15:47:06.130472 #20231] DEBUG -- : Added a reservation: 10.111.135.211:00:8c:fa:0a:09:2f:4b05c-eb21-sp

A complete and unedited log is here - https://gist.github.com/korekhov/86c2262186c92e20bd3054ff115e2914

Please let me know if I can provide more information to help.

Thanks!

On ruby 2.3.1 with the changes in http://projects.theforeman.org/issues/17387 applied, smart-proxy starts in less than 9 seconds (although there's that weird pause of about 8 seconds at the very end) when the config and leases from http://projects.theforeman.org/issues/2687 are used:

```
D, [2016-11-18T09:43:17.217051 ] DEBUG -- : Added a subnet: 169.254.1.0
D, [2016-11-18T09:43:17.217165 ] DEBUG -- : Added a subnet: 43.240.109.0
D, [2016-11-18T09:43:17.217197 ] DEBUG -- : Added a subnet: 43.183.159.0
<snip>
D, [2016-11-18T09:43:17.907471 ] DEBUG -- : Added a reservation: 43.212.113.253:11:22:33:09:ed:8f:8b03c-0560-sp
D, [2016-11-18T09:43:17.907499 ] DEBUG -- : Added a reservation: 43.158.157.235:11:22:33:03:40:37:1b01c-c4f7-sp
D, [2016-11-18T09:43:25.645773 ] DEBUG -- : Added a reservation: 169.254.1.203:11:22:33:39:ac:40:blah
D, [2016-11-18T09:43:25.645855 ] DEBUG -- : Deleted a reservation: 169.254.1.203:11:22:33:39:ac:40:blah
I, [2016-11-18T09:43:25.717711 ] INFO -- : Successfully initialized 'dhcp_isc'
```

It looks like the pause is due to ruby VM garbage collector which runs several major and minor garbage collections in those 8 seconds. I'll play with vm settings to see if I can reduce the amount of time spent in GC.

Confirmed - the fix from https://github.com/theforeman/smart-proxy/pull/476 is amazing!

It only takes 30 sec to load ~15K networks, ~80K leases and ~30K host reservations - down from ~8 hours (and still failed).

Thank you very much, Dmitri!

There are only a couple of things left - regular expressions in white/blacklist to include/exclude networks. In my particular situation, only ~3000 networks out of that ~15K is what Foreman should be managing. Should this be a new feature request?

The second thing is a global search for a MAC, IP or hostname - I went through current DHCP APIs and I don't see a way to do so. I previously had http://projects.theforeman.org/issues/2482 open for that feature, but it was closed due to inactivity. What do you think?

Thanks again!

One more thing I just realized - none of the leases were actually loaded, only host reservations. Is this by design?

This is not an issue in 1.11.x and prior as far as I can tell, but 1.13.1 has it for sure. Don't have 1.12 anymore, so can't check.

One more thing I just realized - none of the leases were actually loaded, only host reservations. Is this by design?

Is this with the dataset in the original ticket? Lease record won't get loaded now if it's expired (based on 'ends' attribute) and its 'next_state' is 'free'. EDIT: this code is present in 1.11 and hasn't changed since.

Konstantin Orekhov wrote:

There are only a couple of things left - regular expressions in white/blacklist to include/exclude networks. In my particular situation, only ~3000 networks out of that ~15K is what Foreman should be managing. Should this be a new feature request?

Yes, please.

The second thing is a global search for a MAC, IP or hostname - I went through current DHCP APIs and I don't see a way to do so. I previously had http://projects.theforeman.org/issues/2482 open for that feature, but it was closed due to inactivity. What do you think?

The ticket's still open, only the PR was closed. I'd suggest opening a new PR with the requested changes.

There are only a couple of things left - regular expressions in white/blacklist to include/exclude networks. In my particular situation, only ~3000 networks out of that ~15K is what Foreman should be managing. Should this be a new feature request?

Is this is a performance or a security concern? If it's the former, filtering out networks won't affect loading/reloading performance much, as records need to be parsed first, which is what bulk of the time is being spent on.

Is this is a performance or a security concern? If it's the former, filtering out networks won't affect loading/reloading performance much, as records need to be parsed first, which is what bulk of the time is being spent on.

It is mostly about keeping Foreman itself in a cleaner state - I'd like to have an ability to filter out subnets that I know Foreman would never manage so they don't even show up in UI/API and clog things up for me.

It is mostly about keeping Foreman itself in a cleaner state - I'd like to have an ability to filter out subnets that I know Foreman would never manage so they don't even show up in UI/API and clog things up for me.

I wonder if this would be better implemented on Foreman side, as it would be easier to manage the whitelist from there: no need to make a trip to smart-proxy, manually update the config and restart smart-proxy.

I wonder if this would be better implemented on Foreman side, as it would be easier to manage the whitelist from there: no need to make a trip to smart-proxy, manually update the config and restart smart-proxy.

Yep, that would be much better. It is just a question of how long a new feature takes vs. augmenting (semi)-existing one.

Is this with the dataset in the original ticket? Lease record won't get loaded now if it's expired (based on 'ends' attribute) and its 'next_state' is 'free'. EDIT: this code is present in 1.11 and hasn't changed since.

Yes, you're right. This wasn't the original lease file, but still it had all the leases expired.

Going back to performance, HA and such.
I was asking that question in my original issue, but never got the answer - why smart-proxy needs to read leases file directly? If OMAPI is used for writes, why not to use it for IP/MAC/hostname lookups as well?
This way you don't even have to read leases file at all. This in its turn removes the need for inotify observers, etc., NFS can be used again (for config files), DHCP server itself could be located on different machine and so forth.

My worry is that in my large installation where leases file changes pretty much every second, smart-proxy just won't be able to keep up with that if needs to spend 30 sec to re-read leases file on file change. if OMAPI is used instead, we have much better chance.

The second thing is a global search for a MAC, IP or hostname - I went through current DHCP APIs and I don't see a way to do so. I previously had http://projects.theforeman.org/issues/2482 open for that feature, but it was closed due to inactivity. What do you think?

The ticket's still open, only the PR was closed. I'd suggest opening a new PR with the requested changes.

Before I submit a PR, would you please review https://github.com/korekhov/smart-proxy/commit/ce25338033c4d5bb80c57d992ed8db9552925d69 ?
That certainly works for me, I'm just not sure if it is most efficient way - I just followed find_by_ examples and created new definitions and memory storage.

Thanks!

I was asking that question in my original issue, but never got the answer - why smart-proxy needs to read leases file directly? If OMAPI is used for writes, why not to use it for IP/MAC/hostname lookups as well?

Omapi is limited to lookup/create/update/delete for leases, reservations, and a couple more objects. We need to parse config and leases files to support operations unavailable to us via omapi, such as: anything subnet-related, list multiple leases or reservations, detect changes in leases and reservations, or detect free ips. Once we keep an internal copy of dhcpd state, it's faster to make queries against it, rather than with omapi. This is the reason why omapi is used for create/update/delete operations only on leases and reservations.

Before I submit a PR, would you please review https://github.com/korekhov/smart-proxy/commit/ce25338033c4d5bb80c57d992ed8db9552925d69

Please submit a PR, as I'd rather keep development-related conversations and feedback on github. I need to think if memory storage can be re-written to support the queries you are proposing (to minimize the number of mapping we keep, as adding keys to each isn't free time-wise).

Omapi is limited to lookup/create/update/delete for leases, reservations, and a couple more objects. We need to parse config and leases files to support operations unavailable to us via omapi, such as: anything subnet-related, list multiple leases or reservations, detect changes in leases and reservations, or detect free ips. Once we keep an internal copy of dhcpd state, it's faster to make queries against it, rather than with omapi. This is the reason why omapi is used for create/update/delete operations only on leases and reservations.

Well, here's a look at the same points from a different angle:

- Parsing leases file does not provide network info either - all of that comes from smart-proxy parsing of config files and marrying that to parsed data from leases file.

- Yes, OMAPI does not provide network details, but you still could do the IP/network mapping using above parsed config files just like you do today - on a start up parse config files, then on DHCP API request talk to OMAPI. If an OMAPI query returns a good result to you, combine it with your in-memory parsed network data and return that to the user.

- Why do you really need to detect the changes to leases? Not all of those changes/updates to lease file is something that Foreman or DHCP API user have to know about. Certainly not the moment the change happens, so really the need to keep latest copy of the parsed leases in memory is just a duplication of data and efforts to me - OMAPI can give you the answer on demand as you don't really need to know about each of the change in DHCP data, you just need to be able to query that and OMAPI is quite good at that. And with the most important benefit - you don't need to do anything to maintain it.

- I can't speak for other environments, but duplicate records are rather rare condition in my env. And yes, OMAPI can't help with that. But does this condition outweighs inability to scale for the whole product? Are there any other ways to deal with that?

- Detection of free IP - again, if you have a starting IP, you can always query OMAPI and build out your own map of free/used IPs per subnet. It may be a second or so longer (to be tested), but again, scalability is not sacrificed. You could also keep OMAPI connection open (with some idle timeout, of course) to speed things up for very busy environments.

- Having an extra cache layer between Foreman (or an API user) and OMAPI would not scale for large deployments - as I mentioned earlier, in my DHCP environment, leases file updated several times a second, but even with all of the great improvements you recently did, it takes 30 sec just to re-read that file and by the time it is done, leases file is new ahead so smart-proxy would have to reload it again. That essentially blocks DHCP API from ever having a chance to server user requests. This is my biggest worry at this point and I'll do actual experiment for this next week and will provide more data, if you're interested.

So, to summarize my thoughts on this topic - IMHO, in order for DHCP smart-proxy to get highly-scalable, highly-available yet stay fast enough regardless of the underlying FS used (NFS, local or whatever), smart-proxy should not try to do the same work that OMAPI already does and use OMAPI services in full instead - for all lookups and writes. Extra data not supplied by OMAPI can be added by smart-proxy by parsing config files and joining that with OMAPI results. Smaller environments would not notice any difference in such implementation, but smart-proxy will get an ability to serve large ones. Something that is regrettably been missing for a long time.

Thanks!

Parsing leases file does not provide network info either - all of that comes from smart-proxy parsing of config files and marrying that to parsed data from leases file.

Pretty much. Unfortunately, we cannot rely on omapi for queries that return contents of a given network: retrieving all records for a subnet by iterating over address space one address at a time would be slow even if used native omapi interface, esp. on large subnets. Once we have an internal representation of dhcpd state, it's faster to use that for pretty much all calls.

Well, the underlying issue is not resolved, though. It is unfortunate that you think that it is resolved.

As I was anticipating, in smaller environments where changes to dhcpd.leases are infrequent (~1-5 per minute), reloading the changes only works pretty well:

D, [2016-11-30T18:28:00.901659 #17578] DEBUG -- : caught :modify event on /dhcp/conf/dhcpd.leases.
D, [2016-11-30T18:28:11.120706 #17578] DEBUG -- : caught :modify event on /dhcp/conf/dhcpd.leases.
D, [2016-11-30T18:28:38.948416 #17578] DEBUG -- : caught :modify event on /dhcp/conf/dhcpd.leases.
D, [2016-11-30T18:28:43.787301 #17578] DEBUG -- : caught :modify event on /dhcp/conf/dhcpd.leases.

But as soon as the rate increases to 10-15 per second, smart-proxy just can't keep up with that and actually start snow-balling as inotify events are coming in faster than SmP can process them.

I'm experimenting more with this, but so far things don't look too promising for large networks.

Konstantin, out of curiousity: I am working (#13806) on dnsmasq smart-proxy provider that will work fast. It's a small and efficient (yet pretty powerful) DHCP/DNS/TFTP server, the integration will be much cleaner in this case (drop a file, SIGHUP). My goal is to provide an alternative for those who are not satisfied with ISC DHCP implementation. Would be migration to dnsmasq an option for your deployment?

I reopened the issue and also created a ticket dedicated to inotify watcher not being able to keep up with the events (http://projects.theforeman.org/issues/17554). Please use ticket #17554 for comments or feedback on inotify problem. Note there's now a PR open that addresses the problem.

Konstantin, out of curiousity: I am working (#13806) on dnsmasq smart-proxy provider that will work fast. It's a small and efficient (yet pretty powerful) DHCP/DNS/TFTP server, the integration will be much cleaner in this case (drop a file, SIGHUP). My goal is to provide an alternative for those who are not satisfied with ISC DHCP implementation. Would be migration to dnsmasq an option for your deployment?

Hi, Lukas!

dnsmasq is not designed for large networks in a first place. Even according to their authors [1], it is for small networks. But my real problem with ISC is a lack of built-in HA (one can have a couple of instances to back each other up, but it is clumsy and rather unreliable, IMHO) and a lack of proper API. There's an OMAPI, but as we all know, it only allows for records and leases manipulation, but not networks, options, etc.
I do not believe dnsmasq even has something like OMAPI thus leaving the only way to manage it - file(s) manipulation, pretty much just like with ISC config files. That brings in a problem of synchronizing of said files across members of HA cluster, etc. Basically, exactly the same issues I have with ISC now. So, to me, dnsmasq is not attractive at all as I won't be solving my fundamental problems with that and possibly getting another scalability problem later on.

Now, if you wanted to provide a very valuable thing that would truly scale, I'd suggest you take a look and possible write a provider for KEA, next-gen ISC DHCP implementation [2]. The beauty of KEA is in its DB backend (not just for leases, but for all configurations and such) and proper API to manipulate both records and configuration. Obvious benefits:
- you can have several members of an HA cluster running in different places of your environment as they share the same backend DB (or better yet you can have multi-node replicated DB cluster for that)
- no file manipulation whatsoever, so no restarts, node access requirements, etc.
- true SOA that allows remote configuration - add/remove networks, scopes, records, options, etc.
- unparalleled scalability

What is missing though is RESTful API front-end for KEA API as it is currently implemented as UNIX sockets [3] (it is rather limited at this point, but I know that they are working on more APIs to be exposed and I can get you in touch with KEA leads). This is where Foreman Smart-Proxy fits in just perfectly - it could be installed on each of KEA nodes, implement a RESTful HTTP/S endpoint for user/Foreman to send requests to, translate that into KEA API, query and send the response back. No caching, no config file reads, only on-demand requests (just like a proper proxy should do).

I can create a feature request for this if you're interested. I know I am :)

Going back to current ISC DHCP smart-proxy provider/module for a second as KEA will take a while to implement no matter how we slice it or dice it.
If the above approach (no-caching, on-demand queries only) make sense to you, I strongly believe it could be implemented for OMAPI as well. The biggest problem with current implementation is attempt to cache just everything and that does not scale very well, as practice show. However, if one digs a bit deeper into why caching is needed, one may realize that it is not - Foreman queries are on-demand and are very infrequent as compared to actual DHCP operations. An attempt to keep the cache up-to-date even if there are no requests for that is just chasing your own tail. I'll definitely test Dmitri's latest PR for that, but the approach is far from ideal as makes it very hard for implement HA on top of that (especially now when NFS support is gone).

[1] - http://www.thekelleys.org.uk/dnsmasq/doc.html
[2] - http://kea.isc.org/wiki
[3] - http://kea.isc.org/docs/kea-guide.html#ctrl-channel

Kea has no management api whatsoever (it's a step backwards compared to dhcpd in this regard), which makes it pretty challenging to interface with it.

If the above approach (no-caching, on-demand queries only) make sense to you, I strongly believe it could be implemented for OMAPI as well.

There is no way we can drop local state without reducing functionality/scope of smart proxy dhcp api.

Kea has no management api whatsoever (it's a step backwards compared to dhcpd in this regard), which makes it pretty challenging to interface with it.

Well, yes, currently it is pretty much non-existent, but if it were, would you be interested in picking this up as alternate DHCP provider? I have been out of touch with them last 2-3 months, but can certainly get the latest if there's an interest.

BTW, what is fully supported today is record manipulation directly in a DB - http://kea.isc.org/wiki/HostReservationsHowTo.

There is no way we can drop local state without reducing functionality/scope of smart proxy dhcp api.

Caching config file is OK as that does not happen as often as updating the leases and the latter is the killer really.
So, if we step back and review the requirements for caching leases, what would that be? You mentioned keeping track of free IPs in a range. Anything else?

The reason I'm asking is this - even ISC DHCP itself does not keep a state of a complete scope for a subnet and free IPs in it but only does it on a request from a client. So, maybe this requirement in Foreman can be reviewed or some other solution found? Clearly, DHCP caching in smart-proxy as an underlying service is a good and fast thing, but that imposes the scalability limitations for whole Foreman and other user applications in larger environments.

Well, yes, currently it is pretty much non-existent, but if it were, would you be interested in picking this up as alternate DHCP provider?

Absolutely. The richer/better the api is the easier the task on our end becomes (see MS dhcp provider for an example.).

BTW, what is fully supported today is record manipulation directly in a DB - http://kea.isc.org/wiki/HostReservationsHowTo.

It's something, and in a way better than parsing leases file (oth it's worse -- db schemas change more often, which makes such an interface brittle). I don't remember finding db schema docs though, and personally would be hesitant to jump in before the project publishes this information.

So, if we step back and review the requirements for caching leases, what would that be? You mentioned keeping track of free IPs in a range. Anything else?

GET /dhcp/:subnet_ip call (see https://github.com/theforeman/smart-proxy/blob/develop/modules/dhcp/dhcp_api.rb#L40). There's been talk of adding subnet usage api endpoint too, which would require information about the whole subnet.

Well, yes, currently it is pretty much non-existent, but if it were, would you be interested in picking this up as alternate DHCP provider?

Absolutely. The richer/better the api is the easier the task on our end becomes (see MS dhcp provider for an example.).

BTW, what is fully supported today is record manipulation directly in a DB - http://kea.isc.org/wiki/HostReservationsHowTo.

It's something, and in a way better than parsing leases file (oth it's worse -- db schemas change more often, which makes such an interface brittle). I don't remember finding db schema docs though, and personally would be hesitant to jump in before the project publishes this information.

Let me check on both of these items.

So, if we step back and review the requirements for caching leases, what would that be? You mentioned keeping track of free IPs in a range. Anything else?

GET /dhcp/:subnet_ip call (see https://github.com/theforeman/smart-proxy/blob/develop/modules/dhcp/dhcp_api.rb#L40). There's been talk of adding subnet usage api endpoint too, which would require information about the whole subnet.

Well, if it is OK with Foreman developers and community to sacrifice the performance and scope of Foreman as product and position it only for small deployments, then I guess I have to write a no-caching module in-house.

Before I do that though, I'd like to understand if this is an official Foreman position on this or your personal one?

Well, if it is OK with Foreman developers and community to sacrifice the performance and scope of Foreman as product and position it only for small deployments, then I guess I have to write a no-caching module in-house.

It is not what I've been talking about at all. I've been trying to point out that it is not possible to not drop functionality from dhcp api and not keep local state in isc dhcpd provider, for performance reasons. There is a way to handle HA setups that involves state synchronization between smart proxy instances, but no one is working on this atm.

Before I do that though, I'd like to understand if this is an official Foreman position on this or your personal one?

Feel free to ask the question on the mailing list.

It is not what I've been talking about at all. I've been trying to point out that it is not possible to not drop functionality from dhcp api and not keep local state in isc dhcpd provider, for performance reasons. There is a way to handle HA setups that involves state synchronization between smart proxy instances, but no one is working on this atm.

No, no, I completely understand where you coming from. I was trying to start a rethinking process that would potentially have solution that is both HA-capable, scalable and still supports all the features required.

Created http://projects.theforeman.org/issues/17615 for KEA DHCP provider feature.
Let's move all KEA-related conversations over there.