Foreman » Smart Proxy

Yes, this is a single dataset from one of the datacenters, and we have several of those. We do have our reasons not to split that into smaller pieces and on its own is a separate topic. At this point my goal is make Foreman smart-proxy a lot more efficient to be able to cope with any size of the dataset thrown at it.

I'm checking internally if I can share that with you under NDA we have with RedHat, but it may be faster to find a way to generate that set on your own. I'll take a look into masking our data with some fakes for you.

In terms of the cache. Not sure if it is good idea in places where direct interaction with omshell is possible. For example, for record validation/global search based on hostname, IP or MAC, the following code works like a charm, no cache/map is needed:

obj: <null>
> new host
obj: host
> set name = "some" 
obj: host
name = "some" 
> open
obj: host
name = "some" 
ip-address = 0a:6d:56:c7
hardware-address = ff:ff:ff:ff:ff:ff
hardware-type = 00:00:00:01
>

Or during host create operation, why not to solely rely on omshell to do the validation like this?

>
obj: <null>
> new host
obj: host
> set hardware-address = ff:ff:ff:ff:ff:ff
obj: host
hardware-address = ff:ff:ff:ff:ff:ff
> set ip-address = 1.1.1.1
obj: host
hardware-address = ff:ff:ff:ff:ff:ff
ip-address = 01:01:01:01
> set name = "blah" 
obj: host
hardware-address = ff:ff:ff:ff:ff:ff
ip-address = 01:01:01:01
name = "blah" 
> create
can't open object: key conflict
obj: host
hardware-address = ff:ff:ff:ff:ff:ff
ip-address = 01:01:01:01
name = "blah" 
>

One thing is still unclear to me. What is the significance of the subnet information for DHCP record in Foreman? Why user has to provide a subnet and IP for a new record in its API call, while in the background the subnet info is being dropped and not used by omshell at all? Is the subnet info returned by the sample call below is used by Foreman anywhere later on?

# curl -k -X GET https://dhcp.vip:8443/dhcp/10.109.86.0/ff:ff:ff:ff:ff:ff
{"subnet":"10.109.86.0/255.255.254.0","ip":"10.109.86.199","hostname":"some","mac":"ff:ff:ff:ff:ff:ff"}

If there's no special meaning for this and no other flows currently rely on subnet information, I'd propose to remove that as it'll greatly simplify the smart-proxy operations. In the example above, the following happens in the background (from my understanding of the code, which could be wrong/incomplete):

1. dhcpd.conf is read for subnet info
2. dhcpd.lease is read for leases info
3. a map of subnets is created and populated with corresponding leases/reservations
4. a search of specified MAC is performed in that map
5. found record returned

If there is no meaning to subnet info returned with found record, then all of these 5 steps could be replaced with just one (my first example of omshell call at the beginning), which takes a lot less time.

Does this make sense to you?

Any thoughts on this, folks? I was hoping providing a sample with large dataset would trigger some updates, but there's nothing in 19 days now...

Thanks!

Konstantin,

this is great feedback, thanks for all the measurements. Out of my curiosity, what Ruby version are you running smart-proxy on?

The config/lease loading definitely nees more love, parsing code is a bit ugly and all in-memory. Also, there is a possibility of creating Ruby "pickled" version of both files and checking timestamp every request which could improve parsing when no changes are done. We are discussing possibility of putting a database to smart proxy which could open other possibilities too.

I use ruby 1.8.7.

Thanks for considering the changes. What is the priority you think this issue is at?

Yes, I completely agree that parsing both config and lease files on every request is something that has to go first. Agree on the timestamp checking on the config files as well, but on leases - it still could be changing too rapidly (timestamp-wise, not necessarily new-data-wise) in large environments.

I really think you should consider using omshell over re-reading/keeping-state of the lease file. Since omshell already provides a lot of the same checks implemented by smart-proxy (like duplicate entries, etc.), removing that duplication from your ruby code and relying more on omshell (since it is much faster in dealing with lease file) is the way to go here.

Thanks!

I realize this is an old ticket but figured I would start here. We are seeing much higher load on our dhcp server / foreman smart-proxy after upgrading to 1.9.2 from 1.8. It would appear that every puppet run is now calling into the dhcp proxy with a call similar to "/dhcp/subnet/IP". Looking at debug logging of the smart-proxy, it is enumerating all IP addresses in all subnets before returning results. We are seeing calls take 15 to 20 seconds each. Should it be enumerating all leases, is the subnet-based lookup not coded correctly to only look in the specific subnet?

Our dhcp infrastructure is not overly large (8 24bit subnets in foreman, approx 70% used)

I can provide additional information or start a new ticket if needed.

Thanks,
Andrew

Dmitri, do the changes from http://projects.theforeman.org/issues/11866 help here at all? Or are they even supposed to?

It's an improvement, albeit a small one. The biggest improvement will come when we stop parsing the lease file on every request, something I'm planning on working on once https://github.com/theforeman/smart-proxy/pull/312 gets merged.

Great! One thing though - if you read through my notes from 2 years ago, parsing a lease file is not the only problem (a). Parsing all of config files is another issue IMHO (b).

To fix a problem (a), I believe relying more on features of OMAPI itself is the way to go as Foreman smart-proxy does not really need to duplicate the safeguards (like checking for duplicate records, etc.) already implemented by OMAPI. In fact, OMAPI does these checks much-much faster than smart-proxy, so I think this is where the biggest benefit will come from.

However, OMAPI will not solve problem (b) as there's no way to get subnet info through OMAPI :(
To get a list of subnets, parsing of config files would still have to happen for ISC, but it should be made more intelligent and cache things and use that unless the changes detected.

Does this make sense?

Yeah, I was thinking I can use omapi for all calls except 'unused_ip' one, which will require parsing of the lease file. dhcpd config file doesn't change nearly as often as the lease file, using a cache there will work.

What will prevent you from using OMAPI instead of parsing the whole lease file to make sure the IP is not given away?

But regardless of the implementation, I'm SO glad this finally gets some traction! Thank you! Any ideas on the timeframe? :)

I need to find an ip that might be useable first. The choices I have atm is parsing the lease file or pinging the whole subnet. I can start once https://github.com/theforeman/smart-proxy/pull/312 is merged. Any help with reviewing and/or testing of that PR would be greatly appreciated!

I see. Well, I think it is to always a good idea to ping an IP to be suggested to make sure it is not live on the network outside of DHCP knowledge. This will help prevent possible duplicates.

As for review - I'm afraid I personally can't help you with that as I'm not a reviewer or even a proper developer for that matter :) I just run Foreman at scale, so sometimes running into issues that most people just don't see.

Dmitri, what do I need to do to try this out as well in my environment?
I'm running 1.11.1 and 1.11.2 now - is your PR applicable to these versions?

Thanks!

The PR isn't going to work with 1.11 branch. The easiest thing to do atm is to use a source install from the "develop" branch. Alternatively, you can install a nightly foreman-proxy build (get it here: http://yum.theforeman.org/nightly) and apply this PR to it.

Hello, guys!

Thanks for working on this, but I still have any issue even after upgrading to 1.13.1. After the upgrade and modifications to dhcp.yml, restart of smart-proxy took 4 hours (based on the debug messages in the log), then went quiet for another 4 hours and finally self-disabled dhcp_isc module. Please see proxy.log here - https://gist.github.com/korekhov/4630630292626d25f192e03d13d33651

I do have a large dataset, and there are a lot of networks I'd love to exclude from smart-proxy as it would not manage those for sure - for example, /30s and /31s. Just to give you an idea on how many subnet sizes I have in one of the locations (and there are several like this):

746 255.255.252.0
    817 255.255.254.0
    909 255.255.255.0
    101 255.255.255.128
    168 255.255.255.192
   5454 255.255.255.252
   8328 255.255.255.254

As you can see, if I could blacklist \*/255.255.255.254 and \*/255.255.255.252 (or whitelist !\*/255.255.255.254 and !\*/255.255.255.252), smart-proxy would have to deal with dramatically smaller number of networks.
(I'm escaping stars above as redmine converts them into bold font)

In either case, can we re-open this issue?
Thanks!

Please open a new issue for any further problems, this one is linked to already shipped changes so won't be re-opened.