Foreman » Smart Proxy

Environment:
ISC-DHCP-Server version 4.3.3 on Ubuntu 16.04 LTS server
Smart-Proxy version 1.13.1 on separate Ubuntu 14.04 LTS server

Smart-proxy must be located on the same server as isc dhcp server. NFS-mounted partitions will not work anymore.

does it make sense to check if its mounted as NFS ?
also, should we provide a fall back option?

I think NFS support is important, as I know multiple users who uses HA solution for dhcp proxy using shared NFS storage (e.g. using pacemaker)

Accessing leases file over NFS will absolutely destroy the performance: we no longer will be able to track updates and will have to parse the leases on every request. On top of that, accessing leases over NFS adds additional latency. If HA is required, we'll need to address this via different architecture, and not by relying on NFS.

I feel we shouldn't support NFS at all -- it's a corner case that hugely impacts performance.

Yes, we're mounting over NFS. The smart-proxy and DCHP server are in the same cluster and traffic is layer 2 so latency should be very low. I can run pref tests if needed. I'll test with the DHCP server and smart-proxy on the same host.

Restarting the smart-proxy after deleting the host does not remove the entry from the leases file.

This may be unrelated but when provisioning a new host the TFTP smart-proxy adds entries to the UEFI grub2 folder even though the PXE loader is set as PXELinux and we're using PXELinux provisioning templates. Will see if issue persists when running the DHCP server and smart-proxy on the same box.

Thanks you

While not ideal, NFS is still a very important part of HA strategies, mainly because of its simplicity. As we discussing in other thread, most of the DHCP smart-proxy performance already handled by your recent changes, Dmitri, so may be it makes sense to re-test the performance over NFS with those?

In terms of the latency - any of the file-sync'ing solutions for HA (cluster FS, rsync, inotify, etc.) will always have some sort of a delay or latency associated with that. I think the application should be able to handle that.

Just my 2 cents since this dropping NFS support in 1.13 just as performance issues were resolved is huge bummer for me - seems like I just can't get all the features I need, something is always missing :(

While not ideal, NFS is still a very important part of HA strategies, mainly because of its simplicity. As we discussing in other thread, most of the DHCP smart-proxy performance already handled by your recent changes, Dmitri, so may be it makes sense to re-test the performance over NFS with those?

In your environment you'll be looking at 30+ seconds processing time: each request would require fetching leases file over the network and then parsing it. Additionally one web-server thread will be tied up for the duration of the request. I don't think the way forward is to rely on NFS as it sacrifices too much of performance, but to have a smart-proxy based HA.

I don't think we ought to discuss degrees of horribleness when we know we can do better.

To reiterate: If we were to use NFS, we'd have to retrieve and parse leases file on each request, which doesn't scale at all. I don't want to support and maintain a solution that only works for some installations. If HA setups are a must, we need a solution that supports them without degradation in performance compared to baseline, and not an NFS-based workaround.

Dmitri Dolguikh wrote:

I don't think we ought to discuss degrees of horribleness when we know we can do better.

To reiterate: If we were to use NFS, we'd have to retrieve and parse leases file on each request, which doesn't scale at all. I don't want to support and maintain a solution that only works for some installations. If HA setups are a must, we need a solution that supports them without degradation in performance compared to baseline, and not an NFS-based workaround.

Just my two cents - I just encountered this my self.
My workaround was to patch the proxy to start a thread which computes checksum in a loop and call observer.leases_recreated if the digest changed.
This works perfectly, and it seems there is no noticeable performance hit.

If that is a valid course of action, I can open a PR for this.

Imri, would you mind sharing your patch? I don't know how large is your environment, mine is very large (several dozen updates to leases in a second) and I could test your approach from performance point of view.

Dmitri, if the above approach does not for whatever reason, I was wondering if you've given a thought of putting local DHCP cache that SmartProxy creates into memcached?

This way it is replicated across all memcached cluster nodes w/o extra efforts from user or smart-proxy perspective to allow multi-node access - I'm sure I don't really have explain to you the advantages of this product.

Performance may be a concern, of course, but I can offer my assistance in testing it out.

Dmitri, any thoughts on using memcached to keep DHCP leases and reservations in sync between multiple smart-proxy nodes?

Dmitri, any thoughts on using memcached to keep DHCP leases and reservations in sync between multiple smart-proxy nodes?

I haven't thought about this; I suspect any distributed key-value store can be made to work (etcd would be my personal choice) at a cost of increased complexity and slightly increased response time. Probably a good candidate for a dhcp module provider.

PS> Not sure if you noticed but there's a provider that supports loading of leases file from remote filesystems: https://github.com/theforeman/smart_proxy_dhcp_remote_isc. In all likelihood it's too slow for your network though.

Hello, Dmitri!

Finally got a chance to try dhcp_remote_isc provider and ran into some issues, which I think maybe related to some missing documentation.

I followed your readme in a git repo and here are the steps I took:

1. gem list | grep dhcp
   smart_proxy_dhcp_remote_isc (0.0.2)

- add that GEM into bundler.d (BTW, you probably want to update your README as it references smart_proxy_dhcp_infoblox GEM instead of smart_proxy_dhcp_remote_isc)
cat /usr/share/foreman-proxy/bundler.d/dhcp_remote_isc.rb
gem 'smart_proxy_dhcp_remote_isc'

- based on your doc, i created a symlink from /etc/foreman-proxy/settings.d/dhcp_isc.yml to /etc/foreman-proxy/settings.d/dhcp_remote_isc.yml, otherwise I would get this:

I, [2017-05-29T12:52:28.768926 ] INFO -- : WEBrick::HTTPServer#start done.
W, [2017-05-29T12:58:14.333731 ] WARN -- : Couldn't find settings file /etc/foreman-proxy/settings.d/dhcp_remote_isc.yml. Using default settings.
E, [2017-05-29T12:58:14.337632 ] ERROR -- : Disabling all modules in the group ['dhcp_remote_isc', 'dhcp'] due to a failure in one of them: cannot load such file -- dhcp_common/isc/omapi_provider

- but now I'm stuck here - even though I believe I fulfilled all the requirements in a readme, I get this error:
I, [2017-05-29T13:30:34.861203 ] INFO -- : WEBrick::HTTPServer#start done.
E, [2017-05-29T13:30:35.201942 ] ERROR -- : Disabling all modules in the group ['dhcp_remote
_isc', 'dhcp'] due to a failure in one of them: cannot load such file -- dhcp_common/isc/oma
pi_provider
I, [2017-05-29T13:30:35.579158 ] INFO -- : Successfully initialized 'dynflow'
I, [2017-05-29T13:30:35.581172 ] INFO -- : Successfully initialized 'ssh'
I, [2017-05-29T13:30:35.581282 ] INFO -- : Successfully initialized 'foreman_proxy'
I, [2017-05-29T13:30:35.581350 ] INFO -- : Successfully initialized 'tftp'
I, [2017-05-29T13:30:35.583296 ] INFO -- : Successfully initialized 'puppet_proxy_legacy'
I, [2017-05-29T13:30:35.583431 ] INFO -- : Successfully initialized 'puppet'
I, [2017-05-29T13:30:35.583517 ] INFO -- : Successfully initialized 'bmc'
I, [2017-05-29T13:30:35.583655 ] INFO -- : Successfully initialized 'logs'

- if I update dhcp.yml back to use dhcp_isc provider, SmP starts just fine:

I, [2017-05-29T13:43:41.862860 ] INFO -- : going to shutdown ...
I, [2017-05-29T13:43:41.863003 ] INFO -- : WEBrick::HTTPServer#start done.
I, [2017-05-29T13:43:42.575187 ] INFO -- : Successfully initialized 'dynflow'
I, [2017-05-29T13:43:42.577613 ] INFO -- : Successfully initialized 'ssh'
I, [2017-05-29T13:43:42.577704 ] INFO -- : Successfully initialized 'foreman_proxy'
I, [2017-05-29T13:43:42.577771 ] INFO -- : Successfully initialized 'tftp'
I, [2017-05-29T13:43:42.609535 ] INFO -- : Successfully initialized 'dhcp_isc'
I, [2017-05-29T13:43:42.609626 ] INFO -- : Successfully initialized 'dhcp'
I, [2017-05-29T13:43:42.611096 ] INFO -- : Successfully initialized 'puppet_proxy_legacy'
I, [2017-05-29T13:43:42.611178 ] INFO -- : Successfully initialized 'puppet'
I, [2017-05-29T13:43:42.611222 ] INFO -- : Successfully initialized 'bmc'
I, [2017-05-29T13:43:42.611267 ] INFO -- : Successfully initialized 'logs'

I must be missing something is a config somewhere, just not able to determine what that is so far.
If you have any suggestions, please let me know.

Thanks!

Make sure you are using smart-proxy version 1.15. I'll update the provider to work with develop branch today.

I released v. 0.0.3 of smart_proxy_dhcp_remote_isc gem that is required if smart-proxy develop branch is used.

Ah, I was on 1.14 actually, so will have to upgrade today while I have a chance.

And now I can't upgrade as I do need discovery working - https://groups.google.com/forum/#!topic/foreman-users/M_DcyFMZwxM

Dhcp-related issues have been fixed in develop, with fixes available in the next stable release.