Unable to change host's MAC address
I ran into this issue with 1.16.1 and was able to resolve by downgrading the smart proxy back to 1.15.6. The "Found in release" should be 1.17.1 which isn't an option it seems. Once again downgrading from 1.17.1 to 1.15.6 of the proxy resolved my issue.
I make changes to hosts via API and received this message:
ERROR: Failed to update https://foreman-test.ten.DOMAIN/api/hosts/o0074.ten.DOMAIN, message=[u'Remove DHCP Settings for o0074.ten.DOMAIN task failed with the following error: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 7c:d3:0a:b0:57:46 ([RestClient::NotFound]: 404 Not Found) for proxy https://bootsrv1-test.ten.DOMAIN:8443/dhcp'], details=
I'm attaching logs from Foreman and proxy from this incident.
#1 Updated by Lukas Zapletal over 2 years ago
- Status changed from New to Need more information
- Category set to DHCP
- Project changed from Foreman to Smart Proxy
smart-proxy does not appear to be picking up reservation/leases? Is your /var/lib/dhcpd directory a local volume? Proxy no longer supports file systems which do not allow inotify (e.g. NFS).
#2 Updated by Trey Dockendorf over 2 years ago
The /var path is on ext4:
/dev/mapper/vg0-lv_var /var ext4 defaults 1 2
When I restart foreman-proxy I see with DEBUG several "Add a reservation" lines but they all appear to be for static leases defined in /etc/dhcpd. Will the proxy also print the leases found in /var/lib/dhcpd/dhcpd.leases?
Permissions seem open enough to allow foreman-proxy user access:
# ls -la /var/lib/dhcpd/ total 16 drwxr-xr-x 2 dhcpd dhcpd 4096 Jun 1 12:26 . drwxr-xr-x. 42 root root 4096 Jun 4 03:23 .. -rw-r--r-- 1 dhcpd dhcpd 0 Aug 9 2016 dhcpd6.leases -rw-r--r-- 1 dhcpd dhcpd 3288 Jun 1 13:24 dhcpd.leases -rw-r--r-- 1 dhcpd dhcpd 4035 Jun 1 12:26 dhcpd.leases~
#3 Updated by Trey Dockendorf over 2 years ago
- File proxy-rebuild.log proxy-rebuild.log added
- File production-rebuild.log production-rebuild.log added
The DHCP browser plugin shows nothing for subnets that have hosts in /var/lib/dhcpd/dhcpd.leases. Also rebuilding a host's config that should redeploy DHCP is failing in a similar way to changing a host's MAC address.
The test environment I have mimics my production environment in that it has a subset of same hosts but only a handful are actually in /var/lib/dhcpd/dhcpd.leases. This situation would more closely match an environment restored from backup where the proxy DHCP configuration is empty and has to be repopulated. I've run into this same error in production but the logs would be harder to look at due to high volume of activity.
I'm attaching logs of an attempt to rebuild configs for host o0001 with MAC 7c:d3:0a:b1:67:2c. The logs are kind of lengthy as BMCs got queried after failed attempt to rebuild from Host index page. The host that was supposed to be rebuilt was NOT already in /var/lib/dhcpd/dhcpd.leases but was already in Foreman as managed.
#5 Updated by Trey Dockendorf over 2 years ago
I see the problem. Attached log that shows startup failing to parse a line in dhcpd.leases. The "server-duid" was missing from the file for some reason. I restarted dhcpd which fixed dhcpd.leases and a restart of foreman-proxy after resolved the errors. DHCP updates through our normal means now work.
#9 Updated by Trey Dockendorf over 2 years ago
Actually I don't know if warrants different case or what but dhcpd.leases just lost server-duid line again and the proxy broke. I don't know if this is a bug with ISC DHCP or something that can be resolved in the proxy. Seems like the server server-duid line isn't necessary for proxy to function but it's absence breaks the proxy. This makes the proxy unnecessarily fragile.
#10 Updated by Trey Dockendorf over 2 years ago
I may have a reproducer.
1) Fix DHCP and Proxy
2) Trigger proxy to add new DHCP record that puts server-duid as not being the last line in file
3) Restart proxy and get failure to parse keyword_server_duid
D, [2018-06-05T08:40:53.275458 ] DEBUG -- : ["/usr/share/gems/gems/rsec-0.4.2/lib/rsec/parser.rb:19:in `parse!'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:404:in `subnets_hosts_and_leases'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/subnet_service_initialization.rb:17:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:156:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:130:in `block in do_start'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `block in group_changes'", "/usr/share/ruby/monitor.rb:211:in `mon_synchronize'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `group_changes'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:128:in `do_start'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:80:in `start'", "/usr/share/foreman-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:42:in `start'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:238:in `block in start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:221:in `configure_plugin'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `block in configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `block in initialize_plugins'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `initialize_plugins'", "/usr/share/foreman-proxy/lib/launcher.rb:166:in `launch'", "/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'"] E, [2018-06-05T08:40:53.275623 ] ERROR -- : Disabling all modules in the group ['dhcp_isc', 'dhcp'] due to a failure in one of them: in /var/lib/dhcpd/dhcpd.leases:146 at 1, expect token [ keyword_server_duid ] } ^
I've attached dhcpd.leases that causes this problem. Note Line 146 is the last line in file but server-duid is line 131 because I triggered Foreman to add DHCP records for o0597 and c6320-0597. Seems the proxy expects server-duid to be last line or near end of file and that's not always the case.
#14 Updated by Lukas Zapletal over 2 years ago
- Status changed from Need more information to Duplicate
- Tracker changed from Support to Bug
Trey, backport request was sent into 1.17.2 on the other issue. Here, we have a RedMine technical problem, we can't change Status for some reason. Asked our admin to take a look. This will be closed as dupe: #23433