Bug #23784
closed
Unable to change host's MAC address
Description
I ran into this issue with 1.16.1 and was able to resolve by downgrading the smart proxy back to 1.15.6. The "Found in release" should be 1.17.1 which isn't an option it seems. Once again downgrading from 1.17.1 to 1.15.6 of the proxy resolved my issue.
I make changes to hosts via API and received this message:
ERROR: Failed to update https://foreman-test.ten.DOMAIN/api/hosts/o0074.ten.DOMAIN, message=[u'Remove DHCP Settings for o0074.ten.DOMAIN task failed with the following error: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 7c:d3:0a:b0:57:46 ([RestClient::NotFound]: 404 Not Found) for proxy https://bootsrv1-test.ten.DOMAIN:8443/dhcp'], details=
I'm attaching logs from Foreman and proxy from this incident.
Files
Updated by Lukas Zapletal over 6 years ago
- Project changed from Foreman to Smart Proxy
- Category set to DHCP
- Status changed from New to Need more information
Hey,
smart-proxy does not appear to be picking up reservation/leases? Is your /var/lib/dhcpd directory a local volume? Proxy no longer supports file systems which do not allow inotify (e.g. NFS).
Updated by Trey Dockendorf over 6 years ago
The /var path is on ext4:
/dev/mapper/vg0-lv_var /var ext4 defaults 1 2
When I restart foreman-proxy I see with DEBUG several "Add a reservation" lines but they all appear to be for static leases defined in /etc/dhcpd. Will the proxy also print the leases found in /var/lib/dhcpd/dhcpd.leases?
Permissions seem open enough to allow foreman-proxy user access:
# ls -la /var/lib/dhcpd/ total 16 drwxr-xr-x 2 dhcpd dhcpd 4096 Jun 1 12:26 . drwxr-xr-x. 42 root root 4096 Jun 4 03:23 .. -rw-r--r-- 1 dhcpd dhcpd 0 Aug 9 2016 dhcpd6.leases -rw-r--r-- 1 dhcpd dhcpd 3288 Jun 1 13:24 dhcpd.leases -rw-r--r-- 1 dhcpd dhcpd 4035 Jun 1 12:26 dhcpd.leases~
Updated by Trey Dockendorf over 6 years ago
- File production-rebuild.log production-rebuild.log added
- File proxy-rebuild.log proxy-rebuild.log added
The DHCP browser plugin shows nothing for subnets that have hosts in /var/lib/dhcpd/dhcpd.leases. Also rebuilding a host's config that should redeploy DHCP is failing in a similar way to changing a host's MAC address.
The test environment I have mimics my production environment in that it has a subset of same hosts but only a handful are actually in /var/lib/dhcpd/dhcpd.leases. This situation would more closely match an environment restored from backup where the proxy DHCP configuration is empty and has to be repopulated. I've run into this same error in production but the logs would be harder to look at due to high volume of activity.
I'm attaching logs of an attempt to rebuild configs for host o0001 with MAC 7c:d3:0a:b1:67:2c. The logs are kind of lengthy as BMCs got queried after failed attempt to rebuild from Host index page. The host that was supposed to be rebuilt was NOT already in /var/lib/dhcpd/dhcpd.leases but was already in Foreman as managed.
Updated by Anonymous over 6 years ago
Requests to dhcp/ endpoint return 404s, which is most likely due to an issue with dhcp module. Please check startup log for errors, it should clarify what the issue is.
Updated by Trey Dockendorf over 6 years ago
- File proxy-start.log proxy-start.log added
I see the problem. Attached log that shows startup failing to parse a line in dhcpd.leases. The "server-duid" was missing from the file for some reason. I restarted dhcpd which fixed dhcpd.leases and a restart of foreman-proxy after resolved the errors. DHCP updates through our normal means now work.
Updated by Anonymous over 6 years ago
- Tracker changed from Bug to Support
- Priority changed from High to Normal
Updated by Lukas Zapletal over 6 years ago
Permissions are sometimes an issue as well. Can this be closed?
Updated by Trey Dockendorf over 6 years ago
I think this can be closed. I'll re-open or link to new issue if problem manifests itself again and dhcpd.leases missing lines isn't the issue.
Thanks,
- Trey
Updated by Trey Dockendorf over 6 years ago
Actually I don't know if warrants different case or what but dhcpd.leases just lost server-duid line again and the proxy broke. I don't know if this is a bug with ISC DHCP or something that can be resolved in the proxy. Seems like the server server-duid line isn't necessary for proxy to function but it's absence breaks the proxy. This makes the proxy unnecessarily fragile.
Updated by Trey Dockendorf over 6 years ago
- File dhcpd.leases dhcpd.leases added
I may have a reproducer.
1) Fix DHCP and Proxy
2) Trigger proxy to add new DHCP record that puts server-duid as not being the last line in file
3) Restart proxy and get failure to parse keyword_server_duid
Error:
D, [2018-06-05T08:40:53.275458 ] DEBUG -- : ["/usr/share/gems/gems/rsec-0.4.2/lib/rsec/parser.rb:19:in `parse!'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:404:in `subnets_hosts_and_leases'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/subnet_service_initialization.rb:17:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:156:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:130:in `block in do_start'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `block in group_changes'", "/usr/share/ruby/monitor.rb:211:in `mon_synchronize'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `group_changes'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:128:in `do_start'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:80:in `start'", "/usr/share/foreman-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:42:in `start'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:238:in `block in start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:221:in `configure_plugin'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `block in configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `block in initialize_plugins'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `initialize_plugins'", "/usr/share/foreman-proxy/lib/launcher.rb:166:in `launch'", "/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'"] E, [2018-06-05T08:40:53.275623 ] ERROR -- : Disabling all modules in the group ['dhcp_isc', 'dhcp'] due to a failure in one of them: in /var/lib/dhcpd/dhcpd.leases:146 at 1, expect token [ keyword_server_duid ] } ^
I've attached dhcpd.leases that causes this problem. Note Line 146 is the last line in file but server-duid is line 131 because I triggered Foreman to add DHCP records for o0597 and c6320-0597. Seems the proxy expects server-duid to be last line or near end of file and that's not always the case.
Updated by Trey Dockendorf over 6 years ago
This should probably be moved back to a bug and not support?
Updated by Lukas Zapletal over 6 years ago
Can you try #23433 patch if that helps? I believe this was fixed.
Updated by Trey Dockendorf over 6 years ago
The patch for #23433 worked to solve the problem. Any chance of backport in 1.17 branch?
Updated by Lukas Zapletal over 6 years ago
- Tracker changed from Support to Bug
- Status changed from Need more information to Duplicate
Trey, backport request was sent into 1.17.2 on the other issue. Here, we have a RedMine technical problem, we can't change Status for some reason. Asked our admin to take a look. This will be closed as dupe: #23433
Updated by Lukas Zapletal over 6 years ago
Solved it, subscribe to the other issue to see the CP progress.