Project

General

Profile

Bug #23784

Unable to change host's MAC address

Added by Trey Dockendorf over 2 years ago. Updated over 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
DHCP
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

I ran into this issue with 1.16.1 and was able to resolve by downgrading the smart proxy back to 1.15.6. The "Found in release" should be 1.17.1 which isn't an option it seems. Once again downgrading from 1.17.1 to 1.15.6 of the proxy resolved my issue.

I make changes to hosts via API and received this message:

ERROR: Failed to update https://foreman-test.ten.DOMAIN/api/hosts/o0074.ten.DOMAIN, message=[u'Remove DHCP Settings for o0074.ten.DOMAIN task failed with the following error: ERF12-0635 [ProxyAPI::ProxyException]: Unable to delete DHCP entry for 7c:d3:0a:b0:57:46 ([RestClient::NotFound]: 404 Not Found) for proxy https://bootsrv1-test.ten.DOMAIN:8443/dhcp'], details=

I'm attaching logs from Foreman and proxy from this incident.

production.log production.log 47.1 KB Trey Dockendorf, 06/01/2018 04:27 PM
proxy.log proxy.log 6.05 KB Trey Dockendorf, 06/01/2018 04:27 PM
proxy-rebuild.log proxy-rebuild.log 83.2 KB Trey Dockendorf, 06/04/2018 01:46 PM
production-rebuild.log production-rebuild.log 139 KB Trey Dockendorf, 06/04/2018 01:46 PM
proxy-start.log proxy-start.log 28.1 KB Trey Dockendorf, 06/04/2018 04:22 PM
dhcpd.leases dhcpd.leases 4.66 KB Trey Dockendorf, 06/05/2018 12:45 PM

History

#1 Updated by Lukas Zapletal over 2 years ago

  • Status changed from New to Need more information
  • Category set to DHCP
  • Project changed from Foreman to Smart Proxy

Hey,

smart-proxy does not appear to be picking up reservation/leases? Is your /var/lib/dhcpd directory a local volume? Proxy no longer supports file systems which do not allow inotify (e.g. NFS).

#2 Updated by Trey Dockendorf over 2 years ago

The /var path is on ext4:

/dev/mapper/vg0-lv_var  /var                    ext4    defaults        1 2

When I restart foreman-proxy I see with DEBUG several "Add a reservation" lines but they all appear to be for static leases defined in /etc/dhcpd. Will the proxy also print the leases found in /var/lib/dhcpd/dhcpd.leases?

Permissions seem open enough to allow foreman-proxy user access:

# ls -la /var/lib/dhcpd/
total 16
drwxr-xr-x   2 dhcpd dhcpd 4096 Jun  1 12:26 .
drwxr-xr-x. 42 root  root  4096 Jun  4 03:23 ..
-rw-r--r--   1 dhcpd dhcpd    0 Aug  9  2016 dhcpd6.leases
-rw-r--r--   1 dhcpd dhcpd 3288 Jun  1 13:24 dhcpd.leases
-rw-r--r--   1 dhcpd dhcpd 4035 Jun  1 12:26 dhcpd.leases~

#3 Updated by Trey Dockendorf over 2 years ago

The DHCP browser plugin shows nothing for subnets that have hosts in /var/lib/dhcpd/dhcpd.leases. Also rebuilding a host's config that should redeploy DHCP is failing in a similar way to changing a host's MAC address.

The test environment I have mimics my production environment in that it has a subset of same hosts but only a handful are actually in /var/lib/dhcpd/dhcpd.leases. This situation would more closely match an environment restored from backup where the proxy DHCP configuration is empty and has to be repopulated. I've run into this same error in production but the logs would be harder to look at due to high volume of activity.

I'm attaching logs of an attempt to rebuild configs for host o0001 with MAC 7c:d3:0a:b1:67:2c. The logs are kind of lengthy as BMCs got queried after failed attempt to rebuild from Host index page. The host that was supposed to be rebuilt was NOT already in /var/lib/dhcpd/dhcpd.leases but was already in Foreman as managed.

#4 Updated by Dmitri Dolguikh over 2 years ago

Requests to dhcp/ endpoint return 404s, which is most likely due to an issue with dhcp module. Please check startup log for errors, it should clarify what the issue is.

#5 Updated by Trey Dockendorf over 2 years ago

I see the problem. Attached log that shows startup failing to parse a line in dhcpd.leases. The "server-duid" was missing from the file for some reason. I restarted dhcpd which fixed dhcpd.leases and a restart of foreman-proxy after resolved the errors. DHCP updates through our normal means now work.

#6 Updated by Dmitri Dolguikh over 2 years ago

  • Priority changed from High to Normal
  • Tracker changed from Bug to Support

#7 Updated by Lukas Zapletal over 2 years ago

Permissions are sometimes an issue as well. Can this be closed?

#8 Updated by Trey Dockendorf over 2 years ago

I think this can be closed. I'll re-open or link to new issue if problem manifests itself again and dhcpd.leases missing lines isn't the issue.

Thanks,
- Trey

#9 Updated by Trey Dockendorf over 2 years ago

Actually I don't know if warrants different case or what but dhcpd.leases just lost server-duid line again and the proxy broke. I don't know if this is a bug with ISC DHCP or something that can be resolved in the proxy. Seems like the server server-duid line isn't necessary for proxy to function but it's absence breaks the proxy. This makes the proxy unnecessarily fragile.

#10 Updated by Trey Dockendorf over 2 years ago

I may have a reproducer.

1) Fix DHCP and Proxy
2) Trigger proxy to add new DHCP record that puts server-duid as not being the last line in file
3) Restart proxy and get failure to parse keyword_server_duid

Error:

D, [2018-06-05T08:40:53.275458 ] DEBUG -- : ["/usr/share/gems/gems/rsec-0.4.2/lib/rsec/parser.rb:19:in `parse!'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/configuration_parser.rb:404:in `subnets_hosts_and_leases'", "/usr/share/foreman-proxy/modules/dhcp_common/isc/subnet_service_initialization.rb:17:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:156:in `load_leases_file'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:130:in `block in do_start'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `block in group_changes'", "/usr/share/ruby/monitor.rb:211:in `mon_synchronize'", "/usr/share/foreman-proxy/modules/dhcp_common/subnet_service.rb:154:in `group_changes'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:128:in `do_start'", "/usr/share/foreman-proxy/modules/dhcp_isc/isc_state_changes_observer.rb:80:in `start'", "/usr/share/foreman-proxy/modules/dhcp_isc/inotify_leases_file_observer.rb:42:in `start'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:238:in `block in start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:236:in `start_services'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:221:in `configure_plugin'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `block in configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:92:in `configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `block in initialize_plugins'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:170:in `initialize_plugins'", "/usr/share/foreman-proxy/lib/launcher.rb:166:in `launch'", "/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'"]
E, [2018-06-05T08:40:53.275623 ] ERROR -- : Disabling all modules in the group ['dhcp_isc', 'dhcp'] due to a failure in one of them: 
in /var/lib/dhcpd/dhcpd.leases:146 at 1, expect token [ keyword_server_duid ]
}
 ^

I've attached dhcpd.leases that causes this problem. Note Line 146 is the last line in file but server-duid is line 131 because I triggered Foreman to add DHCP records for o0597 and c6320-0597. Seems the proxy expects server-duid to be last line or near end of file and that's not always the case.

#11 Updated by Trey Dockendorf over 2 years ago

This should probably be moved back to a bug and not support?

#12 Updated by Lukas Zapletal over 2 years ago

Can you try #23433 patch if that helps? I believe this was fixed.

#13 Updated by Trey Dockendorf over 2 years ago

The patch for #23433 worked to solve the problem. Any chance of backport in 1.17 branch?

#14 Updated by Lukas Zapletal over 2 years ago

  • Status changed from Need more information to Duplicate
  • Tracker changed from Support to Bug

Trey, backport request was sent into 1.17.2 on the other issue. Here, we have a RedMine technical problem, we can't change Status for some reason. Asked our admin to take a look. This will be closed as dupe: #23433

#15 Updated by Lukas Zapletal over 2 years ago

Solved it, subscribe to the other issue to see the CP progress.

Also available in: Atom PDF