Bug #21120
closed
Multiple DHCP orchestration is no longer possible with PXELoader
Description
Hey,
since upgrading from foreman 14.x to 15.4 we have issues when modifying dhcp entries.
whenever we chage the bootloader from Grub2 UEFI to PXELinux BIOS, or vice versa, the foreman smart proxy start creating duplicate dhcp entries for this host.
how to repoduce:
1. create host with PXELinux BIOS bootloader
2. modify host to to use Grub2 UEFI
2. press submit (you will get a modal saying there already is a dhcp enty, and if you like to overwrite it)
3. acknowledge the modal and press the "overwrite" botton
4. now the "submit" button of the host is named "overwrite" -> press it
5. you will get a error message, complaining about a HTTP 409 from the dhcp smart proxy
if you check the leases file of the dhcp server, you will now have at least 2 entries for the same host. i have already see up to 33 entries for the same host within a file.
it also seems that the deleting a host will not always remove a entry from the dhcp leases file, but i have to admit that i am not 100% sure if the host will directly disappear from the leases file after it has been deleted.
to "fix" the issue, you have to:
- delete the host
- stop dhcpd
- remove broken entries from leases file
- start dhcpd
- recreate the host
here is a example from broken dhcp entries for the same host:
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:a2:84;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:9b:fb;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:d5:f7;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:ae:e9;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
Updated by Lukas Zapletal over 6 years ago
- Related to Refactor #19706: Don't treat DHCP leases as conflicts added
Updated by Lukas Zapletal over 6 years ago
- Status changed from New to Need more information
Most likely #19706 - there is a patch you can try, leave a comment in the PR if it works for you.
Updated by Stephan Schultchen over 6 years ago
Lukas Zapletal wrote:
Most likely #19706 - there is a patch you can try, leave a comment in the PR if it works for you.
i installed the changed record.rb file from the merge request, but i still get the same error: here is what i get in the WebUI:
Create DHCP Settings for stsforeman01fra.example.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on DHCP conflicts removal for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on Remove DHCP Settings for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on DHCP conflicts removal for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Updated by Lukas Zapletal over 6 years ago
- Subject changed from DHCP duplicate entries in Foreman 1.15.4 to Updating PXE loader causes DHCP conflicts
- Status changed from Need more information to New
The patch passed tests and review and it will be part of 1.15.5 update. But I did not read properly - PXE loader is the real cause behind this. This needs to be fixed. I will take a look later.
Updated by Stephan Schultchen over 6 years ago
great, when you are ready, i am happy to check if the fix is working for me
Updated by Lukas Zapletal over 6 years ago
- Related to Bug #21482: Unable to change host's PXE loader added
Updated by Trey Dockendorf over 6 years ago
Seeing this in 1.15.6 and just verified it making any change to a hosts PXE Loader. Initially it was hitting me going from PXELinux BIOS to Grub2 UEFI but I'm also getting it trying to go from None to PXELinux BIOS.
Updated by Lukas Zapletal over 6 years ago
So first of all, make sure you have DHCP IPAM enabled, otherwise DHCP is not orchestrated. Now, when I looked into this, it looks like things get orchestrated twice for some reason, the second attempt fails:
::1 - - [12/Dec/2017:14:51:11 CET] "GET /tftp/serverName HTTP/1.1" 200 30 ::1 - - [12/Dec/2017:14:51:11 CET] "GET /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 217 ::1 - - [12/Dec/2017:14:51:11 CET] "GET /dhcp/192.168.99.0/ip/192.168.99.131 HTTP/1.1" 200 219 ::1 - - [12/Dec/2017:14:51:16 CET] "GET /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 217 ::1 - - [12/Dec/2017:14:51:16 CET] "GET /dhcp/192.168.99.0/ip/192.168.99.131 HTTP/1.1" 200 219 ::1 - - [12/Dec/2017:14:51:16 CET] "GET /tftp/serverName HTTP/1.1" 200 30 ::1 - - [12/Dec/2017:14:51:16 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0 ::1 - - [12/Dec/2017:14:51:17 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0 ::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 200 0 ::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 409 49 ::1 - - [12/Dec/2017:14:51:17 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0 ::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 200 0 ::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 409 49
Updated by The Foreman Bot over 6 years ago
- Status changed from New to Ready For Testing
- Assignee set to Lukas Zapletal
- Pull request https://github.com/theforeman/foreman/pull/5079 added
Updated by Lukas Zapletal over 6 years ago
- Subject changed from Updating PXE loader causes DHCP conflicts to Multiple DHCP orchestration is no longer possible with PXELoader
So the problem is deep, there is now a PR that fixes it. Test it please.
Updated by Lukas Zapletal over 6 years ago
The root of the problem is in proxy, it used to accept multiple create calls. Now any additional call errors out with 409:
curl -#k --cert $HOME/.puppet/ssl/certs/$(hostname).pem --km --cacert $HOME/.puppet/ssl/certs/ca.pem -X POST -d '' "https://$(hostname):8443/dhcp/192.168.99.0?ip=192.168.99.14&mac=52:51:00:aa:bb:14&name=test14&filename=pxelinux.0" curl -#k --cert $HOME/.puppet/ssl/certs/$(hostname).pem --km --cacert $HOME/.puppet/ssl/certs/ca.pem -X POST -d '' "https://$(hostname):8443/dhcp/192.168.99.0?ip=192.168.99.14&mac=52:51:00:aa:bb:14&name=test14&filename=pxelinux.0" Record 192.168.99.0/192.168.99.14 already exists
The correct approach which I am working on is to prevent Foreman from orchestrating same actions twice. But I am not sure if this is feasible to backport into 1.15 series, looks like a change.
Updated by Stephan Schultchen over 6 years ago
Lukas Zapletal wrote:
The root of the problem is in proxy, it used to accept multiple create calls. Now any additional call errors out with 409:
[...]
The correct approach which I am working on is to prevent Foreman from orchestrating same actions twice. But I am not sure if this is feasible to backport into 1.15 series, looks like a change.
no worries, i am going to upgrade to 1.16 within the next few days anyhow.
Updated by Lukas Zapletal over 6 years ago
Version 1.16 does not change a thing, it's still broken in develop. Anyway, here is more context. I compared two workflows:
1) When you change a IP address of an interface.
2) When you change PXELoader.
In case (1) we only enqueue DHCP orchestration once, but in case (2) it is done twice. This stems from the fact that PXELoader is a flag on host rather than flag on NIC:
[lzap@box ]$ grep Enqueued *txt | grep DHCP ip.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue ip.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue ip.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue pxeloader.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
Only if I could find some easy woraround for 1.15/1.16 installations, because the patch I am currently working on is huge change in orchestration queueing that might be dangerous to backport.
Updated by The Foreman Bot over 6 years ago
- Pull request https://github.com/theforeman/foreman/pull/5085 added
Updated by Lukas Zapletal over 6 years ago
I filed an alternative backport-friendly version of the patch:
https://github.com/theforeman/foreman/pull/5085
You can easily apply it:
cd /usr/share/foreman
wget https://github.com/theforeman/foreman/pull/5085.patch
patch -p1 < 5085.patch
After httpd restart PXELoader can be edited now. Leave a comment in the PR if it works for you please.
Updated by Lukas Zapletal over 6 years ago
- Related to Bug #21975: DHCP filename option is ignored added
Updated by Stephan Schultchen over 6 years ago
i have just upgraded to 1.16.0, can i also use the patch for this version?
Updated by Daniel Lobato Garcia over 6 years ago
Stephan, yes - you may use it. I've tested it in 1.15.6 and 1.16.0 and it seems to work well in both.
Updated by Lukas Zapletal over 6 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 98433fced6230e4812f9e6a3153acb61f32074a3.
Updated by Daniel Lobato Garcia over 6 years ago
- translation missing: en.field_release set to 332
Updated by Lukas Zapletal over 6 years ago
- Status changed from Closed to Ready For Testing
To give you guys some news, we merged temporary patch into 1.16 and develop (https://github.com/theforeman/foreman/pull/5085) and I will do more complicated solution upstream after Christmas.
Updated by Marek Hulán about 6 years ago
- Status changed from Ready For Testing to Closed
Alternative fix, "the proper way", was merged into 1.18.0, see https://github.com/theforeman/foreman/pull/5085 for more details.
Updated by Lukas Zapletal about 5 years ago
- Related to Bug #26104: Multiple NIC orchestrations are not orchestrated added
Updated by Lukas Zapletal over 4 years ago
- Related to Bug #27877: DHCP conflict when editing PXE loader for existing host added