Project

General

Profile

Bug #21120

Multiple DHCP orchestration is no longer possible with PXELoader

Added by Stephan Schultchen about 1 year ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
DHCP
Target version:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Hey,

since upgrading from foreman 14.x to 15.4 we have issues when modifying dhcp entries.

whenever we chage the bootloader from Grub2 UEFI to PXELinux BIOS, or vice versa, the foreman smart proxy start creating duplicate dhcp entries for this host.

how to repoduce:

1. create host with PXELinux BIOS bootloader
2. modify host to to use Grub2 UEFI
2. press submit (you will get a modal saying there already is a dhcp enty, and if you like to overwrite it)
3. acknowledge the modal and press the "overwrite" botton
4. now the "submit" button of the host is named "overwrite" -> press it
5. you will get a error message, complaining about a HTTP 409 from the dhcp smart proxy

if you check the leases file of the dhcp server, you will now have at least 2 entries for the same host. i have already see up to 33 entries for the same host within a file.

it also seems that the deleting a host will not always remove a entry from the dhcp leases file, but i have to admit that i am not 100% sure if the host will directly disappear from the leases file after it has been deleted.

to "fix" the issue, you have to:
- delete the host
- stop dhcpd
- remove broken entries from leases file
- start dhcpd
- recreate the host

here is a example from broken dhcp entries for the same host:

host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:a2:84;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:9b:fb;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:d5:f7;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}
host samplehost.example.com {
dynamic;
deleted;
}
host samplehost.example.com {
dynamic;
hardware ethernet 00:50:56:91:ae:e9;
fixed-address 172.21.55.57;
supersede server.filename = "pxelinux.0";
supersede server.next-server = ac:15:3a:05;
supersede host-name = "samplehost.example.com";
}


Related issues

Related to Foreman - Refactor #19706: Don't treat DHCP leases as conflictsClosed2017-05-30
Related to Foreman - Bug #21482: Unable to change host's PXE loaderDuplicate2017-10-26
Related to Smart Proxy - Bug #21975: DHCP filename option is ignoredClosed2017-12-14

Associated revisions

Revision 98433fce (diff)
Added by Lukas Zapletal 11 months ago

Fixes #21120 - DHCP update no longer queued twice

Revision 5f10b6af (diff)
Added by Lukas Zapletal 8 months ago

Fixes #21120 - orchestration tasks are added only once (#5079)

History

#1 Updated by Lukas Zapletal about 1 year ago

#2 Updated by Lukas Zapletal about 1 year ago

  • Status changed from New to Need more information

Most likely #19706 - there is a patch you can try, leave a comment in the PR if it works for you.

#3 Updated by Stephan Schultchen about 1 year ago

Lukas Zapletal wrote:

Most likely #19706 - there is a patch you can try, leave a comment in the PR if it works for you.

i installed the changed record.rb file from the merge request, but i still get the same error: here is what i get in the WebUI:

Create DHCP Settings for stsforeman01fra.example.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on DHCP conflicts removal for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on Remove DHCP Settings for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp
Failed to perform rollback on DHCP conflicts removal for stsforeman01fra.example.com - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://fmsmart01fra.example.com:8443/dhcp

#4 Updated by Lukas Zapletal about 1 year ago

  • Subject changed from DHCP duplicate entries in Foreman 1.15.4 to Updating PXE loader causes DHCP conflicts
  • Status changed from Need more information to New

The patch passed tests and review and it will be part of 1.15.5 update. But I did not read properly - PXE loader is the real cause behind this. This needs to be fixed. I will take a look later.

#5 Updated by Stephan Schultchen about 1 year ago

great, when you are ready, i am happy to check if the fix is working for me

#6 Updated by Lukas Zapletal about 1 year ago

  • Related to Bug #21482: Unable to change host's PXE loader added

#7 Updated by Trey Dockendorf about 1 year ago

Seeing this in 1.15.6 and just verified it making any change to a hosts PXE Loader. Initially it was hitting me going from PXELinux BIOS to Grub2 UEFI but I'm also getting it trying to go from None to PXELinux BIOS.

#8 Updated by Lukas Zapletal 11 months ago

So first of all, make sure you have DHCP IPAM enabled, otherwise DHCP is not orchestrated. Now, when I looked into this, it looks like things get orchestrated twice for some reason, the second attempt fails:

::1 - - [12/Dec/2017:14:51:11 CET] "GET /tftp/serverName HTTP/1.1" 200 30
::1 - - [12/Dec/2017:14:51:11 CET] "GET /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 217
::1 - - [12/Dec/2017:14:51:11 CET] "GET /dhcp/192.168.99.0/ip/192.168.99.131 HTTP/1.1" 200 219
::1 - - [12/Dec/2017:14:51:16 CET] "GET /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 217
::1 - - [12/Dec/2017:14:51:16 CET] "GET /dhcp/192.168.99.0/ip/192.168.99.131 HTTP/1.1" 200 219
::1 - - [12/Dec/2017:14:51:16 CET] "GET /tftp/serverName HTTP/1.1" 200 30
::1 - - [12/Dec/2017:14:51:16 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0
::1 - - [12/Dec/2017:14:51:17 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0
::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 200 0
::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 409 49
::1 - - [12/Dec/2017:14:51:17 CET] "DELETE /dhcp/192.168.99.0/mac/52:54:00:62:2a:1d HTTP/1.1" 200 0
::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 200 0
::1 - - [12/Dec/2017:14:51:17 CET] "POST /dhcp/192.168.99.0 HTTP/1.1" 409 49

#9 Updated by The Foreman Bot 11 months ago

  • Assignee set to Lukas Zapletal
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/5079 added

#10 Updated by Lukas Zapletal 11 months ago

  • Subject changed from Updating PXE loader causes DHCP conflicts to Multiple DHCP orchestration is no longer possible with PXELoader

So the problem is deep, there is now a PR that fixes it. Test it please.

#11 Updated by Lukas Zapletal 11 months ago

The root of the problem is in proxy, it used to accept multiple create calls. Now any additional call errors out with 409:

curl -#k --cert $HOME/.puppet/ssl/certs/$(hostname).pem --km --cacert $HOME/.puppet/ssl/certs/ca.pem -X POST -d '' "https://$(hostname):8443/dhcp/192.168.99.0?ip=192.168.99.14&mac=52:51:00:aa:bb:14&name=test14&filename=pxelinux.0" 
curl -#k --cert $HOME/.puppet/ssl/certs/$(hostname).pem --km --cacert $HOME/.puppet/ssl/certs/ca.pem -X POST -d '' "https://$(hostname):8443/dhcp/192.168.99.0?ip=192.168.99.14&mac=52:51:00:aa:bb:14&name=test14&filename=pxelinux.0" 
Record 192.168.99.0/192.168.99.14 already exists

The correct approach which I am working on is to prevent Foreman from orchestrating same actions twice. But I am not sure if this is feasible to backport into 1.15 series, looks like a change.

#12 Updated by Stephan Schultchen 11 months ago

Lukas Zapletal wrote:

The root of the problem is in proxy, it used to accept multiple create calls. Now any additional call errors out with 409:

[...]

The correct approach which I am working on is to prevent Foreman from orchestrating same actions twice. But I am not sure if this is feasible to backport into 1.15 series, looks like a change.

no worries, i am going to upgrade to 1.16 within the next few days anyhow.

#13 Updated by Lukas Zapletal 11 months ago

Version 1.16 does not change a thing, it's still broken in develop. Anyway, here is more context. I compared two workflows:

1) When you change a IP address of an interface.
2) When you change PXELoader.

In case (1) we only enqueue DHCP orchestration once, but in case (2) it is done twice. This stems from the fact that PXELoader is a flag on host rather than flag on NIC:

[lzap@box ]$ grep Enqueued *txt | grep DHCP
ip.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue
ip.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
ip.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'DHCP conflicts removal for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'Remove DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue
pxeloader.txt:[app|D] Enqueued task 'Create DHCP Settings for joan-jacoby.home.lan' to 'Host::Managed Main' queue

Only if I could find some easy woraround for 1.15/1.16 installations, because the patch I am currently working on is huge change in orchestration queueing that might be dangerous to backport.

#14 Updated by The Foreman Bot 11 months ago

  • Pull request https://github.com/theforeman/foreman/pull/5085 added

#15 Updated by Lukas Zapletal 11 months ago

I filed an alternative backport-friendly version of the patch:

https://github.com/theforeman/foreman/pull/5085

You can easily apply it:

cd /usr/share/foreman
wget https://github.com/theforeman/foreman/pull/5085.patch
patch -p1 < 5085.patch

After httpd restart PXELoader can be edited now. Leave a comment in the PR if it works for you please.

#16 Updated by Lukas Zapletal 11 months ago

  • Related to Bug #21975: DHCP filename option is ignored added

#17 Updated by Stephan Schultchen 11 months ago

i have just upgraded to 1.16.0, can i also use the patch for this version?

#18 Updated by Daniel Lobato Garcia 11 months ago

Stephan, yes - you may use it. I've tested it in 1.15.6 and 1.16.0 and it seems to work well in both.

#19 Updated by Lukas Zapletal 11 months ago

  • % Done changed from 0 to 100
  • Status changed from Ready For Testing to Closed

#20 Updated by Daniel Lobato Garcia 11 months ago

  • Legacy Backlogs Release (now unused) set to 332

#21 Updated by Lukas Zapletal 11 months ago

  • Status changed from Closed to Ready For Testing

To give you guys some news, we merged temporary patch into 1.16 and develop (https://github.com/theforeman/foreman/pull/5085) and I will do more complicated solution upstream after Christmas.

#22 Updated by Lukas Zapletal 11 months ago

  • Bugzilla link set to 1527806

#23 Updated by Marek Hulán 8 months ago

  • Status changed from Ready For Testing to Closed

Alternative fix, "the proper way", was merged into 1.18.0, see https://github.com/theforeman/foreman/pull/5085 for more details.

Also available in: Atom PDF