Bug #17197

DNS being recreated on host update when ip6 field changes from nil to empty

Added by Paul Smyth 9 months ago. Updated 9 months ago.

Status:Closed
Priority:Normal
Assigned To:Dominic Cleal
Category:DNS
Target version:-
Difficulty: Bugzilla link:
Found in release:1.13.1 Pull request:https://github.com/theforeman/foreman/pull/3990
Story points-
Velocity based estimate-
Release1.13.2Release relationshipAuto

Description

Since the most recent update of Foreman to 1.13.1 route 53 entries are being deleted whenever I make a change via puppet to a host. The relevant log entries are attached with obfuscation of domain names/ips.

From what I can see, puppet is attempting to delete and recreate the entry even though they haven't changed and while this may be a change in puppet, I believe this is related to a annoyance I've seen already.

When I've created host using provisioning, there have been occasions where a mistake has been made causing me to delete and recreate the host. This fails because when it tries to add the Route 53 entry it fails with an error that the entry already exists. The work around is to add the entry through the Amazon console and the delete it again. The entry can then be created through the smart proxy. If this underlying issue was resolved I believe my current issue would go away. It appears to me to be the same cause.

I am now in a position where I have to manually go to the Amazon console and recreate entries every time I make a change to a host. This is highly impractical and defeats the whole point of using Foreman in the first place. While the longer term issue relating to route 53 is likely the piece failing, the unnecessary deletion and recreation of route 53 entries is what's causing me the problem. I'd appreciate some help.

Thanks

Apologies if this seems a little rambling.

foreman-route53.stack.trace.txt Magnifier (5.86 KB) Paul Smyth, 11/03/2016 06:12 AM

foreman-proxy.log.txt Magnifier (2.72 KB) Dmitri Dolguikh, 11/03/2016 07:42 AM

foreman-debug-stack-trace.txt Magnifier (31.1 KB) Paul Smyth, 11/03/2016 08:11 AM


Related issues

Related to Foreman - Refactor #15635: add tests for dns orchestration Closed 07/10/2016

Associated revisions

Revision b7cc39f8
Added by Dominic Cleal 9 months ago

fixes #17197 - don't update DNS on ip6 change to empty string

History

#1 Updated by Paul Smyth 9 months ago

The proxy.log hasn't been written to since October 18th

#2 Updated by Dominic Cleal 9 months ago

  • Category changed from Compute resources - EC2 to DNS
  • Priority changed from Urgent to Normal

#3 Updated by Dmitri Dolguikh 9 months ago

Relevant logs from proxy attached.

#4 Updated by Dominic Cleal 9 months ago

It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.

The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).

If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes? method (circa line 99) to:

def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end

And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.

#5 Updated by Paul Smyth 9 months ago

Hi, this has been downgraded to normal but this is an absolute show stopper. Every time we make a change to a puppet entry IT DELETES THE ROUTE53 DNS ENTRIES RELATING TO THE HOST, THIS IS NOT ACCEPTABLE

#6 Updated by Paul Smyth 9 months ago

Just seen your edit after that last post Dominc, will try it

#7 Updated by Paul Smyth 9 months ago

- It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.

Nope, no change to ip or host name

- The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).

Nope definitely not
- If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes? method (circa line 99) to:

def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end

And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.

Will do

#8 Updated by Paul Smyth 9 months ago

Further stack trace attached

#9 Updated by Dominic Cleal 9 months ago

  • Subject changed from Route 53 Entries being deleted on Puppet edit and update to DNS being recreated on host update when ip6 field changes from nil to empty
  • Release set to 1.13.2

Thank you, that confirms the theory above that the ip6 field is changing from nil to an empty string, causing the DNS records to be recreated.

2016-11-03T12:07:59 88341fd7 [app] [I] IPv4: "XXX.XXX.100.240" to "XXX.XXX.100.240" 
2016-11-03T12:07:59 88341fd7 [app] [I] IPv6: nil to ""
2016-11-03T12:07:59 88341fd7 [app] [I] Hostname: "jon-script-test-2.aws-eu-west.XXX.XXX" to "jon-script-test-2.aws-eu-west.XXX.XXX"

#10 Updated by Dominic Cleal 9 months ago

#11 Updated by Dominic Cleal 9 months ago

  • Status changed from New to Assigned
  • Assigned To set to Dominic Cleal

#12 Updated by Paul Smyth 9 months ago

Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?
2. Is it possible to configure Foreman to ignore ipv6 altogether?

#13 Updated by Dominic Cleal 9 months ago

Paul Smyth wrote:

Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?

The two sets of records are bundled together - all forward and reverse records recreated if there's a change.

2. Is it possible to configure Foreman to ignore ipv6 altogether?

No.

#14 Updated by Paul Smyth 9 months ago

So what you're saying that even though the ipv4 address is the same and still exists a change in the ipv6 address justifies removing the DNS entry for the host? Please advise on next steps

#15 Updated by Dominic Cleal 9 months ago

You've already filed the bug report, I've offered to fix it by assigning it to myself. It's likely that a fix will be in an upcoming patch release.

#16 Updated by The Foreman Bot 9 months ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3990 added

#17 Updated by Paul Smyth 9 months ago

Thanks Dominic

#18 Updated by Dominic Cleal 9 months ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF