Project

General

Profile

Actions

Bug #17197

closed

DNS being recreated on host update when ip6 field changes from nil to empty

Added by Paul Smyth over 7 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
DNS
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

Since the most recent update of Foreman to 1.13.1 route 53 entries are being deleted whenever I make a change via puppet to a host. The relevant log entries are attached with obfuscation of domain names/ips.

From what I can see, puppet is attempting to delete and recreate the entry even though they haven't changed and while this may be a change in puppet, I believe this is related to a annoyance I've seen already.

When I've created host using provisioning, there have been occasions where a mistake has been made causing me to delete and recreate the host. This fails because when it tries to add the Route 53 entry it fails with an error that the entry already exists. The work around is to add the entry through the Amazon console and the delete it again. The entry can then be created through the smart proxy. If this underlying issue was resolved I believe my current issue would go away. It appears to me to be the same cause.

I am now in a position where I have to manually go to the Amazon console and recreate entries every time I make a change to a host. This is highly impractical and defeats the whole point of using Foreman in the first place. While the longer term issue relating to route 53 is likely the piece failing, the unnecessary deletion and recreation of route 53 entries is what's causing me the problem. I'd appreciate some help.

Thanks

Apologies if this seems a little rambling.


Files

foreman-route53.stack.trace.txt foreman-route53.stack.trace.txt 5.86 KB Paul Smyth, 11/03/2016 06:12 AM
foreman-proxy.log.txt foreman-proxy.log.txt 2.72 KB Anonymous, 11/03/2016 07:42 AM
foreman-debug-stack-trace.txt foreman-debug-stack-trace.txt 31.1 KB Paul Smyth, 11/03/2016 08:11 AM

Related issues 1 (0 open1 closed)

Related to Foreman - Refactor #15635: add tests for dns orchestrationClosedTimo Goebel07/10/2016Actions
Actions #1

Updated by Paul Smyth over 7 years ago

The proxy.log hasn't been written to since October 18th

Actions #2

Updated by Dominic Cleal over 7 years ago

  • Category changed from Compute resources - EC2 to DNS
  • Priority changed from Urgent to Normal
Actions #3

Updated by Anonymous over 7 years ago

Relevant logs from proxy attached.

Actions #4

Updated by Dominic Cleal over 7 years ago

It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.

The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).

If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes? method (circa line 99) to:

def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end

And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.

Actions #5

Updated by Paul Smyth over 7 years ago

Hi, this has been downgraded to normal but this is an absolute show stopper. Every time we make a change to a puppet entry IT DELETES THE ROUTE53 DNS ENTRIES RELATING TO THE HOST, THIS IS NOT ACCEPTABLE

Actions #6

Updated by Paul Smyth over 7 years ago

Just seen your edit after that last post Dominc, will try it

Actions #7

Updated by Paul Smyth over 7 years ago

- It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.

Nope, no change to ip or host name

- The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).

Nope definitely not
- If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes? method (circa line 99) to:

def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end

And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.

Will do

Actions #8

Updated by Paul Smyth over 7 years ago

Further stack trace attached

Actions #9

Updated by Dominic Cleal over 7 years ago

  • Subject changed from Route 53 Entries being deleted on Puppet edit and update to DNS being recreated on host update when ip6 field changes from nil to empty
  • translation missing: en.field_release set to 203

Thank you, that confirms the theory above that the ip6 field is changing from nil to an empty string, causing the DNS records to be recreated.

2016-11-03T12:07:59 88341fd7 [app] [I] IPv4: "XXX.XXX.100.240" to "XXX.XXX.100.240" 
2016-11-03T12:07:59 88341fd7 [app] [I] IPv6: nil to ""
2016-11-03T12:07:59 88341fd7 [app] [I] Hostname: "jon-script-test-2.aws-eu-west.XXX.XXX" to "jon-script-test-2.aws-eu-west.XXX.XXX"
Actions #10

Updated by Dominic Cleal over 7 years ago

Actions #11

Updated by Dominic Cleal over 7 years ago

  • Status changed from New to Assigned
  • Assignee set to Dominic Cleal
Actions #12

Updated by Paul Smyth over 7 years ago

Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?
2. Is it possible to configure Foreman to ignore ipv6 altogether?

Actions #13

Updated by Dominic Cleal over 7 years ago

Paul Smyth wrote:

Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?

The two sets of records are bundled together - all forward and reverse records recreated if there's a change.

2. Is it possible to configure Foreman to ignore ipv6 altogether?

No.

Actions #14

Updated by Paul Smyth over 7 years ago

So what you're saying that even though the ipv4 address is the same and still exists a change in the ipv6 address justifies removing the DNS entry for the host? Please advise on next steps

Actions #15

Updated by Dominic Cleal over 7 years ago

You've already filed the bug report, I've offered to fix it by assigning it to myself. It's likely that a fix will be in an upcoming patch release.

Actions #16

Updated by The Foreman Bot over 7 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3990 added
Actions #17

Updated by Paul Smyth over 7 years ago

Thanks Dominic

Actions #18

Updated by Dominic Cleal over 7 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF