Bug #17197
closedDNS being recreated on host update when ip6 field changes from nil to empty
Description
Since the most recent update of Foreman to 1.13.1 route 53 entries are being deleted whenever I make a change via puppet to a host. The relevant log entries are attached with obfuscation of domain names/ips.
From what I can see, puppet is attempting to delete and recreate the entry even though they haven't changed and while this may be a change in puppet, I believe this is related to a annoyance I've seen already.
When I've created host using provisioning, there have been occasions where a mistake has been made causing me to delete and recreate the host. This fails because when it tries to add the Route 53 entry it fails with an error that the entry already exists. The work around is to add the entry through the Amazon console and the delete it again. The entry can then be created through the smart proxy. If this underlying issue was resolved I believe my current issue would go away. It appears to me to be the same cause.
I am now in a position where I have to manually go to the Amazon console and recreate entries every time I make a change to a host. This is highly impractical and defeats the whole point of using Foreman in the first place. While the longer term issue relating to route 53 is likely the piece failing, the unnecessary deletion and recreation of route 53 entries is what's causing me the problem. I'd appreciate some help.
Thanks
Apologies if this seems a little rambling.
Files
Updated by Paul Smyth about 8 years ago
The proxy.log hasn't been written to since October 18th
Updated by Dominic Cleal about 8 years ago
- Category changed from Compute resources - EC2 to DNS
- Priority changed from Urgent to Normal
Updated by Anonymous about 8 years ago
- File foreman-proxy.log.txt foreman-proxy.log.txt added
Relevant logs from proxy attached.
Updated by Dominic Cleal about 8 years ago
It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.
The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).
If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes?
method (circa line 99) to:
def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end
And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.
Updated by Paul Smyth about 8 years ago
Hi, this has been downgraded to normal but this is an absolute show stopper. Every time we make a change to a puppet entry IT DELETES THE ROUTE53 DNS ENTRIES RELATING TO THE HOST, THIS IS NOT ACCEPTABLE
Updated by Paul Smyth about 8 years ago
Just seen your edit after that last post Dominc, will try it
Updated by Paul Smyth about 8 years ago
- It's likely that pending_dns_record_changes? is true in the DNS orchestration when the host is updated.
Nope, no change to ip or host name
- The only way I see for that to happen is either for the IPv4 address, IPv6 address or hostname to have changed - though the log doesn't indicate any hostname or v4 change, or for the empty IPv6 field to change from NULL to "" or similar (it's not robustly checking for a change).
Nope definitely not
- If you're willing to add debug to your instance, please edit ~foreman/app/models/concerns/orchestration/dns.rb and change the pending_dns_record_changes? method (circa line 99) to:
def pending_dns_record_changes?
Rails.logger.info("IPv4: #{old.ip.inspect} to #{ip.inspect}")
Rails.logger.info("IPv6: #{old.ip6.inspect} to #{ip6.inspect}")
Rails.logger.info("Hostname: #{old.hostname.inspect} to #{hostname.inspect}")
old.ip != ip || old.ip6 != ip6 || old.hostname != hostname
end
And restart Foreman (Apache/httpd). You should see three additional log lines when updating the host.
Will do
Updated by Paul Smyth about 8 years ago
Further stack trace attached
Updated by Dominic Cleal about 8 years ago
- Subject changed from Route 53 Entries being deleted on Puppet edit and update to DNS being recreated on host update when ip6 field changes from nil to empty
- Translation missing: en.field_release set to 203
Thank you, that confirms the theory above that the ip6 field is changing from nil to an empty string, causing the DNS records to be recreated.
2016-11-03T12:07:59 88341fd7 [app] [I] IPv4: "XXX.XXX.100.240" to "XXX.XXX.100.240"
2016-11-03T12:07:59 88341fd7 [app] [I] IPv6: nil to ""
2016-11-03T12:07:59 88341fd7 [app] [I] Hostname: "jon-script-test-2.aws-eu-west.XXX.XXX" to "jon-script-test-2.aws-eu-west.XXX.XXX"
Updated by Dominic Cleal about 8 years ago
- Related to Refactor #15635: add tests for dns orchestration added
Updated by Dominic Cleal about 8 years ago
- Status changed from New to Assigned
- Assignee set to Dominic Cleal
Updated by Paul Smyth about 8 years ago
Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?
2. Is it possible to configure Foreman to ignore ipv6 altogether?
Updated by Dominic Cleal about 8 years ago
Paul Smyth wrote:
Thanks Dominic. Two questions:
1. Why does a change of ipv6 with no change of ipv4 result in deletion of the DNS entry?
The two sets of records are bundled together - all forward and reverse records recreated if there's a change.
2. Is it possible to configure Foreman to ignore ipv6 altogether?
No.
Updated by Paul Smyth about 8 years ago
So what you're saying that even though the ipv4 address is the same and still exists a change in the ipv6 address justifies removing the DNS entry for the host? Please advise on next steps
Updated by Dominic Cleal about 8 years ago
You've already filed the bug report, I've offered to fix it by assigning it to myself. It's likely that a fix will be in an upcoming patch release.
Updated by The Foreman Bot about 8 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/3990 added
Updated by Dominic Cleal about 8 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset b7cc39f867cc4092cebd173ee3dec05202a96321.