Bug #25246
closedIPv6 record info destroyed on fact importing
Description
Short story:
So, it appears I can't rebuild or edit hosts when the IPv6 record is broken. I get an "Failed to enable <fqdn> for installation: ["Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]".
Long story:
I created a virtual host, for which I've configured an IPv6 subnet (set to use EUI-64). Usually this remains empty (as the compute resource didn't assign a MAC address yet), untill I save it, after which (if I'm lucky) the EUI-64 is calculated and assigned. All is (quite) well up untill here.
However, at some point (still figuring out what that point in time is, will update this issue as soon as I figured it out), the IPv6 information is simply lost, no mention of it's existing anywhere. If I look at my domain db, then there's still an AAAA record assigned for that host. However, when rebuilding, it complains:
"Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]"
If I try to edit my host, and simply remove the IPv6 (as a work-around), it always tries to delete the IPv6 record (which makes sense) but this fails with the same error:
"Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]"
Looking at the YAML, the information regarding my IPv6 address, however, is still there, but it doesn't show up in the UI.
The foreman production logs show this:
2018-10-19T11:16:38 [W|app|e3f29] Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass NoMethodError: undefined method `destroy' for nil:NilClass /usr/share/foreman/app/models/concerns/dns_interface.rb:51:in `del_dns_record' /usr/share/foreman/app/models/concerns/orchestration.rb:221:in `execute' /usr/share/foreman/app/models/concerns/orchestration.rb:145:in `block in process' /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `each' /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `process' /usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'
Work around: last time I had to destroy and recreate my host altogether
Although it would be one thing to be more forgiving if the record is non-existing, I'd actually like to try and figure out what the root-cause is before trying to ignore erroneous situations.
Updated by Lukas Zapletal about 6 years ago
- Triaged changed from No to Yes
Hey, looking at the code there are couple of requirements you need to met in order to orchestrate the change:
SETTINGS[:unattended] && (host.nil? || host.managed?) && managed? && hostname.present? dns_ready? && ip6_available? && domain.present? && domain.proxy.present? dns_ready? && ip6_available? && subnet6.present? && subnet6.dns?
Basically:
- host is managed
- host has a hostname
- NIC has IPv6 address filled in
- NIC has a subnet with IPv6 info associated
- NIC has a domain associated
Updated by Arend Lapere about 6 years ago
Acually, I comply to all those pre-requisites. I've compared the YAML from a correctly working host (so, one I can rebuild) compared to the failing host and it doesn't appear that there are any differences, however, the UI behaves differently:
- Both hosts have an IPv6 address set in YAML but the failing host doesn't show it in the UI at all
- The IPv6 subnet is correctly showing up for both hosts, however, trying to remove the IPv6 subnet never works, it keeps reverting the removal and so, a subnet is always added.
Tonight I'm going to deep-dive into the code to really try and understand why the UI fails, by this I hope to understand why the rest if failing. Perhaps I've configured something incorrectly somewhere but at this point it's not apparent.
Will keep you guys posted!
Updated by Arend Lapere about 6 years ago
Okay, so I managed to "overcome" the problem by doing the following:
def del_dns_record(type) return true if dns_record(type).nil? dns_record(type).destroy end
So basically the code becomes a bit more forgiving, however, this does not yet gives me the root cause of why the IPv6 was removed to begin with. I long suspected the rebuild config, as it seems to destroy all my DNS records (A & AAAA records AFAIK) but never brings them back. However, the ip6 field is never cleared by that as well. Still looking at what could've done this...
I found why the build_config fails to do as expected: my DNS records are replicated to a master DNS server, however this DNS server did not remove the records if I deleted them in the slave (no notifications were set-up). Because of this, it seems somewhere in code, the records are tested for existence in another server and thus it isn't recreated.
I suspect that my current system-architecture, in which I don't notify my DNS master but let a record expire instead, probably destroyed the ip6 record and so on. Will look into where in the code this DNS resolve is being done instead of a record existence on the local DNS server and will try to fix that as a sub-optimal DNS architecture shouldn't affect the recreation of my DNS records.
Updated by Arend Lapere about 6 years ago
So, currently I've manage to overcome the DNS recreate issue by naivly always deleting the DNS records and recreating them.
I've noticed that there are more issues related to this:
https://projects.theforeman.org/issues/13696
https://projects.theforeman.org/issues/2668
Will look if there's a less naive but equally qualitative patch that can be made for this. Again, I haven't managed to recreate the situation that triggered my original issue; so I'll keep focussing on that.
Updated by Arend Lapere about 6 years ago
So, I've finally found the root-cause! My ansible callback seems to remove the IPv6 address, so now this should be trivial to fix.
Expect a few pull-requests to be posted in the next few days including tests.
Sorry for the monologue but just wanted to keep everyone in the loop on my research (in case someone had some pointers)
Updated by Arend Lapere about 6 years ago
- Project changed from Foreman to Ansible
- Subject changed from Can't rebuild due to non-existing IPv6 record to IPv6 record info destroyed on fact importing
- Category deleted (
DNS)
Changed this to Ansible project as the error would seem to be in the way Ansible handles facts parsing.
Updated by The Foreman Bot about 6 years ago
- Status changed from New to Ready For Testing
- Pull request https://github.com/theforeman/foreman_ansible/pull/219 added
Updated by Marek Hulán about 6 years ago
- Fixed in Releases foreman-ansible 2.3.0 added
Updated by Anonymous about 6 years ago
- Status changed from Ready For Testing to Closed
Applied in changeset foreman_ansible|28862d3b6d924118bdc601f45a45617e33ae160b.