Project

General

Profile

Actions

Bug #25246

closed

IPv6 record info destroyed on fact importing

Added by Arend Lapere about 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Difficulty:
Triaged:
Yes
Fixed in Releases:
Found in Releases:

Description

Short story:
So, it appears I can't rebuild or edit hosts when the IPv6 record is broken. I get an "Failed to enable <fqdn> for installation: ["Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]".

Long story:
I created a virtual host, for which I've configured an IPv6 subnet (set to use EUI-64). Usually this remains empty (as the compute resource didn't assign a MAC address yet), untill I save it, after which (if I'm lucky) the EUI-64 is calculated and assigned. All is (quite) well up untill here.

However, at some point (still figuring out what that point in time is, will update this issue as soon as I figured it out), the IPv6 information is simply lost, no mention of it's existing anywhere. If I look at my domain db, then there's still an AAAA record assigned for that host. However, when rebuilding, it complains:
"Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]"

If I try to edit my host, and simply remove the IPv6 (as a work-around), it always tries to delete the IPv6 record (which makes sense) but this fails with the same error:
"Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass"]"

Looking at the YAML, the information regarding my IPv6 address, however, is still there, but it doesn't show up in the UI.

The foreman production logs show this:

2018-10-19T11:16:38 [W|app|e3f29] Remove IPv6 DNS record for <fqdn> task failed with the following error: undefined method `destroy' for nil:NilClass
NoMethodError: undefined method `destroy' for nil:NilClass
/usr/share/foreman/app/models/concerns/dns_interface.rb:51:in `del_dns_record'
/usr/share/foreman/app/models/concerns/orchestration.rb:221:in `execute'
/usr/share/foreman/app/models/concerns/orchestration.rb:145:in `block in process'
/usr/share/foreman/app/models/concerns/orchestration.rb:137:in `each'
/usr/share/foreman/app/models/concerns/orchestration.rb:137:in `process'
/usr/share/foreman/app/models/concerns/orchestration.rb:44:in `around_save_orchestration'

Work around: last time I had to destroy and recreate my host altogether

Although it would be one thing to be more forgiving if the record is non-existing, I'd actually like to try and figure out what the root-cause is before trying to ignore erroneous situations.

Actions #1

Updated by Arend Lapere about 6 years ago

  • Description updated (diff)
Actions #2

Updated by Arend Lapere about 6 years ago

  • Description updated (diff)
Actions #3

Updated by Arend Lapere about 6 years ago

  • Description updated (diff)
Actions #4

Updated by Lukas Zapletal about 6 years ago

  • Triaged changed from No to Yes

Hey, looking at the code there are couple of requirements you need to met in order to orchestrate the change:

SETTINGS[:unattended] && (host.nil? || host.managed?) && managed? && hostname.present?
dns_ready? && ip6_available? && domain.present? && domain.proxy.present?
dns_ready? && ip6_available? && subnet6.present? && subnet6.dns?

Basically:

  • host is managed
  • host has a hostname
  • NIC has IPv6 address filled in
  • NIC has a subnet with IPv6 info associated
  • NIC has a domain associated
Actions #5

Updated by Arend Lapere about 6 years ago

Acually, I comply to all those pre-requisites. I've compared the YAML from a correctly working host (so, one I can rebuild) compared to the failing host and it doesn't appear that there are any differences, however, the UI behaves differently:
- Both hosts have an IPv6 address set in YAML but the failing host doesn't show it in the UI at all
- The IPv6 subnet is correctly showing up for both hosts, however, trying to remove the IPv6 subnet never works, it keeps reverting the removal and so, a subnet is always added.

Tonight I'm going to deep-dive into the code to really try and understand why the UI fails, by this I hope to understand why the rest if failing. Perhaps I've configured something incorrectly somewhere but at this point it's not apparent.

Will keep you guys posted!

Actions #6

Updated by Arend Lapere about 6 years ago

Okay, so I managed to "overcome" the problem by doing the following:

  def del_dns_record(type)
    return true if dns_record(type).nil?
    dns_record(type).destroy
  end

So basically the code becomes a bit more forgiving, however, this does not yet gives me the root cause of why the IPv6 was removed to begin with. I long suspected the rebuild config, as it seems to destroy all my DNS records (A & AAAA records AFAIK) but never brings them back. However, the ip6 field is never cleared by that as well. Still looking at what could've done this...

I found why the build_config fails to do as expected: my DNS records are replicated to a master DNS server, however this DNS server did not remove the records if I deleted them in the slave (no notifications were set-up). Because of this, it seems somewhere in code, the records are tested for existence in another server and thus it isn't recreated.

I suspect that my current system-architecture, in which I don't notify my DNS master but let a record expire instead, probably destroyed the ip6 record and so on. Will look into where in the code this DNS resolve is being done instead of a record existence on the local DNS server and will try to fix that as a sub-optimal DNS architecture shouldn't affect the recreation of my DNS records.

Actions #7

Updated by Arend Lapere about 6 years ago

So, currently I've manage to overcome the DNS recreate issue by naivly always deleting the DNS records and recreating them.

I've noticed that there are more issues related to this:
https://projects.theforeman.org/issues/13696
https://projects.theforeman.org/issues/2668

Will look if there's a less naive but equally qualitative patch that can be made for this. Again, I haven't managed to recreate the situation that triggered my original issue; so I'll keep focussing on that.

Actions #8

Updated by Arend Lapere about 6 years ago

So, I've finally found the root-cause! My ansible callback seems to remove the IPv6 address, so now this should be trivial to fix.
Expect a few pull-requests to be posted in the next few days including tests.

Sorry for the monologue but just wanted to keep everyone in the loop on my research (in case someone had some pointers)

Actions #9

Updated by Arend Lapere about 6 years ago

  • Project changed from Foreman to Ansible
  • Subject changed from Can't rebuild due to non-existing IPv6 record to IPv6 record info destroyed on fact importing
  • Category deleted (DNS)

Changed this to Ansible project as the error would seem to be in the way Ansible handles facts parsing.

Actions #10

Updated by The Foreman Bot about 6 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman_ansible/pull/219 added
Actions #11

Updated by Marek Hulán about 6 years ago

  • Fixed in Releases foreman-ansible 2.3.0 added
Actions #12

Updated by Anonymous about 6 years ago

  • Status changed from Ready For Testing to Closed
Actions

Also available in: Atom PDF