Project

General

Profile

Bug #13696

Rebuild Config deletes PTR records but does not add them

Added by Roger Spencer over 4 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
DNS
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Select a host in the all hosts section and choosing "Rebuild Config" fails every time with DNS failure. Foreman and Proxy logs show the deletion of the PTR and A records but only shows the A record being added after. Unsure why Foreman reports an error because it doesn't log anything about an error.

Using dns_nsupdate_gss on the proxy. Both Foreman and Proxy are at version 1.10.1. DNS is AD hosted. Proxy running on CentOS 6 & 7 (tried both OSs).

Logs (identifying data masked/changed)

Proxy log

I, [2016-02-13T16:14:11.437756 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.549343 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "DELETE /som-deploy1.humedica.net HTTP/1.1" 200 - 0.1123

I, [2016-02-13T16:14:11.599534 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.707506 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "DELETE /101.68.30.172.in-addr.arpa HTTP/1.1" 200 - 0.1085

I, [2016-02-13T16:14:11.797363 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.885362 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "POST / HTTP/1.1" 200 - 0.0891

Foreman Log

 | Started POST "/hosts/submit_rebuild_config?host_ids%5B%5D=618" for 172.XX.X.XX at 2016-02-13 16:14:09 -0500
 2016-02-13 16:14:09 [app] [I] Processing by HostsController#submit_rebuild_config as HTML
 2016-02-13 16:14:09 [app] [I]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"lrJWHLkduis0PMavAKLJDxU3Rohwu/XbAIdXXZ5N25s=", "host_ids"=>["618"]}
 2016-02-13 16:14:09 [app] [I] Delete DHCP reservation for servername.humedica.net-99:60:56:b0:46:9d/172.XX.XX.XXX
 2016-02-13 16:14:10 [app] [I] Create DHCP reservation for servername.humedica.net-99:60:56:b0:46:9d/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Delete the DNS A record for servername.humedica.net/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Delete the DNS PTR record for 172.XX.XX.XXX/servername.humedica.net
 2016-02-13 16:14:11 [app] [I] Add DNS A record for servername.humedica.net/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Redirected to https://foreman.domain.net/hosts
 2016-02-13 16:14:11 [app] [I] Completed 302 Found in 2858ms (ActiveRecord: 2.9ms)
 2016-02-13 16:14:12 [app] [I]

Changing host information in the Edit Host section works as expected. Changing IP or host name (or both) and both A and PTR records are updated properly.


Related issues

Related to Foreman - Bug #2668: Foreman appears to be incorrectly checking the local resolvers rather than SOANew2013-06-14

History

#1 Updated by Dominic Cleal over 4 years ago

  • Category set to DNS

I think this might be the effect of #2668, where Foreman only uses the caching resolver to check for PTR records, but it uses the SOA for A records.

The rebuild will check if the record's already there before trying to create it, so it might see a cached record (even after deletion) and skip the step.

#2 Updated by Dominic Cleal over 4 years ago

  • Related to Bug #2668: Foreman appears to be incorrectly checking the local resolvers rather than SOA added

#3 Updated by Alex Fisher about 4 years ago

I've also found that sometimes A records aren't created either. The problem for me seems to be that the smart proxy updates DNS via one DNS server, but the changes aren't replicated (via AD) to all the other DNS servers foreman has in its resolv.conf until several seconds to a few minutes later.

Reording the nameserver entries in resolv.conf has worked so far. I'm not sure what happens if DNS is temporarily down on the 'primary' server though. The resolver tries the DNS servers in the order they appear in the resolv.conf, but after it moves on to using the 2nd or 3rd entry does it ever switch back to using the first again?

#4 Updated by Alexander Olofsson over 3 years ago

Dominic Cleal wrote:

I think this might be the effect of #2668, where Foreman only uses the caching resolver to check for PTR records, but it uses the SOA for A records.

The rebuild will check if the record's already there before trying to create it, so it might see a cached record (even after deletion) and skip the step.

Just ran into this problem as well, when re-orchestrating a host the changes don't quite manage to propagate from the Infoblox master to the SOA NSes quick enough for foreman.
Which meant dropped PTR records, even losing several A and AAAA records for a few machines.

Is there any particular reason why Foreman doesn't use the naive workflow of "Delete, and then always create anew"?

In foreman/app/models/concerns/dns_interface.rb on line 37;

  def recreate_dns_record(type)
    return true if dns_record(type).nil? || dns_record(type).valid?
    set_dns_record(type)
  end

Wouldn't it be possible to just skip the second check? Since the function is only ever called right after - successfully - deleting the record in question.

Also available in: Atom PDF