Project

General

Profile

Actions

Bug #24918

open

After upgrade to 1.19, Creating host results in error: Errno::EHOSTUNREACH: No route to host - recvfrom(2)

Added by Jason Harris over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Orchestration
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

Foreman OS: CentOS 7
DNS / DHCP are local to Foreman
Firewalld is disabled (for debugging)
oVirt integration is present

This was working fine before update to 1.19. After the update, I did have to re-create the local smart proxy configurations for DHCP / DNS so this could entirely (and probably is) a configuration problem on my end.

When creating a new host (and likely in other areas), the following error occurs:

Errno::EHOSTUNREACH: No route to host - recvfrom(2)

The error logs don't exactly help in tracking down the actual problem, and Google searches just point me to random Ruby programming tips, so I am opening a ticket to hopefully get pointed in the right direction.


Files

foreman_error.txt foreman_error.txt 17.7 KB Full Trace Log Jason Harris, 09/12/2018 03:38 PM
production.log production.log 1.15 MB Jason Harris, 09/12/2018 03:39 PM
Actions #1

Updated by Tomer Brisker over 5 years ago

  • Category set to Orchestration
  • Target version set to 1.19.1
  • Found in Releases 1.19.0 added
Actions #2

Updated by Marek Hulán over 5 years ago

  • Status changed from New to Need more information

it seems the DNS resolution didn't work. This could be caused by #18765 that was introduced in 1.19. But looking at the trace, it seems the error is raised sooner, during DNS orchestration. Foreman tries to check there's no existing DNS record before it creates new. It uses local DNS resolver for this. Can you check /etc/resolv.conf is configured correctly on your Foreman host? Also does host whatever.host.your.building work fine on this machine? Perhaps if you ran foreman-installer on upgrade, it might have overridden something.

Actions #3

Updated by Jason Harris over 5 years ago

Marek Hulán wrote:

it seems the DNS resolution didn't work. This could be caused by #18765 that was introduced in 1.19. But looking at the trace, it seems the error is raised sooner, during DNS orchestration. Foreman tries to check there's no existing DNS record before it creates new. It uses local DNS resolver for this. Can you check /etc/resolv.conf is configured correctly on your Foreman host? Also does host whatever.host.your.building work fine on this machine? Perhaps if you ran foreman-installer on upgrade, it might have overridden something.

Thanks for the response. DNS was the first thing I checked, and it is working ok. I can't do a host newhost.fqdn because Foreman's DNS smart proxy is configured to take care of that. However, DNS is working fine on the local system (Foreman and DNS are on the same system, as well as DHCP, and I am also using that Smart Proxy).

I double-checked the smart proxy documentation and it seems I have everything in order. I just am stumped because I can't trace the error message through Ruby's logic (not a Ruby dev by any means) and I didn't see any other warnings or errors in the logs.

Here is some output from the local system.

[root@foreman ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search rivalab.geocomputing.net
nameserver 192.168.110.129
nameserver 192.168.100.104

[root@foreman ~]# nslookup 192.168.110.129
Server:         192.168.110.129
Address:        192.168.110.129#53

129.110.168.192.in-addr.arpa    name = foreman.rivalab.geocomputing.net.

[root@foreman ~]# host foreman
foreman.rivalab.geocomputing.net has address 192.168.110.129

[root@foreman settings.d]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

[root@foreman settings.d]# cat dns.yml dns_nsupdate.yml
---
# DNS management
:enabled: yes
# valid providers:
#   dns_dnscmd (Microsoft Windows native implementation)
#   dns_nsupdate
#   dns_nsupdate_gss (for GSS-TSIG support)
#   dns_libvirt (dnsmasq via libvirt)
:use_provider: dns_nsupdate
# use this setting if you want to override default TTL setting (86400)
:dns_ttl: 86400
---
#
# Configuration file for 'nsupdate' dns provider
#

:dns_key: /etc/rndc.key
# use this setting if you are managing a dns server which is not localhost though this proxy
:dns_server: 127.0.0.1
Actions #4

Updated by Lukas Zapletal over 5 years ago

  • Tracker changed from Bug to Support

Beware, Foreman does NOT use local resolver until you told it to do. By default, it reaches out to managed DNS server, so you need to make sure the entry is there. Or set to resolve locally via "query_local_nameservers" setting in Administer - Provisioning.

These kind of inquiries belong to our community forum, there's faster response time as well.

Actions #5

Updated by Jason Harris over 5 years ago

Lukas Zapletal wrote:

Beware, Foreman does NOT use local resolver until you told it to do. By default, it reaches out to managed DNS server, so you need to make sure the entry is there. Or set to resolve locally via "query_local_nameservers" setting in Administer - Provisioning.

These kind of inquiries belong to our community forum, there's faster response time as well.

I don't understand what you mean.

As has been stated already, the managed DNS server is running locally on the foreman server. Foreman's smart-proxy DNS feature is operational and handles the dynamic creation of new system names. This is the entire point of the smart-proxy. Why would I need to manually add the new system to the smart-proxy managed DNS server? Is the smart proxy DNS service no longer a feature of Foreman? If so, is this regression documented somewhere?

Also, I didn't arbitrarily create this ticket on this board. I was directed to do so by the error message from Ruby from within Foreman. If there is a community board somewhere, then I will be happy to submit this query there.

Actions #6

Updated by Marek Hulán over 5 years ago

  • Tracker changed from Support to Bug
  • Status changed from Need more information to New

Lukas are you sure? I don't see any conditions for this

https://github.com/theforeman/foreman/blob/develop/app/models/concerns/orchestration/dns.rb#L6
https://github.com/theforeman/foreman/blob/develop/app/models/concerns/orchestration/dns.rb#L110-L132
https://github.com/theforeman/foreman/blob/develop/lib/net/dns/forward_record.rb#L20-L23

I think this was a long term issue that Foreman tries to detect conflicts even though it might not be on the same network as proxy's DNS is

But I have no idea what changed in this regard in 1.19, it's reported to work before in the same env.

Actions #7

Updated by Tomer Brisker over 5 years ago

  • Target version changed from 1.19.1 to 961

1.19.1 release is in progress and this was not fixed in time, pushing out to 1.19.2.

Actions #8

Updated by Tomer Brisker over 5 years ago

  • Target version deleted (961)

not sure if this is a bug or a configuration error, but for the record, the community support board is at https://community.theforeman.org/

Actions

Also available in: Atom PDF