Project

General

Profile

Bug #27160

Foreman fails to find IP address for boot server if main resolver is down

Added by Alexander Olofsson 7 months ago. Updated 8 days ago.

Status:
Closed
Priority:
Normal
Category:
DNS
Target version:
-
Difficulty:
Triaged:
Yes
Bugzilla link:

Description

Foreman seems to fail to correctly failover between the resolvers listed in /etc/resolv.conf, leading to failures to build hosts when the first listed resolver is unavailable.

The error;

Failed to set Build on host.example.com: ["failed to detect boot server: ERF50-9294 [Foreman::WrappedException]: Unable to find IP address for 'dhcp.example.com' ([Net::Error]: execution expired)", "Error connecting to 'example.com' domain DNS servers: ns2.example.com - check query_local_nameservers and dns_conflict_timeout settings"]

The backtrace;

failed to detect boot server: ERF50-9294 [Foreman::WrappedException]: Unable to find IP address for 'dhcp.example.com' ([Net::Error]: execution expired)
Foreman::WrappedException: ERF50-9294 [Foreman::WrappedException]: Unable to find IP address for 'dhcp.example.com' ([Net::Error]: execution expired)
/usr/share/foreman/app/services/nic_ip_resolver.rb:24:in `rescue in to_ip_address'
/usr/share/foreman/app/services/nic_ip_resolver.rb:15:in `to_ip_address'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:91:in `boot_server'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:126:in `dhcp_attrs'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:103:in `build_dhcp_record'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:30:in `block in dhcp_records'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:29:in `map'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:29:in `dhcp_records'
/usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:202:in `dhcp_conflict_detected?'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:517:in `each'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:517:in `invoke_after'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:133:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:816:in `_run_validation_callbacks'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activemodel-5.2.1/lib/active_model/validations/callbacks.rb:118:in `run_validations!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activemodel-5.2.1/lib/active_model/validations.rb:339:in `valid?'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-5.2.1/lib/active_record/validations.rb:67:in `valid?'
/usr/share/foreman/app/models/concerns/orchestration.rb:86:in `valid?'
/usr/share/foreman/app/models/host/managed.rb:935:in `trigger_nic_orchestration'
...


Related issues

Related to Foreman - Bug #27585: default dns timeout value is nilClosed

Associated revisions

Revision e75700fd (diff)
Added by Lukáš Zapletal 6 months ago

Fixes #27160 - do not use Ruby Timeout for DNS (#6861)

Revision 38a02a7b (diff)
Added by Ewoud Kohl van Wijngaarden 6 months ago

Refs #27160 - Update the DNS timeout description

This corrects the short descriptive name to match the internal name. It
also changes the description to indicate it's examples. This is
important because plugins can also use these functions.

History

#1 Updated by Lukas Zapletal 7 months ago

  • Triaged changed from No to Yes
  • Assignee set to Lukas Zapletal
  • Status changed from New to Assigned
  • Category set to DNS

Hello, we use Timeout Ruby class in our codebase which is not good approach. Actually Ruby DNS Resolver reads all "nameserver" entries and tries one after another, however the way this is implemented is that the default Ruby DNS timeout is high (5, 10, 20, 40 seconds) and it probably does not reply fast enough and the global timeout (in our codebase) fires off.

The solution is to remove the extra timeout we have there and rely only on the Ruby DNS resolver timeout. No idea why this wasn't there from the day one. Try the patch and let me know if it solved the problem.

#2 Updated by The Foreman Bot 7 months ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/6861 added

#3 Updated by The Foreman Bot 6 months ago

  • Fixed in Releases 1.24.0 added

#4 Updated by Anonymous 6 months ago

  • Status changed from Ready For Testing to Closed

#5 Updated by The Foreman Bot 6 months ago

  • Pull request https://github.com/theforeman/foreman/pull/6954 added

#6 Updated by Amir Fefer 5 months ago

  • Related to Bug #27585: default dns timeout value is nil added

#7 Updated by Amir Fefer 5 months ago

I think its very non-trivial for a user (like me) to understand how to use this setting.... also, this seems a lot like implementation details that leak into the settings page, and we should find a better way to represent it?

Also available in: Atom PDF