Bug #39406
open
ansible-callback service fails at boot due to DNS resolution failure
Description
Summary¶
On nodes provisioned by Foreman, ansible-callback.service fails to send a callback to AWX at OS boot time.
The root cause is that curl is executed before NetworkManager has finished writing DNS configuration to /etc/resolv.conf.
A fix is being prepared and will be submitted as a separate PR.
Details¶
Although ansible-callback.service specifies After=network-online.target, the boot order itself works correctly.
The problem is that DNS is not yet usable at the time network-online.target completes.
The root cause is the following chain of events:
1. /etc/resolv.conf write is delayed after network-online.target completes
NetworkManager selects rc-manager=symlink (auto), but since systemd-resolved is not installed,
there is no valid target for the symlink and DNS configuration cannot be applied immediately.
Jun 09 11:29:02 NetworkManager[818]: <info> dns-mgr: init: dns=default,systemd-resolved rc-manager=symlink (auto) Jun 09 11:29:02 dracut[1382]: dracut module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found! Jun 09 11:29:02 dracut[1382]: dracut module 'systemd-resolved' will not be installed, because command '/usr/lib/systemd/systemd-resolved' could not be found!
2. NetworkManager-dispatcher takes over DNS configuration, but takes approximately 10 seconds to complete
Although network-online.target completes at 11:29:02, NetworkManager-dispatcher.service continues running until 11:29:12 while it applies the DNS configuration.
Jun 09 11:29:02 systemd[1]: Finished Network Manager Wait Online. Jun 09 11:29:12 systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
3. ansible-callback executes curl before DNS configuration is applied and fails
ansible-callback.service starts immediately after network-online.target completes,
and curl is executed at 11:29:14 while NetworkManager-dispatcher is still applying DNS settings, resulting in a host resolution failure.
Jun 09 11:29:02 systemd[1]: Starting Provisioning callback to Ansible Tower... Jun 09 11:29:12 systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully. Jun 09 11:29:14 systemd[1]: ansible-callback.service: Main process exited, code=exited, status=6/NOTCONFIGURED Jun 09 11:29:14 systemd[1]: ansible-callback.service: Failed with result 'exit-code'.
This is also confirmed by systemd-analyze critical-chain, which shows ansible-callback.service failing 12 seconds after network-online.target completes.
ansible-callback.service +12.306s
└─ network-online.target @2.057s
└─ NetworkManager-wait-online.service @2.000s +55ms
Affected file:app/views/unattended/provisioning_templates/snippet/ansible_tower_callback_service.erb
Steps to reproduce¶
- Provision a node running RHEL/Rocky Linux 9 from Foreman
- Enable AWX integration parameters (
ansible_tower_api_url,ansible_job_template_id,ansible_host_config_key) - After provisioning completes, check the status of
ansible-callback.serviceon the first boot
systemctl status ansible-callback
Actual result¶
× ansible-callback.service - Provisioning callback to Ansible Tower
Loaded: loaded (/etc/systemd/system/ansible-callback.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Tue 2026-06-09 xx:xx:xx JST
Process: 853 ExecStart=/usr/bin/curl -k -s --data host_config_key=xxxxxx
https://awx-server.example.local:8043/api/v2/job_templates/xx/callback/
(code=exited, status=6)
Main PID: 853 (code=exited, status=6)
Jun 09 xx:xx:xx systemd[1]: Starting Provisioning callback to Ansible Tower...
Jun 09 xx:xx:xx systemd[1]: ansible-callback.service: Main process exited, code=exited, status=6/NOTCONFIGURED
Jun 09 xx:xx:xx systemd[1]: ansible-callback.service: Failed with result 'exit-code'.
Jun 09 xx:xx:xx systemd[1]: Failed to start Provisioning callback to Ansible Tower.
curl exits with code 6 (Could not resolve host) and the callback fails.
Expected result¶
curl is executed only after DNS resolution is available at boot time, and the callback to AWX succeeds.
Files
Updated by Yusuke Hirota about 1 month ago
- File 2026-06-09 135737.jpg 2026-06-09 135737.jpg added
A timeline diagram illustrating the root cause is attached to this issue for reference.
Updated by Yusuke Hirota 15 days ago
Correction of Root Cause¶
After further investigation, the initial root cause analysis (delayed writing to /etc/resolv.conf due to the absence of systemd-resolved) turned out to be incorrect. The true root cause was not on the Foreman side, but in the network configuration of the environment.
Environment¶
In this environment, the AWX server, Foreman, and the provisioning target node are all virtual machines (VMs), connected through a bridge (br-pxe) manually created on the physical host that runs these VMs.
True Root Cause¶
The bridge (br-pxe) on the physical host to which the VMs are connected had STP (Spanning Tree Protocol) enabled.
stp_state 1 forward_delay 1500
When STP is enabled, a bridge port has a waiting period after startup before it actually begins forwarding packets. During this waiting period, packets sent immediately after the VM boots are discarded by the bridge.
As a result, the DNS query sent by ansible-callback.service on the provisioning target VM at boot time was discarded by the bridge on the physical host, and curl failed with exit code 6 (Could not resolve host).
Why STP is Enabled and forward-delay is 15 seconds¶
br-pxe is managed by NetworkManager, and was configured as follows:
bridge.stp: yes bridge.forward-delay: 15
These are the default values of NetworkManager's bridge settings. According to the NetworkManager reference manual, the default for stp is TRUE (enabled), and the default for forward-delay is 15 (seconds).
- NetworkManager Reference Manual - bridge settings
https://www.networkmanager.dev/docs/api/latest/settings-bridge.html
Note¶
The reason a manual execution succeeds after some time has passed since OS boot is that, by that point, the bridge has reached the forwarding state.
Workaround¶
This can be temporarily avoided by disabling STP on the bridge of the physical host to which the provisioning target VM is connected.
nmcli connection modify br-pxe bridge.stp no nmcli connection up br-pxe
Possible Improvement on the Foreman Side¶
This issue is not a defect in the Foreman template. However, we believe there is room for improvement on the Foreman side so that it works reliably even in environments like this.
The Foreman "Kickstart default" template includes a script that starts ansible-callback.service as a one-shot service, which runs curl to send a callback to the AWX server. For example, by adding a mechanism that waits until the DNS server becomes reachable and retries curl before executing it, the callback could be made to succeed even in situations where the network is temporarily unavailable, such as during the STP waiting period of a bridge.
Affected file:app/views/unattended/provisioning_templates/snippet/ansible_tower_callback_service.erb
Applicability to Physical Machines and Physical Switches¶
This issue is not specific to VMs. The same problem can occur with physical machines connected to physical switches.
The STP forwarding delay (approximately 30 seconds with traditional 802.1D) is standard behavior for physical switches as well. When a physical server boots and attempts network communication such as DHCP or DNS resolution before the switch port reaches the forwarding state, the same failure can occur.
Cisco's documentation describes this delay as follows:
Source:That convergence delay, roughly 30 seconds with traditional 802.1D, is appropriate for switch-to-switch links but unnecessary for ports connected to end hosts such as PCs, servers, or printers, which cannot create a switching loop on their own.
- Understand Spanning Tree PortFast and BPDU Guard Features - Cisco
https://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/10586-65.html
Because this delay can occur regardless of whether the environment is virtual or physical, the improvement suggested above on the Foreman side (waiting until the network is reachable and retrying curl before sending the callback) would be effective as a general-purpose measure that does not depend on the network configuration of the environment.
Updated by The Foreman Bot 5 days ago
- Status changed from New to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/11108 added