How to provide enough info to get help?

Foreman is pretty complex software project with many components and extensive configuration. Always try to describe best your specific case (OS, version, ruby and foreman version, database, specific configuration). The foreman-debug script can collect all information and also filters out things like passwords or tokens. It is installed by default but the foreman-debug bash script can be used directly from git as well.

ERF error code

If you see an ERF error code, such as ERF12-7740, look it up on ErrorCodes.

Unable to save: failed to detect boot server: getaddrinfo: Name or service not known

if you have registered your smart-proxys via name, and these name are not resolvable, you get this error, you can add the name to your dns server or set
:tftp_servername: in smart-proxy settings to the ip of the smart-proxy.

I use puppet enterprise, what now?

Using Puppet Enterprise with Foreman is untested. It may work but there are no instructions on how to set it up.

error: Entry 'some file' not uptodate. Cannot merge.

If you downloaded Foreman from source (git), it could be that some of the files you have modified (or were modified automatically) conflicts with another file Foreman wants to upgrade.

If you don't think that you edited this file (e.g. db/schema.rb) it is usually safe to do:

git checkout 'some file'

This will revert the file to its previous condition (as in the repo at the time of checkout) and now you should be able to get the latest version by:

git pull

Strange errors with passenger

Passenger executes foreman, based on the owner of the config/environments.rb file, make sure that this use can actually access:
  • sqlite database (if using sqlite)
  • write to log, tmp directories

error: Could not send facts to Foreman: wrong Content-Length format (RuntimeError)

You might see this error if you run the HTTP push facts script or the sample external nodes script.
This is most likely due to older version of the mongrel gem. Please try and update your gems.

gem update mongrel

Is my Foreman instance running?

There is simple status service that returns JSON with "result" message "ok" when the instance is up and running. It also "pings" database and returns lag. Example:

$ curl -k -H "Accept: application/json" http://instance:3000/status

How do I enable debugging?

See Foreman debugging from the documentation.

Compute resource debugging

Do not run this command unless specifically told to do so.
To enable debugging of Compute Resources HTTP requests, you should pass some environment variables:

  su foreman -s /bin/bash
  cd ~foreman
  EXCON_DEBUG=true DEBUG=true ./script/rails s

How do I enable smart proxy debugging?

Edit /etc/foreman-proxy/settings.yml and change or uncomment the log_level parameter, setting it to DEBUG:

:log_level: DEBUG

Then restart the service:

service foreman-proxy restart

Logs will go to /var/log/foreman-proxy/proxy.log and access.log by default.

If the service is crashing immediately, failing to start up or continually returns 500 Internal Server error, start it up in foreground mode. To do this, disable the daemonize parameter in /etc/foreman-proxy/settings.yml:

:daemonize: false

And then run the smart proxy process:

sudo -u foreman-proxy /usr/share/foreman-proxy/bin/smart-proxy

Make the request from Foreman again, and logging should be shown on the console. Don't forget to change daemonize back to true before starting the service.

Unattended Provisioning Troubleshooting

Smart-proxy do not recognize my puppet environment

If I query smart-proxy and it return empty puppet environment :

curl -k -H "Content-Type:application/json" -H "Accept:application/json" http://puppet:8443/puppet/environments
=> []

You may have to add in your puppetmaster's puppet.conf environment definition like :

    manifest   = /etc/puppet/manifests/site.pp
    modulepath = /etc/puppet/modules
    fileserverconfig = /etc/puppet/fileserver.conf

    manifest   = /etc/puppet/preprod/manifests/site.pp
    modulepath = /etc/puppet/preprod/modules
    fileserverconfig = /etc/puppet/preprod/fileserver.conf

    manifest   = /etc/puppet/development/manifests/site.pp
    modulepath = /etc/puppet/development/modules
    fileserverconfig = /etc/puppet/development/fileserver.conf

    manifest   = /etc/puppet/test/manifests/site.pp
    modulepath = /etc/puppet/test/modules
    fileserverconfig = /etc/puppet/test/fileserver.conf

Unable to connect to Hypervisor?

Make sure the user that's actually running foreman can connect to your remote hypervisor (for instance by running sudo -u foreman virsh -c qemu+ssh://root@<host>/system list).

If you are running foreman through phusion passenger, ps auxwwf | grep "R[a]ils: /usr/share/foreman" | awk '{ print $1 }' will give you the user that's running foreman. If you find that it's not being run by foreman, do the following:
cd ~foreman
chown foreman config/environment.rb
touch ~foreman/tmp/restart.txt

Routing errors when running rake test?

Edit the config/settings.yaml and set the :login: setting to true

Causes of "Error 400 Bad Request", and other smart-proxy related errors in the Foreman interface:

  • Check the sudoers file on the proxy, if the user "foreman-proxy" can run "puppetca". The command puppetca is un-available in puppet 3.0. Workaround is to create a wrapper script.
  • From the cmd line, check if the user can run "puppetca" and "nsupdate" properly.
  • Check if Bind is listening on The proxy connects to localhost only, so this is required.
  • Check if the foreman-proxy user "foreman-proxy" can read the Bind rndc keys.
  • In Ubuntu, you will have to tell apparmor to allow Bind to write to zone files and journals. If your zone files are in /etc/bind/zones/, add "/etc/bind/zones/** rw," to /etc/apparmor.d/usr.sbin.named.
  • If using Ubuntu Libvirt, and the "Virtual Machine" tab is empty, then you most likely need to create a default storage pool:
    cat /etc/libvirt/storage/default.xml 
    <pool type='dir'>
  • On Ubuntu Libvirt, you may have to change /etc/libvirt/libvirtd.conf to listen on TCP.

Using Webrick you get : Error 400 on SERVER: Could not find node '<node fqdn>'; cannot compile

If you are using Foreman with webrick and you get an error downloading your catalog, maybe you encounter the issue noted in bug #1507
Edit the node.rb script and replace the following :

<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000*/*",<br />

by this :

<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000",<br />

DHCP Provisioning Fails with: "dhcpctl_connect: not found"

Depending on the version of ISC DHCP you are using a wrong key will return "dhcpctl_connect: not found" , this misleads to a connection problem.
To solve basically check if your KEY NAME matches your proxy settings.yml and also matches on your dhcpd.conf

Foreman is showing : Error message: Could not find json-1.5.5 in any of the sources

If you have a problem after doing a yum update ruby bundler needs to re-run.
This could happen after an update or change of ruby packages.

rm ~foreman/Gemfile.lock
cd ~foreman 
bundle install --local

VNC console in the Foreman UI refuses to connect to my VM: Server Disconnected

Check what version of python you have installed - noVNC requires python2. If you have multiple python versions installed, you can edit `extras/noVNC/` and change the shebang on line 1

How do I change the FQDN of the Foreman host?

This documentation is now part of the manual at

My deleted host keeps reappearing in the Hosts tab

After a period of time a deleted host reappears in the Host tab although the host no longer exists. A potential reason for this could be the --push-facts cron job is pushing the facts to the Foreman and the Foreman expects the host to check in but never does.

To fix:

rm /var/lib/puppet/yaml/facts/$hostname.yaml
rm /var/lib/puppet/yaml/node/$hostname.yaml

My node's environment is being reset to 'X' even though the puppet.conf on the host has environment = 'Y'. The Foreman settings have default_puppet_environment set as 'X' and enc_environment as False. I am using Puppet 3 on the node.

Foreman expects the node to specify it's own environment, if enc_environment is False. If no environment is returned by the node, it will override it with the value of default_puppet_environment. This was good till Puppet 3. But Puppet 3 no longer returns environment as a fact and hence Foreman assumes that the environment is not set.

No A and/or PTR records are created in DNS for new hosts

First ensure you have a smart proxy registered with the DNS feature, with a DNS server installed and configured for the appropriate forward and reverse DNS zones.

In Foreman, under More, Provisioning, Domains, edit your domain, change the DNS proxy setting to the appropriate proxy server. This will enable A records to be created for hosts built in that domain.

Next, under More, Provisioning, Subnets, edit your subnet, change the DNS proxy to the appropriate proxy for the reverse DNS zone. This will cause PTR records to be added for hosts with NICs in that subnet.

Both parts are needed for a complete DNS setup.

No TFTP menus or files are created for new hosts

This requires a proxy server registered with the TFTP feature, and a TFTP daemon running on it. Foreman will write directly to the TFTP root directory, as configured in the proxy settings. This should create a file per provisioning interface under /var/lib/tftpboot/pxelinux.cfg/ or similar.

Do ensure the "tftp_servername" setting in /etc/foreman-proxy/settings.d/tftp.yml is also correct, this is the IP address that gets passed to the DHCP server for the next-server option.

In Foreman, there are a number of conditions for this to happen. After fixing any of them, you either create a new host, or rebuild configs on an existing one from the host list or cycle the build mode (Cancel Build, then Build) to trigger an update.

Check the following

  1. Host must be managed, not discovered from Puppet etc. Edit the host and click the Manage host button.
  2. Host must have a provisioning network interface, check the icons on the left under the host's Interfaces tab.
  3. The provisioning interface must have a subnet set.
  4. The provisioning interface must be marked as Managed.
  5. The subnet must have a TFTP proxy set: under Infrastructure > Subnets, select the subnet, set the TFTP Proxy under the Proxies tab.
  6. The host must have an operating system set.
  7. The host must have network build selected as its provisioning method, under the OS tab. This can't be changed on an existing host.
  8. Unattended mode must be enabled in /etc/foreman/settings.yaml

No DHCP reservations are created for new hosts

This requires a proxy server registered with the DHCP feature, and a DHCP daemon running on it. Foreman will typically create DHCP reservations for every managed network interface on a host.

When using the ISC DHCP server (dhcpd), Foreman will use its OMAPI interface to request that the daemon writes a reservation, which it then writes to the leases file, e.g. /var/lib/dhcpd/dhcpd.leases or /var/lib/dhcp3/dhcpd.leases. Do note that the smart proxy never writes to config or leases files itself, this is all through the dhcpd - it only reads them. The leases file is maintained by dhcpd and regularly flushed and cleaned up, so it may not exactly reflect reality.

In Foreman, there are a number of conditions required for reservations to be created. After fixing any of them, you either create a new host, or rebuild configs on an existing one from the host list or cycle the build mode (Cancel Build, then Build) to trigger an update.

Check the following

  1. Host must be managed, not discovered from Puppet etc. Edit the host and click the Manage host button.
  2. The interface must have a subnet set.
  3. The interface must be marked as Managed.
  4. The interface must have an IP address set, or a compute resource that allocates IPs (e.g. EC2) must be used.
  5. The interface must have a MAC address address set, or a compute resource that allocates MACs (e.g. Libvirt, VMware) must be used.
  6. The subnet must have a DHCP proxy set: under Infrastructure > Subnets, select the subnet, set the DHCP Proxy under the Proxies tab.
  7. The subnet must have DHCP set as the boot mode: under Infrastructure > Subnets, select the subnet, set Boot Mode to DHCP (see also #14905)
  8. The host must have an operating set if it has a provisioning network interface
  9. Unattended mode must be enabled in /etc/foreman/settings.yaml

In the release candidates, the error was:

ActionController::RoutingError (No route matches [POST] "/fact_values/create")
ActionController::RoutingError (No route matches [POST] "/reports/create")

Unprocessable Entity error during installation

Foreman installer registers proxy which is being deployed automatically. If it is, for any reason, already present but under different name, Foreman reject to register the proxy with HTTP 422 error. Proxy registration can be either turned off, or name can be changed using --foreman-proxy-register-in-foreman or --foreman-proxy-registered-name installer options.

Foreman proxy fails to start with Are the values correct in settings.yml and do permissions allow reading?: Permission denied

If you are using puppet CA see here:

FreeIPA realm proxy fails with SSLv3 read server certificate B: certificate verify failed

If you see error messages like this in proxy.log:

E, [2014-11-21T06:09:59.630126 #5631] ERROR -- : SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

The certificate authority (CA) that signed your IPA CA cert is not trusted by the proxy host. This is fairly common if the "foreman proxy" host is not an IPA server. You can trust the CA certificate of the IPA server...

cp /etc/ipa/ca.crt /etc/pki/ca-trust/source/source

FreeIPA realm proxy fails with "generic preauthentication failure"

If you see error messages like this in your proxy logs:

E, [2014-05-15T19:28:08.211121 #3595] ERROR -- : Failed to initialise credential cache from keytab: krb5_get_init_creds_keytab:  Generic preauthentication failure
E, [2014-05-15T19:28:08.211515 #3595] ERROR -- : Failed to initailize credentials cache from keytab: krb5_get_init_creds_keytab: Generic preauthentication failure
D, [2014-05-15T19:28:08.211614 #3595] DEBUG -- : /usr/share/foreman-proxy/bin/../lib/proxy/kerberos.rb:13:in `init_krb5_ccache'

First, check permissions, the /etc/foreman-proxy/freeipa.keytab file should be owned by foreman-proxy and mode 600.

You may be running a much newer version of FreeIPA than the client which provides some unknown encryption types.

Run `klist -etk /etc/foreman-proxy/freeipa.keytab` and you'll see some unnamed enryption types like this:

1 05/14/14 21:14:17  (etype 25)
1 05/14/14 21:14:17 (etype 26)

To fix it, delete /etc/foreman-poxy/freeipa.keytab and refetch it, specifying only the enctypes your system knows about:

ipa-getkeytab -s -p  -k /etc/foreman-proxy/freeipa.keytab --enctypes=aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac

How to log REST client calls

Sometimes it is useful to see content of calls that are being made by Foreman using rest-client library (compute resources, proxy communication etc). This is as easy as dropping the following file and restarting Foreman:

$ cat >/usr/share/foreman/config/initializers/00_rest_client.rb <<'EOT'
require 'rest_client'
RestClient.log = do |proxy|
    def proxy.<<(message) message

I'm getting connection timeouts when Foreman tries to connect to ec2 when running behind a http proxy

The error can look like this.

<div id="backtrace" class="alert alert-block alert-danger base in fade hide">
  <strong>connect timeout reached</strong><br>
  app/models/compute_resources/foreman/model/ec2.rb:59:in `regions'
<br>app/models/compute_resources/foreman/model/ec2.rb:72:in `test_connection'
<br>app/models/compute_resource.rb:120:in `new_vm'
<br>app/views/hosts/_compute.html.erb:1:in `_app_views_hosts__compute_html_erb__2073528223_70084372690700'
<br>app/controllers/hosts_controller.rb:139:in `compute_resource_selected'
<br>app/models/taxonomy.rb:48:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:143:in `as_location'
<br>app/models/taxonomy.rb:47:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:108:in `as_org'<br>app/models/taxonomy.rb:46:in `as_taxonomy'
<br>app/controllers/hosts_controller.rb:135:in `compute_resource_selected'<br>app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
<br>lib/middleware/catch_json_parse_errors.rb:9:in `call'
<p><a href="/" data-id="aid_not_defined">Back</a></p>

The problem is that fog isn't picking up your environment's proxy settings, so the solution is for apache to pass them to it.

On Debian/Ubuntu, edit /etc/apache2/envvars to include the following

export http_proxy=http://<host>:<port>
export https_proxy=https://<host>:<port>

On Red Hat OSes, edit /etc/sysconfig/httpd instead.

Restart apache.

More information at

How to take a thread dump of running Foreman processes

It's possible to trigger running Foreman applications to log thread dumps - if they're not responding, this can help determine which piece of code they're running.

If you're using Passenger (default setup with foreman-installer), run pkill -ABRT -f RackApp to send an abort signal to Passenger. This will be written to /var/log/httpd/error_log (EL/Fedora) or /var/log/apache2/error.log (Debian/Ubuntu).

If you're using Foreman standalone (WEBrick or other), pkill -TTIN -f rails (or TTIN signal to whatever container you're using) will print the thread dump to stdout - you may have to run Foreman in the foreground to see this.

"ERROR -- : Wrong size. Was 315, should be 196" when using Realm Proxy on EL7

A bug in Ruby 2.0 prevents the Realm proxy from working correctly:

Commenting these out on lines 505-506 in /usr/share/ruby/xmlrpc/client.rb is a temporary workaround:

      #elsif expected != "<unknown>" and expected.to_i != data.bytesize and resp["Transfer-Encoding"].nil?
      #  raise "Wrong size. Was #{data.bytesize}, should be #{expected}" 

This is expected to be fixed in one of the next releases of EL7.

What are the default credentials?

When using foreman-installer, the first admin account will be set up and credentials are printed at the end, e.g.

  * Foreman is running at
      Initial credentials are admin / 3ekw5xtyXCoXxS29

Visit the web UI and log in with the username "admin" and password "3ekw5xtyXCoXxS29" (in this example).

You can customise these when doing the initial installation by using foreman-installer --foreman-admin-password=example. Note that this only works during the first installation - it won't change the password if run again.

If you are installing from source, the random password will be printed when running rake db:seed.

How do I reset the admin password?

Run foreman-rake permissions:reset which will create a new random password for the admin account.

You can specify a particular password by adding password=example to the end of the command, or change a different user with username=exuser. Note that this command is only designed to reset the admin account, so will give the specified user admin privileges and will create the account if it's missing.

SELinux denials

When encountering SELinux denials, it is recommended to re-run foreman-selinux-enable followed by foreman-selinux-relabel and then restart all services (httpd, foreman-proxy, puppet, ISC services). If it does not help, force full filesystem relabel with touch /.autorelabel and rebooting the server.

Only after that, use foreman-debug -u to upload log files and SELinux configuration and contact developers providing the download link (it's readable only to Foreman Core developers). If you prefer pastebin into the list, provide output of the following commands:

  • rpm -q foreman foreman-selinux foreman-proxy-selinux
  • ps auxZ
  • semodule -l
  • semanage boolean -l
  • ausearch -m AVC -m USER_AVC -m SELINUX_ERR | head -n 100

Template or "built" request returns Net::HTTPMethodNotAllowed

When host associated with the token or IP address is not in build mode, Foreman returns: Failed to retrieve built template for hostname - Net::HTTPMethodNotAllowed. This can happen when rendering a template, or at the end of the provisioning script when "built" call is sent via curl/wget.

No PXELinux/PXEGrub/PXEGrub2 templates were found for this host

If you encounter "No PXELinux/PXEGrub/PXEGrub2 templates were found for this host, make sure you define at least one in your OS settings or change PXE loader" error make sure all associated templates do belong to the Organization and Location of the host that is being created or edited.

Enable detailed SQL logger for orchestration messages

In /etc/foreman/settings.yaml set this:

  :level: debug

    :enabled: true

This will ensure the SQL logger is enabled (it is turned off on production environmenty by default).

Troubleshooting multi CA environment

Note: If you're not running CentOS, adapt the following to your environment.

When installing the Foreman in an multi CA environment it's usually better to store the CA certs within the host's CA trust. On CentOS when the command update-ca-trust is run, the file /etc/pki/ca-trust/extracted/openssl/ is created. When installing the Foreman, point --foreman-proxy-foreman-ssl-ca --foreman-proxy-ssl-ca and optionally (if you're not using the Puppet CA) --foreman-proxy-puppet-ssl to the mentioned file. This will make the Foreman trust the CA certs you add in /etc/pki/ca-trust/source/anchors.

If your OS does not build the bundle or you want to point your Foreman/ForemanProxy directly to a file, create the chain file (intermediate cert + root cert) and point Foreman/ForemanProxy to it directly. Note that intermediate cert only will probably not work, since if you specify a file, root certificates in system store are ignored.