Troubleshooting » History » Version 67

Version 66 (Fergus Nelson, 07/24/2014 06:57 AM) → Version 67/100 (Dominic Cleal, 07/31/2014 03:24 AM)


h1. How to provide enough info to get help?

Foreman is pretty complex software project with many components and extensive configuration. Always try to describe best your specific case (OS, version, ruby and foreman version, database, specific configuration). From Foreman version 1.2 you can use _foreman-debug_ script which collects all these information and also filters out things like passwords or tokens. For earlier Foreman releases you can use this bash script directly from git:

h1. ERF error code

If you see an ERF error code, such as ERF12-7740, look it up on [[ErrorCodes]].

rake aborted!

I get the following error while running rake db:migrate
rake aborted!
undefined method `reenable' for <Rake::Task db:schema:dump => [environment]>:Rake::Task

this means you have an old version of rake (older than 0.87), you can verify it by typing:
rake --version

Usually this error is "safe" to ignore, however, it is recommended to upgrade to rake >= 0.87

h1. Unable to save: failed to detect boot server: getaddrinfo: Name or service not known

if you have registred your smart-proxys via name, and these name are not resolvable, you get this error, you can add the name to your dns server or set
:tftp_servername: in smart-proxy settings to the ip of the smart-proxy.

h1. I use puppet enterprise, what now?

As of PE version 1.0, foreman won't install easily without a few modifications. Since PE ships with its own version of Ruby you will need to install a few gems.
Basically you need to tell foreman to use the PE provided ruby library instead. Your not required to run the puppet version of ruby either as many distros now ship with ruby 1.8.7.

# You first need to install the pe-ruby-devel package that ships with PE. You can find this in your packages folder wherever you extracted PE.
a. rpm -ivh pe-ruby-devel-
# Update your path variable to use the ruby version PE ships with (export PATH=/opt/puppet/bin:$PATH)
# Check to make sure your PATH was updated ("which gem" should return /opt/puppet/bin/gem)
# gem install json
# gem install rest-client
(install the gem database driver you use)
# gem install sqlite3 or gem install mysql ( you may have additional dependencies that would need to be resolved first)

Some of the gem dependencies can be resolved by installing the development library for that package (example: mysql-devel)

I found it difficult to install the sqlite3 gem because of dependency hell so I just setup mysql instead which was far easier to setup. Additionally mysql is much more scalable than sqlite.

You can get a list of the currently installed gems by using: gem list.

If your running the foreman init script or /etc/sysconfig/foreman configuration file. You may need to update apache/passenger configurations that point to the PE ruby version.

h1. error: Entry 'some file' not uptodate. Cannot merge.

If you downloaded Foreman from source (git), it could be that some of the files you have modified (or were modified automatically) conflicts with another file Foreman wants to upgrade.

If you don't think that you edited this file (e.g. db/schema.rb) it is usually safe to do:
git checkout 'some file'

This will revert the file to its previous condition (as in the repo at the time of checkout) and now you should be able to get the latest version by:
git pull

h1. Strange errors with passenger

Passenger executes foreman, based on the owner of the config/environmnets.rb file, make sure that this use can actually access:
* puppet configuration files
* sqlite database (if using sqlite)
* write to log, tmp directories

see source:extras/puppet/foreman for complete puppet based setup for Foreman.

h1. error: Could not send facts to Foreman: wrong Content-Length format (RuntimeError)

You might see this error if you run the HTTP push facts script or the sample external nodes script.
This is most likely due to older version of the mongrel gem. Please try and update your gems.

gem update mongrel

h1. Is my Foreman instance running?

There is simple status service that returns JSON with "result" message "ok" when the instance is up and running. It also "pings" database and returns lag. Example:

$ curl -k -H "Accept: application/json" http://instance:3000/status

h1. How do I enable debugging?

Edit config/environments/production.rb and uncomment "config.log_level = :debug"

To enable debugging of Compute Resources HTTP requests, you should pass some environment variables:
cd ~foreman
EXCON_DEBUG=true DEBUG=true ./script/rails s

h1. [[Unattended Provisioning Troubleshooting]]

h1. Puppet and Storeconfigs: err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep

Your activerecord Version is to new, you need activrecord 3.0.10 see

gem uninstall activerecord
gem install activerecord -v 3.0.10

h1. How do I fix this error: "Unable to find directory $confdir/environments/$environment/modules in environment master" ?

If you've followed to
create a multi-environment puppet configuration, you're initial rake may look something like this:
rake puppet:import:puppet_classes RAILS_ENV=production
(in /usr/share/foreman)
Evaluating possible changes to your installation
Problems were detected during the evaluation phase

Unable to find directory $confdir/environments/$environment/modules in environment master

Please fix these issues and try again
This is because Foreman does not currently evaluate $confdir. There is a feature request open on this issue. In the
meantime, hard code your $confdir in your puppet.conf.

h1. Smart-proxy do not recognize my puppet environment

If I query smart-proxy and it return empty puppet environment :
curl -k -H "Content-Type:application/json" -H "Accept:application/json" http://puppet:8443/puppet/environments
=> []

You may have to add in your puppetmaster's puppet.conf environment definition like :

manifest = /etc/puppet/manifests/site.pp
modulepath = /etc/puppet/modules
fileserverconfig = /etc/puppet/fileserver.conf

manifest = /etc/puppet/preprod/manifests/site.pp
modulepath = /etc/puppet/preprod/modules
fileserverconfig = /etc/puppet/preprod/fileserver.conf

manifest = /etc/puppet/development/manifests/site.pp
modulepath = /etc/puppet/development/modules
fileserverconfig = /etc/puppet/development/fileserver.conf

manifest = /etc/puppet/test/manifests/site.pp
modulepath = /etc/puppet/test/modules
fileserverconfig = /etc/puppet/test/fileserver.conf

h1. How do I fix the puppet class import problem with multiple puppet environment ?

If Foreman don't seems to be aware of all your environment and is able to import only your "production" environment (see,
try to add ":puppet_conf: /etc/puppet/puppet.conf" in your settings.yaml.

h1. Unable to connect to Hypervisor?

Make sure the user that's actually running foreman can connect to your remote hypervisor (for instance by running sudo -u foreman virsh -c qemu+ssh://root@<host>/session list).

If you are running foreman through phusion passenger, ps auxwwf | grep "R[a]ils: /usr/share/foreman" | awk '{ print $1 }' will give you the user that's running foreman. If you find that it's not being run by foreman, do the following:
cd ~foreman
chown foreman config/environment.rb
touch ~foreman/tmp/restart.txt

h1. Routing errors when running rake test?

Edit the config/settings.yaml and set the :login: setting to true

h1. cannot load such file -- facter / puppet (LoadError)

If you running foreman under RVM or custom ruby installation, you would need to get facter installed as a gem (as facter packages are not in your ruby search path).

simply add another file to FOREMAN_DIR/bundler.d
echo "gem 'facter'" > bundler.d/Gemfile.local.rb

h2. Pre-Foreman 1.3

Prior to 1.3, puppet was also a dependency, so instead use:

echo "gem 'puppet'" > bundler.d/Gemfile.local.rb

Edit the config/settings.yaml and add: @:puppetgem: true@

h1. Causes of "Error 400 Bad Request", and other smart-proxy related errors in the Foreman interface:

* Check the sudoers file on the proxy, if the user "foreman-proxy" can run "puppetca". The command puppetca is un-available in puppet 3.0. Workaround is to create a wrapper script.
* From the cmd line, check if the user can run "puppetca" and "nsupdate" properly.
* Check if Bind is listening on The proxy connects to localhost only, so this is required.
* Check if the foreman-proxy user "foreman-proxy" can read the Bind rndc keys.
* In Ubuntu, you will have to tell apparmor to allow Bind to write to zone files and journals. If your zone files are in /etc/bind/zones/, add "/etc/bind/zones/** rw," to /etc/apparmor.d/usr.sbin.named.
* If using Ubuntu Libvirt, and the "Virtual Machine" tab is empty, then you most likely need to create a default storage pool:
cat /etc/libvirt/storage/default.xml
<pool type='dir'>
* On Ubuntu Libvirt, you may have to change /etc/libvirt/libvirtd.conf to listen on TCP.

h1. Using Webrick you get : Error 400 on SERVER: Could not find node '<node fqdn>'; cannot compile

If you are using Foreman with webrick and you get an error downloading your catalog, maybe you encouter the issue noted in bug #1507
Edit the node.rb script and replace the following :

<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000*/*",<br />

by this :

<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000",<br />

h1. DHCP Provisioning Fails with: "dhcpctl_connect: not found"

Depending on the version of ISC DHCP you are using a wrong key will return "dhcpctl_connect: not found" , this misleads to a connection problem.
To solve basically check if your KEY NAME matches your proxy settings.yml and also matches on your dhcpd.conf

h1. Foreman is showing : Error message: Could not find json-1.5.5 in any of the sources

If you have a problem after doing a yum update ruby bundler needs to re-run.
This could happen after an update or chane of ruby packages.

rm ~foreman/Gemfile.lock
cd ~foreman
bundle install --local

h1. VNC console in the Foreman UI refuses to connect to my VM: Server Disconnected

Check what version of python you have installed - noVNC requires python2. If you have multiple python versions installed, you can edit `extras/noVNC/` and change the shebang on line 1

h1. Unable to upgrade to Puppet 3.2 due to rubygem-net-ldap dependency issues

There are conflicting version requirements with rubygem-net-ldap between the Foreman and Puppet Labs dependency repos since Puppet 3.2. Since Foreman 1.1, the rubygem-net-ping package is no longer required for the proxy, so first remove this and you should either be able to upgrade or remove rubygem-net-ldap altogether if there's no dependency any more.

h1. How do I change the FQDN of the Foreman host?

# Some settings in the Foreman UI use the hostname, update under More, Settings, e.g.
#* General, foreman_url
#* Provisioning, ssl_certificate and ssl_priv_key
# The registered smart proxy URL if you've installed it, this can be edited via More>Configuration>Smart Proxies
# Puppet SSL certs: generate new ones with @puppet cert generate FQDN@
# Apache configs: update conf.d/{foreman,puppet}.conf with new SSL cert filenames, ServerName and VirtualHost IP addresses if applicable
# Smart proxy configuration: update SSL cert filenames
# URLs in the puppetmaster's ENC script (/etc/puppet/node.rb) and report script (foreman.rb)

h1. My deleted host keeps reappearing in the Hosts tab

After a period of time a deleted host reappears in the Host tab although the host no longer exists. A potential reason for this could be the --push-facts cron job is pushing the facts to the Foreman and the Foreman expects the host to check in but never does.

To fix:
rm /var/lib/puppet/yaml/facts/$hostname.yaml
rm /var/lib/puppet/yaml/node/$hostname.yaml

h1. My node's environment is being reset to 'X' even though the puppet.conf on the host has environment = 'Y'. The Foreman settings have default_puppet_environment set as 'X' and enc_environment as False. I am using Puppet 3 on the node.

Foreman expects the node to specify it's own environment, if enc_environment is False. If no environment is returned by the node, it will override it with the value of default_puppet_environment. This was good till Puppet 3. But Puppet 3 no longer returns environment as a fact and hence Foreman assumes that the environment is not set.

h1. No A and/or PTR records are created in DNS for new hosts

First ensure you have a smart proxy registered with the DNS feature, with a DNS server installed and configured for the appropriate forward and reverse DNS zones.

In Foreman, under More, Provisioning, Domains, edit your domain, change the DNS proxy setting to the appropriate proxy server. This will enable A records to be created for hosts built in that domain.

Next, under More, Provisioning, Subnets, edit your subnet, change the DNS proxy to the appropriate proxy for the reverse DNS zone. This will cause PTR records to be added for hosts with NICs in that subnet.

Both parts are needed for a complete DNS setup.

h1. No TFTP menus or files are created for new hosts

This requires a proxy server registered with the TFTP feature, and a TFTP daemon running on it. Foreman will write directly to the TFTP root directory, as configured in the proxy settings. Do ensure the "tftp_servername" setting is also correct, this is the IP address that gets passed to the DHCP server for the @next-server@ option.

In Foreman, under More, Provisioning, Subnets, edit your subnet, change the TFTP proxy to the appropriate proxy server for the subnet. This will cause hosts built in this subnet to receive a PXELinux menu for their primary MAC address.

h1. After Foreman 1.3 upgrade, reports and ENC fact uploads no longer work

We changed how the fact and report uploads work in the 1.3 release. During 1.3 upgrade, read "release notes": and update both the report processor (foreman.rb) and ENC script (node.rb), otherwise errors will be logged in the production.log (return code 400):

DEPRECATION: /reports/create is deprecated, update your report processor to POST to /api/reports
See the Foreman 1.3 release notes for a new example report processor

In the release candidates, the error was:

ActionController::RoutingError (No route matches [POST] "/fact_values/create")
ActionController::RoutingError (No route matches [POST] "/reports/create")

h1. Unprocessable Entity error during installation

Foreman installer registers proxy which is being deployed automatically. If it is, for any reason, already present but under different name, Foreman reject to register the proxy with HTTP 422 error. Proxy registration can be either turned off, or name can be changed using @--foreman-proxy-register-in-foreman@ or @--foreman-proxy-registered-name@ installer options.

h1. Installation errors on Puppet 3.4.x or 2.7.24

When using Puppet 3.4.0 or 3.4.1, installation errors will be seen and running "puppet --version" after install will show the following:

Could not intialize global default settings: undefined method `mode=' for #<Puppet::Settings::AutosignSetting:0x7fa026ad44e0>

This is caused by Puppet bug "PUP-1015": On 3.4.1 and 2.7.24, additionally the /etc/puppet/manifests/site.pp file will be set to mode 0600 rather than the default 0644. This is "PUP-1255":

Puppet 3.4.2 and 2.7.25 fix both of these issues, please ensure you use these versions instead of 3.4.0/1 or 2.7.24.

h1. Yum fails to install foreman-installer, Requires: rubygem(rest-client)

If you get this error, it indicates the EPEL repository is missing:

Error: Package: rubygem-foreman_api-0.1.9-1.el6.noarch (foreman)
Requires: rubygem(rest-client) >= 1.6.1

This is a requirement for the installer and Foreman. Download and install the epel-release RPM from here:

h1. Foreman proxy fails to start with Are the values correct in settings.yml and do permissions allow reading?: Permission denied

If you are using puppet CA see here:

h1. FreeIPA realm proxy fails with "generic preauthentication failure"

If you see error messages like this in your proxy logs:

E, [2014-05-15T19:28:08.211121 #3595] ERROR -- : Failed to initialise credential cache from keytab: krb5_get_init_creds_keytab: Generic preauthentication failure
E, [2014-05-15T19:28:08.211515 #3595] ERROR -- : Failed to initailize credentials cache from keytab: krb5_get_init_creds_keytab: Generic preauthentication failure
D, [2014-05-15T19:28:08.211614 #3595] DEBUG -- : /usr/share/foreman-proxy/bin/../lib/proxy/kerberos.rb:13:in `init_krb5_ccache'

You may be running a much newer version of FreeIPA than the client which provides some unknown encryption types.

Run `klist -etk /etc/foreman-proxy/freeipa.keytab` and you'll see some unnamed enryption types like this:

1 05/14/14 21:14:17 realm-proxy@EXAMPLE.COM (etype 25)
1 05/14/14 21:14:17 realm-proxy@EXAMPLE.COM (etype 26)

To fix it, delete /etc/foreman-poxy/freeipa.keytab and refetch it, specifying only the enctypes your system knows about:

ipa-getkeytab -s -p realm-proxy@EXAMPLE.COM -k /etc/foreman-proxy/freeipa.keytab --enctypes=aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac

h1. FreeIPA realm proxy fails with "LoadError: no such file to load -- ffi_c"

This is due to a broken rubygem-ffi that ships from Puppetlabs:

We need the EPEL one (although with the exact same version).

Ensure you have the latest (rubygem-ffi-1.0.9-11.el6.x86_64):

yum upgrade rubygem-ffi

Now downgrade, but disable the puppetlabs-dependecies repo so we grab the package from EPEL:

yum downgrade --disablerepo=puppetlabs-dependencies rubygem-ffi-1.0.9-9.el6

Restart the proxy.

h1. How to log REST client calls

Sometimes it is useful to see content of calls that are being made by Foreman using rest-client library (compute resources, proxy communication etc). This is as easy as dropping the following file and restarting Foreman:

$ cat >/usr/share/foreman/config/initializers/00_rest_client.rb <<'EOT'
require 'rest_client'
RestClient.log = do |proxy|
def proxy.<<(message) message

h1. I'm getting connection timeouts when Foreman tries to connect to ec2 when running behind a http proxy

The error can look like this.
<div id="backtrace" class="alert alert-block alert-danger base in fade hide">
<strong>connect timeout reached</strong><br>
app/models/compute_resources/foreman/model/ec2.rb:59:in `regions'
<br>app/models/compute_resources/foreman/model/ec2.rb:72:in `test_connection'
<br>app/models/compute_resource.rb:120:in `new_vm'
<br>app/views/hosts/_compute.html.erb:1:in `_app_views_hosts__compute_html_erb__2073528223_70084372690700'
<br>app/controllers/hosts_controller.rb:139:in `compute_resource_selected'
<br>app/models/taxonomy.rb:48:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:143:in `as_location'
<br>app/models/taxonomy.rb:47:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:108:in `as_org'<br>app/models/taxonomy.rb:46:in `as_taxonomy'
<br>app/controllers/hosts_controller.rb:135:in `compute_resource_selected'<br>app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
<br>lib/middleware/catch_json_parse_errors.rb:9:in `call'
<p><a href="/" data-id="aid_not_defined">Back</a></p>

The problem is that fog isn't picking up your environment's proxy settings, so the solution is for apache to pass them to it. Edit /etc/apache2/envvars to include the following

export http_proxy=http://<host>:<port>
export https_proxy=https://<host>:<port>

Restart apache.