Project

General

Profile

Troubleshooting » History » Version 51

Lukas Zapletal, 10/07/2013 12:46 PM

1 3 Ohad Levy
{{toc}}
2 3 Ohad Levy
3 43 Lukas Zapletal
h1. How to provide enough info to get help?
4 43 Lukas Zapletal
5 43 Lukas Zapletal
Foreman is pretty complex software project with many components and extensive configuration. Always try to describe best your specific case (OS, version, ruby and foreman version, database, specific configuration). From Foreman version 1.2 you can use _foreman-debug_ script which collects all these information and also filters out things like passwords or tokens. For earlier Foreman releases you can use this bash script directly from git: https://github.com/theforeman/foreman/blob/develop/script/foreman-debug
6 43 Lukas Zapletal
7 1 Ohad Levy
h1. rake aborted!
8 1 Ohad Levy
9 1 Ohad Levy
I get the following error while running rake db:migrate
10 1 Ohad Levy
<pre>
11 1 Ohad Levy
rake aborted!
12 1 Ohad Levy
undefined method `reenable' for <Rake::Task db:schema:dump => [environment]>:Rake::Task
13 1 Ohad Levy
</pre>
14 1 Ohad Levy
15 1 Ohad Levy
this means you have an old version of rake (older than 0.87), you can verify it by typing:
16 1 Ohad Levy
<pre>
17 1 Ohad Levy
rake --version
18 1 Ohad Levy
</pre>
19 1 Ohad Levy
20 1 Ohad Levy
Usually this error is "safe" to ignore, however, it is recommended to upgrade to rake >= 0.87
21 2 Ohad Levy
22 10 Corey Osman
23 20 Florian Koch
h1. Unable to save: failed to detect boot server: getaddrinfo: Name or service not known
24 1 Ohad Levy
25 20 Florian Koch
if you have registred your smart-proxys via name, and these name are not resolvable, you get this error, you can add the name to your dns server or set 
26 20 Florian Koch
:tftp_servername: in smart-proxy settings to the ip of the smart-proxy.
27 10 Corey Osman
28 7 Corey Osman
h1. I use puppet enterprise, what now?
29 7 Corey Osman
30 7 Corey Osman
As of PE version 1.0, foreman won't install easily without a few modifications.  Since PE ships with its own version of Ruby you will need to install a few gems.
31 12 Corey Osman
Basically you need to tell foreman to use the PE provided ruby library instead.  Your not required to run the puppet version of ruby either as many distros now ship with ruby 1.8.7.
32 7 Corey Osman
33 8 Ohad Levy
# You first need to install the pe-ruby-devel package that ships with PE.  You can find this in your packages folder wherever you extracted PE.
34 1 Ohad Levy
      a. rpm -ivh pe-ruby-devel-1.8.7.302-1.el5.i386.rpm 
35 8 Ohad Levy
# Update your path variable to use the ruby version PE ships with (export PATH=/opt/puppet/bin:$PATH)
36 8 Ohad Levy
# Check to make sure your PATH was updated ("which gem" should return /opt/puppet/bin/gem)
37 8 Ohad Levy
# gem install json
38 8 Ohad Levy
# gem install rest-client
39 7 Corey Osman
(install the gem database driver you use)
40 8 Ohad Levy
# gem install sqlite3 or gem install mysql  ( you may have additional dependencies that would need to be resolved first)
41 7 Corey Osman
42 7 Corey Osman
Some of the gem dependencies can be resolved by installing the development library for that package (example: mysql-devel)
43 7 Corey Osman
44 7 Corey Osman
I found it difficult to install the sqlite3 gem because of dependency hell so I just setup mysql instead which was far easier to setup.  Additionally mysql is much more scalable than sqlite.
45 7 Corey Osman
46 7 Corey Osman
You can get a list of the currently installed gems by using: gem list.
47 1 Ohad Levy
48 10 Corey Osman
If your running the foreman init script or /etc/sysconfig/foreman configuration file.  You may need to update apache/passenger configurations that point to the PE ruby version.
49 6 Corey Osman
50 6 Corey Osman
51 2 Ohad Levy
h1. error: Entry 'some file' not uptodate. Cannot merge.
52 2 Ohad Levy
53 2 Ohad Levy
If you downloaded Foreman from source (git), it could be that some of the files you have modified (or were modified automatically) conflicts with another file Foreman wants to upgrade.
54 2 Ohad Levy
55 2 Ohad Levy
If you don't think that you edited this file (e.g. db/schema.rb) it is usually safe to do:
56 2 Ohad Levy
<pre>
57 2 Ohad Levy
git checkout 'some file'
58 2 Ohad Levy
</pre> 
59 2 Ohad Levy
60 2 Ohad Levy
This will revert the file to its previous condition (as in the repo at the time of checkout) and now you should be able to get the latest version by:
61 2 Ohad Levy
<pre>
62 2 Ohad Levy
git pull
63 2 Ohad Levy
</pre>
64 3 Ohad Levy
65 3 Ohad Levy
h1. Strange errors with passenger
66 3 Ohad Levy
67 3 Ohad Levy
Passenger executes foreman, based on the owner of the config/environmnets.rb file, make sure that this use can actually access:
68 3 Ohad Levy
* puppet configuration files
69 3 Ohad Levy
* sqlite database (if using sqlite)
70 3 Ohad Levy
* write to log, tmp directories
71 3 Ohad Levy
72 3 Ohad Levy
see source:extras/puppet/foreman for complete puppet based setup for Foreman.
73 4 Bash Shell
74 4 Bash Shell
75 4 Bash Shell
76 4 Bash Shell
h1. error: Could not send facts to Foreman: wrong Content-Length format (RuntimeError)
77 4 Bash Shell
78 4 Bash Shell
You might see this error if you run the HTTP push facts script or the sample external nodes script.
79 4 Bash Shell
This is most likely due to older version of the mongrel gem. Please try and update your gems.
80 4 Bash Shell
81 4 Bash Shell
<pre>
82 4 Bash Shell
gem update mongrel
83 4 Bash Shell
</pre>
84 5 Lukas Zapletal
85 5 Lukas Zapletal
h1. Is my Foreman instance running?
86 5 Lukas Zapletal
87 5 Lukas Zapletal
There is simple status service that returns JSON with "result" message "ok" when the instance is up and running. It also "pings" database and returns lag. Example:
88 5 Lukas Zapletal
89 5 Lukas Zapletal
<pre>
90 5 Lukas Zapletal
$ curl -k -H "Accept: application/json" http://instance:3000/status
91 5 Lukas Zapletal
{"status":200,"result":"ok","db_duration_ms":"5"}
92 5 Lukas Zapletal
</pre>
93 9 Dis Connect
94 9 Dis Connect
95 9 Dis Connect
h1. How do I enable debugging?
96 9 Dis Connect
97 9 Dis Connect
Edit config/environments/production.rb and uncomment "config.log_level = :debug"
98 13 Corey Osman
99 13 Corey Osman
h1. [[Unattended Provisioning Troubleshooting]]
100 14 Florian Koch
101 14 Florian Koch
h1. Puppet and Storeconfigs: err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep
102 14 Florian Koch
103 14 Florian Koch
Your activerecord Version is to new, you need activrecord 3.0.10 see http://projects.puppetlabs.com/issues/9290
104 14 Florian Koch
105 14 Florian Koch
<pre>
106 14 Florian Koch
gem uninstall activerecord
107 14 Florian Koch
gem install activerecord -v 3.0.10
108 14 Florian Koch
</pre>
109 15 Romain Vrignaud
110 15 Romain Vrignaud
111 15 Romain Vrignaud
112 15 Romain Vrignaud
h1. How do I fix this error: "Unable to find directory $confdir/environments/$environment/modules in environment master" ?
113 15 Romain Vrignaud
114 15 Romain Vrignaud
If you've followed http://docs.puppetlabs.com/guides/environment.html#configuring-environments-on-the-puppet-master to 
115 15 Romain Vrignaud
create a multi-environment puppet configuration, you're initial rake may look something like this:
116 15 Romain Vrignaud
<pre>
117 15 Romain Vrignaud
rake puppet:import:puppet_classes RAILS_ENV=production
118 15 Romain Vrignaud
(in /usr/share/foreman)
119 15 Romain Vrignaud
Evaluating possible changes to your installation
120 15 Romain Vrignaud
Problems were detected during the evaluation phase
121 15 Romain Vrignaud
122 15 Romain Vrignaud
Unable to find directory $confdir/environments/$environment/modules in environment master
123 15 Romain Vrignaud
124 15 Romain Vrignaud
Please fix these issues and try again
125 15 Romain Vrignaud
</pre>
126 15 Romain Vrignaud
This is because Foreman does not currently evaluate $confdir.  There is a feature request open on this issue.  In the 
127 15 Romain Vrignaud
meantime, hard code your $confdir in your puppet.conf.
128 15 Romain Vrignaud
129 21 Romain Vrignaud
h1. Smart-proxy do not recognize my puppet environment
130 23 Romain Vrignaud
131 21 Romain Vrignaud
If I query smart-proxy and it return empty puppet environment :
132 22 Romain Vrignaud
<pre>
133 21 Romain Vrignaud
curl -k -H "Content-Type:application/json" -H "Accept:application/json" http://puppet:8443/puppet/environments
134 21 Romain Vrignaud
=> []
135 22 Romain Vrignaud
</pre>
136 21 Romain Vrignaud
137 21 Romain Vrignaud
You may have to add in your puppetmaster's puppet.conf environment definition like :
138 21 Romain Vrignaud
139 22 Romain Vrignaud
<pre>
140 21 Romain Vrignaud
[production]
141 21 Romain Vrignaud
    manifest   = /etc/puppet/manifests/site.pp
142 21 Romain Vrignaud
    modulepath = /etc/puppet/modules
143 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/fileserver.conf
144 21 Romain Vrignaud
    templatedir=/etc/puppet/templates/
145 21 Romain Vrignaud
146 21 Romain Vrignaud
[preprod]
147 21 Romain Vrignaud
    manifest   = /etc/puppet/preprod/manifests/site.pp
148 21 Romain Vrignaud
    modulepath = /etc/puppet/preprod/modules
149 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/preprod/fileserver.conf
150 21 Romain Vrignaud
    templatedir=/etc/puppet/preprod/templates
151 21 Romain Vrignaud
152 21 Romain Vrignaud
[development]
153 21 Romain Vrignaud
    manifest   = /etc/puppet/development/manifests/site.pp
154 21 Romain Vrignaud
    modulepath = /etc/puppet/development/modules
155 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/development/fileserver.conf
156 21 Romain Vrignaud
    templatedir=/etc/puppet/development/templates/
157 21 Romain Vrignaud
158 21 Romain Vrignaud
[test]
159 21 Romain Vrignaud
    manifest   = /etc/puppet/test/manifests/site.pp
160 21 Romain Vrignaud
    modulepath = /etc/puppet/test/modules
161 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/test/fileserver.conf
162 21 Romain Vrignaud
    templatedir=/etc/puppet/test/templates/
163 22 Romain Vrignaud
</pre>
164 21 Romain Vrignaud
165 21 Romain Vrignaud
166 15 Romain Vrignaud
h1. How do I fix the puppet class import problem with multiple puppet environment ?
167 15 Romain Vrignaud
168 15 Romain Vrignaud
If Foreman don't seems to be aware of all your environment and is able to import only your "production" environment (see http://theforeman.org/issues/1159),
169 34 Christoph Witzany
try to add ":puppet_conf: /etc/puppet/puppet.conf" in your settings.yaml.
170 17 Martin Hovmöller
171 17 Martin Hovmöller
h1. Unable to connect to Hypervisor?
172 18 Martin Hovmöller
173 17 Martin Hovmöller
Make sure the user that's actually running foreman can connect to your remote hypervisor (for instance by running sudo -u foreman virsh -c qemu+ssh://root@<host>/session list).
174 19 Martin Hovmöller
175 17 Martin Hovmöller
If you are running foreman through phusion passenger, ps auxwwf | grep "R[a]ils: /usr/share/foreman" | awk '{ print $1 }' will give you the user that's running foreman. If you find that it's not being run by foreman, do the following:
176 17 Martin Hovmöller
cd ~foreman
177 17 Martin Hovmöller
chown foreman config/environment.rb
178 17 Martin Hovmöller
touch ~foreman/tmp/restart.txt
179 24 chris alfonso
180 24 chris alfonso
h1. Routing errors when running rake test?
181 24 chris alfonso
182 24 chris alfonso
Edit the config/settings.yaml and set the :login: setting to true
183 25 Ohad Levy
184 25 Ohad Levy
h1. cannot load such file -- facter / puppet (LoadError)
185 25 Ohad Levy
186 26 Ohad Levy
if you running foreman under RVM or custom ruby installation, you would need to get puppet installed as a gem (as puppet / facter packages are not in your ruby search path).
187 25 Ohad Levy
188 25 Ohad Levy
simply add another file to FOREMAN_DIR/bundler.d
189 25 Ohad Levy
<pre>
190 37 Dominic Cleal
echo "gem 'puppet'" > bundler.d/Gemfile.local.rb
191 1 Ohad Levy
</pre>
192 37 Dominic Cleal
193 37 Dominic Cleal
Edit the config/settings.yaml and add: @:puppetgem: true@
194 28 Bash Shell
195 28 Bash Shell
h1. Causes of "Error 400 Bad Request", and other smart-proxy related errors in the Foreman interface:
196 28 Bash Shell
197 29 Bash Shell
* Check the sudoers file on the proxy, if the user "foreman-proxy" can run "puppetca". The command puppetca is un-available in puppet 3.0. Workaround is to create a wrapper script.
198 29 Bash Shell
* From the cmd line, check if the user can run "puppetca" and "nsupdate" properly.
199 28 Bash Shell
* Check if Bind is listening on 127.0.0.1. The proxy connects to localhost only, so this is required. 
200 28 Bash Shell
* Check if the foreman-proxy user "foreman-proxy" can read the Bind rndc keys.
201 29 Bash Shell
* In Ubuntu, you will have to tell apparmor to allow Bind to write to zone files and journals. If your zone files are in /etc/bind/zones/, add "/etc/bind/zones/** rw," to /etc/apparmor.d/usr.sbin.named.
202 28 Bash Shell
* If using Ubuntu Libvirt, and the "Virtual Machine" tab is empty, then you most likely need to create a default storage pool:
203 28 Bash Shell
<pre>
204 28 Bash Shell
cat /etc/libvirt/storage/default.xml 
205 28 Bash Shell
<pool type='dir'>
206 28 Bash Shell
  <name>default</name>
207 28 Bash Shell
  <uuid>35e0bbce-4019-ca88-6dec-b1a0c1be774b</uuid>
208 28 Bash Shell
  <capacity>0</capacity>
209 28 Bash Shell
  <allocation>0</allocation>
210 28 Bash Shell
  <available>0</available>
211 28 Bash Shell
  <source>
212 28 Bash Shell
  </source>
213 28 Bash Shell
  <target>
214 28 Bash Shell
    <path>/var/lib/libvirt/images</path>
215 28 Bash Shell
    <permissions>
216 28 Bash Shell
      <mode>0700</mode>
217 28 Bash Shell
      <owner>-1</owner>
218 28 Bash Shell
      <group>-1</group>
219 28 Bash Shell
    </permissions>
220 28 Bash Shell
  </target>
221 28 Bash Shell
</pool>
222 28 Bash Shell
</pre>
223 28 Bash Shell
* On Ubuntu Libvirt, you may have to change /etc/libvirt/libvirtd.conf to listen on TCP.
224 31 Benjamin Papillon
225 31 Benjamin Papillon
h1. Using Webrick you get : Error 400 on SERVER: Could not find node '<node fqdn>'; cannot compile
226 31 Benjamin Papillon
227 32 Benjamin Papillon
If you are using Foreman with webrick and you get an error downloading your catalog, maybe you encouter the issue noted in bug #1507
228 31 Benjamin Papillon
Edit the node.rb script and replace the following :
229 31 Benjamin Papillon
230 31 Benjamin Papillon
<pre>
231 31 Benjamin Papillon
<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000*/*",<br />
232 31 Benjamin Papillon
</pre>
233 31 Benjamin Papillon
234 31 Benjamin Papillon
by this :
235 31 Benjamin Papillon
236 31 Benjamin Papillon
<pre>
237 31 Benjamin Papillon
<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000",<br />
238 31 Benjamin Papillon
</pre>
239 35 Charles Gomes
240 35 Charles Gomes
h1. DHCP Provisioning Fails with: "dhcpctl_connect: not found"
241 35 Charles Gomes
242 35 Charles Gomes
Depending on the version of ISC DHCP you are using a wrong key will return "dhcpctl_connect: not found" , this misleads to a connection problem.
243 35 Charles Gomes
To solve basically check if your KEY NAME matches your proxy settings.yml and also matches on your dhcpd.conf
244 38 Charles Gomes
245 38 Charles Gomes
h1. Foreman is showing : Error message: Could not find json-1.5.5 in any of the sources
246 38 Charles Gomes
247 38 Charles Gomes
If you have a problem after doing a yum update ruby bundler needs to re-run.
248 38 Charles Gomes
This could happen after an update or chane of ruby packages.
249 39 Charles Gomes
250 39 Charles Gomes
<pre>
251 38 Charles Gomes
rm ~foreman/Gemfile.lock
252 1 Ohad Levy
cd ~foreman 
253 38 Charles Gomes
bundle install --local
254 39 Charles Gomes
</pre>
255 40 Greg Sutcliffe
256 40 Greg Sutcliffe
h1. VNC console in the Foreman UI refuses to connect to my VM: Server Disconnected
257 40 Greg Sutcliffe
258 42 Greg Sutcliffe
Check what version of python you have installed - noVNC requires python2. If you have multiple python versions installed, you can edit `extras/noVNC/websockify.py` and change the shebang on line 1
259 44 Dominic Cleal
260 44 Dominic Cleal
h1. Unable to upgrade to Puppet 3.2 due to rubygem-net-ldap dependency issues
261 44 Dominic Cleal
262 44 Dominic Cleal
There are conflicting version requirements with rubygem-net-ldap between the Foreman and Puppet Labs dependency repos since Puppet 3.2.  Since Foreman 1.1, the rubygem-net-ping package is no longer required for the proxy, so first remove this and you should either be able to upgrade or remove rubygem-net-ldap altogether if there's no dependency any more.
263 45 Dominic Cleal
264 45 Dominic Cleal
h1. How do I change the FQDN of the Foreman host?
265 45 Dominic Cleal
266 45 Dominic Cleal
# Some settings in the Foreman UI use the hostname, update under More, Settings, e.g.
267 45 Dominic Cleal
#* General, foreman_url
268 45 Dominic Cleal
#* Provisioning, ssl_certificate and ssl_priv_key
269 45 Dominic Cleal
# The registered smart proxy URL if you've installed it, this can be edited via More>Configuration>Smart Proxies
270 45 Dominic Cleal
# Puppet SSL certs: generate new ones with @puppet cert generate FQDN@
271 45 Dominic Cleal
# Apache configs: update conf.d/{foreman,puppet}.conf with new SSL cert filenames, ServerName and VirtualHost IP addresses if applicable
272 45 Dominic Cleal
# Smart proxy configuration: update SSL cert filenames
273 46 Dominic Cleal
# URLs in the puppetmaster's ENC script (/etc/puppet/node.rb) and report script (foreman.rb)
274 47 Stephen Hoekstra
275 47 Stephen Hoekstra
h1. My deleted host keeps reappearing in the Hosts tab
276 47 Stephen Hoekstra
277 47 Stephen Hoekstra
After a period of time a deleted host reappears in the Host tab although the host no longer exists.  A potential reason for this could be the --push-facts cron job is pushing the facts to the Foreman and the Foreman expects the host to check in but never does.
278 47 Stephen Hoekstra
279 47 Stephen Hoekstra
To fix:
280 47 Stephen Hoekstra
<pre>
281 47 Stephen Hoekstra
rm /var/lib/puppet/yaml/facts/$hostname.yaml
282 47 Stephen Hoekstra
rm /var/lib/puppet/yaml/node/$hostname.yaml
283 47 Stephen Hoekstra
</pre>
284 48 Samarendra M Hedaoo
285 48 Samarendra M Hedaoo
h1. My node's environment is being reset to 'X' even though the puppet.conf on the host has environment = 'Y'. The Foreman settings have default_puppet_environment set as 'X' and enc_environment as False. I am using Puppet 3 on the node.
286 48 Samarendra M Hedaoo
287 48 Samarendra M Hedaoo
Foreman expects the node to specify it's own environment, if enc_environment is False. If no environment is returned by the node, it will override it with the value of default_puppet_environment. This was good till Puppet 3. But Puppet 3 no longer returns environment as a fact and hence Foreman assumes that the environment is not set.
288 49 Dominic Cleal
289 49 Dominic Cleal
h1. No A and/or PTR records are created in DNS for new hosts
290 49 Dominic Cleal
291 49 Dominic Cleal
First ensure you have a smart proxy registered with the DNS feature, with a DNS server installed and configured for the appropriate forward and reverse DNS zones.
292 49 Dominic Cleal
293 49 Dominic Cleal
In Foreman, under More, Provisioning, Domains, edit your domain, change the DNS proxy setting to the appropriate proxy server.  This will enable A records to be created for hosts built in that domain.
294 49 Dominic Cleal
295 49 Dominic Cleal
Next, under More, Provisioning, Subnets, edit your subnet, change the DNS proxy to the appropriate proxy for the reverse DNS zone.  This will cause PTR records to be added for hosts with NICs in that subnet.
296 49 Dominic Cleal
297 49 Dominic Cleal
Both parts are needed for a complete DNS setup.
298 49 Dominic Cleal
299 49 Dominic Cleal
h1. No TFTP menus or files are created for new hosts
300 49 Dominic Cleal
301 49 Dominic Cleal
This requires a proxy server registered with the TFTP feature, and a TFTP daemon running on it.  Foreman will write directly to the TFTP root directory, as configured in the proxy settings.  Do ensure the "tftp_servername" setting is also correct, this is the IP address that gets passed to the DHCP server for the @next-server@ option.
302 49 Dominic Cleal
303 49 Dominic Cleal
In Foreman, under More, Provisioning, Subnets, edit your subnet, change the TFTP proxy to the appropriate proxy server for the subnet.  This will cause hosts built in this subnet to receive a PXELinux menu for their primary MAC address.
304 50 Dominic Cleal
305 50 Dominic Cleal
h1. After Foreman 1.3 upgrade, reports and ENC fact uploads no longer work
306 50 Dominic Cleal
307 51 Lukas Zapletal
We changed how the fact and report uploads work in the 1.3 release. During 1.3 upgrade, read "release notes":http://theforeman.org/manuals/1.3/index.html#Upgradenotes and update both the report processor (foreman.rb) and ENC script (node.rb), otherwise errors will be logged in the production.log (return code 400):
308 51 Lukas Zapletal
    
309 51 Lukas Zapletal
    DEPRECATION: /reports/create is deprecated, update your report processor to POST to /api/reports
310 51 Lukas Zapletal
    See the Foreman 1.3 release notes for a new example report processor
311 51 Lukas Zapletal
312 51 Lukas Zapletal
In the release candidates, the error was:
313 50 Dominic Cleal
314 50 Dominic Cleal
    ActionController::RoutingError (No route matches [POST] "/fact_values/create")
315 50 Dominic Cleal
    ActionController::RoutingError (No route matches [POST] "/reports/create")