Project

General

Profile

Troubleshooting » History » Version 97

Joel Kåberg, 09/15/2017 06:53 AM

1 3 Ohad Levy
{{toc}}
2 3 Ohad Levy
3 43 Lukas Zapletal
h1. How to provide enough info to get help?
4 43 Lukas Zapletal
5 43 Lukas Zapletal
Foreman is pretty complex software project with many components and extensive configuration. Always try to describe best your specific case (OS, version, ruby and foreman version, database, specific configuration). From Foreman version 1.2 you can use _foreman-debug_ script which collects all these information and also filters out things like passwords or tokens. For earlier Foreman releases you can use this bash script directly from git: https://github.com/theforeman/foreman/blob/develop/script/foreman-debug
6 43 Lukas Zapletal
7 67 Dominic Cleal
h1. ERF error code
8 67 Dominic Cleal
9 67 Dominic Cleal
If you see an ERF error code, such as ERF12-7740, look it up on [[ErrorCodes]].
10 67 Dominic Cleal
11 1 Ohad Levy
h1. rake aborted!
12 1 Ohad Levy
13 1 Ohad Levy
I get the following error while running rake db:migrate
14 1 Ohad Levy
<pre>
15 1 Ohad Levy
rake aborted!
16 1 Ohad Levy
undefined method `reenable' for <Rake::Task db:schema:dump => [environment]>:Rake::Task
17 1 Ohad Levy
</pre>
18 1 Ohad Levy
19 1 Ohad Levy
this means you have an old version of rake (older than 0.87), you can verify it by typing:
20 1 Ohad Levy
<pre>
21 1 Ohad Levy
rake --version
22 1 Ohad Levy
</pre>
23 1 Ohad Levy
24 1 Ohad Levy
Usually this error is "safe" to ignore, however, it is recommended to upgrade to rake >= 0.87
25 2 Ohad Levy
26 10 Corey Osman
27 20 Florian Koch
h1. Unable to save: failed to detect boot server: getaddrinfo: Name or service not known
28 1 Ohad Levy
29 20 Florian Koch
if you have registred your smart-proxys via name, and these name are not resolvable, you get this error, you can add the name to your dns server or set 
30 20 Florian Koch
:tftp_servername: in smart-proxy settings to the ip of the smart-proxy.
31 10 Corey Osman
32 7 Corey Osman
h1. I use puppet enterprise, what now?
33 7 Corey Osman
34 7 Corey Osman
As of PE version 1.0, foreman won't install easily without a few modifications.  Since PE ships with its own version of Ruby you will need to install a few gems.
35 12 Corey Osman
Basically you need to tell foreman to use the PE provided ruby library instead.  Your not required to run the puppet version of ruby either as many distros now ship with ruby 1.8.7.
36 7 Corey Osman
37 8 Ohad Levy
# You first need to install the pe-ruby-devel package that ships with PE.  You can find this in your packages folder wherever you extracted PE.
38 1 Ohad Levy
      a. rpm -ivh pe-ruby-devel-1.8.7.302-1.el5.i386.rpm 
39 8 Ohad Levy
# Update your path variable to use the ruby version PE ships with (export PATH=/opt/puppet/bin:$PATH)
40 8 Ohad Levy
# Check to make sure your PATH was updated ("which gem" should return /opt/puppet/bin/gem)
41 8 Ohad Levy
# gem install json
42 8 Ohad Levy
# gem install rest-client
43 7 Corey Osman
(install the gem database driver you use)
44 8 Ohad Levy
# gem install sqlite3 or gem install mysql  ( you may have additional dependencies that would need to be resolved first)
45 7 Corey Osman
46 7 Corey Osman
Some of the gem dependencies can be resolved by installing the development library for that package (example: mysql-devel)
47 7 Corey Osman
48 7 Corey Osman
I found it difficult to install the sqlite3 gem because of dependency hell so I just setup mysql instead which was far easier to setup.  Additionally mysql is much more scalable than sqlite.
49 7 Corey Osman
50 7 Corey Osman
You can get a list of the currently installed gems by using: gem list.
51 1 Ohad Levy
52 10 Corey Osman
If your running the foreman init script or /etc/sysconfig/foreman configuration file.  You may need to update apache/passenger configurations that point to the PE ruby version.
53 6 Corey Osman
54 6 Corey Osman
55 2 Ohad Levy
h1. error: Entry 'some file' not uptodate. Cannot merge.
56 2 Ohad Levy
57 2 Ohad Levy
If you downloaded Foreman from source (git), it could be that some of the files you have modified (or were modified automatically) conflicts with another file Foreman wants to upgrade.
58 2 Ohad Levy
59 2 Ohad Levy
If you don't think that you edited this file (e.g. db/schema.rb) it is usually safe to do:
60 2 Ohad Levy
<pre>
61 2 Ohad Levy
git checkout 'some file'
62 2 Ohad Levy
</pre> 
63 2 Ohad Levy
64 2 Ohad Levy
This will revert the file to its previous condition (as in the repo at the time of checkout) and now you should be able to get the latest version by:
65 2 Ohad Levy
<pre>
66 2 Ohad Levy
git pull
67 2 Ohad Levy
</pre>
68 3 Ohad Levy
69 3 Ohad Levy
h1. Strange errors with passenger
70 3 Ohad Levy
71 3 Ohad Levy
Passenger executes foreman, based on the owner of the config/environmnets.rb file, make sure that this use can actually access:
72 3 Ohad Levy
* puppet configuration files
73 3 Ohad Levy
* sqlite database (if using sqlite)
74 3 Ohad Levy
* write to log, tmp directories
75 3 Ohad Levy
76 3 Ohad Levy
see source:extras/puppet/foreman for complete puppet based setup for Foreman.
77 4 Bash Shell
78 4 Bash Shell
79 4 Bash Shell
80 4 Bash Shell
h1. error: Could not send facts to Foreman: wrong Content-Length format (RuntimeError)
81 4 Bash Shell
82 4 Bash Shell
You might see this error if you run the HTTP push facts script or the sample external nodes script.
83 4 Bash Shell
This is most likely due to older version of the mongrel gem. Please try and update your gems.
84 4 Bash Shell
85 4 Bash Shell
<pre>
86 4 Bash Shell
gem update mongrel
87 4 Bash Shell
</pre>
88 5 Lukas Zapletal
89 5 Lukas Zapletal
h1. Is my Foreman instance running?
90 5 Lukas Zapletal
91 5 Lukas Zapletal
There is simple status service that returns JSON with "result" message "ok" when the instance is up and running. It also "pings" database and returns lag. Example:
92 5 Lukas Zapletal
93 5 Lukas Zapletal
<pre>
94 5 Lukas Zapletal
$ curl -k -H "Accept: application/json" http://instance:3000/status
95 5 Lukas Zapletal
{"status":200,"result":"ok","db_duration_ms":"5"}
96 5 Lukas Zapletal
</pre>
97 9 Dis Connect
98 9 Dis Connect
h1. How do I enable debugging?
99 9 Dis Connect
100 88 Dominic Cleal
h2. On Foreman 1.9 or newer
101 88 Dominic Cleal
102 88 Dominic Cleal
See "Foreman debugging":http://theforeman.org/manuals/1.9/index.html#7.2Debugging from the documentation.
103 88 Dominic Cleal
104 79 Dominic Cleal
h2. On Foreman 1.8 or older
105 79 Dominic Cleal
106 72 Ashton Davis
Edit ~foreman/config/environments/production.rb and uncomment this line:
107 71 Ashton Davis
<pre>
108 71 Ashton Davis
config.log_level = :debug
109 71 Ashton Davis
</pre>
110 1 Ohad Levy
111 79 Dominic Cleal
And reload the config (nicely):
112 79 Dominic Cleal
<pre>
113 79 Dominic Cleal
touch ~foreman/tmp/restart.txt
114 86 Dominic Cleal
</pre>
115 85 Dominic Cleal
116 79 Dominic Cleal
h2. Compute resource debugging
117 71 Ashton Davis
118 72 Ashton Davis
*Do not run this command unless specifically told to do so.*
119 71 Ashton Davis
To enable debugging of Compute Resources HTTP requests, you should pass some environment variables:
120 71 Ashton Davis
<pre>
121 71 Ashton Davis
  cd ~foreman
122 71 Ashton Davis
  EXCON_DEBUG=true DEBUG=true ./script/rails s
123 71 Ashton Davis
</pre>
124 13 Corey Osman
125 78 Dominic Cleal
h1. How do I enable smart proxy debugging?
126 78 Dominic Cleal
127 78 Dominic Cleal
Edit /etc/foreman-proxy/settings.yml and change or uncomment the log_level parameter, setting it to DEBUG:
128 78 Dominic Cleal
129 78 Dominic Cleal
    :log_level: DEBUG
130 78 Dominic Cleal
131 78 Dominic Cleal
Then restart the service:
132 78 Dominic Cleal
133 78 Dominic Cleal
    service foreman-proxy restart
134 78 Dominic Cleal
135 78 Dominic Cleal
Logs will go to /var/log/foreman-proxy/proxy.log and access.log by default.
136 78 Dominic Cleal
137 78 Dominic Cleal
If the service is crashing immediately, failing to start up or continually returns 500 Internal Server error, start it up in foreground mode.  To do this, disable the daemonize parameter in /etc/foreman-proxy/settings.yml:
138 78 Dominic Cleal
139 78 Dominic Cleal
    :daemonize: false
140 78 Dominic Cleal
141 78 Dominic Cleal
And then run the smart proxy process:
142 78 Dominic Cleal
143 78 Dominic Cleal
    sudo -u foreman-proxy /usr/share/foreman-proxy/bin/smart-proxy
144 78 Dominic Cleal
145 78 Dominic Cleal
Make the request from Foreman again, and logging should be shown on the console.  Don't forget to change daemonize back to true before starting the service.
146 78 Dominic Cleal
147 13 Corey Osman
h1. [[Unattended Provisioning Troubleshooting]]
148 14 Florian Koch
149 14 Florian Koch
h1. Puppet and Storeconfigs: err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep
150 14 Florian Koch
151 14 Florian Koch
Your activerecord Version is to new, you need activrecord 3.0.10 see http://projects.puppetlabs.com/issues/9290
152 14 Florian Koch
153 14 Florian Koch
<pre>
154 14 Florian Koch
gem uninstall activerecord
155 14 Florian Koch
gem install activerecord -v 3.0.10
156 14 Florian Koch
</pre>
157 15 Romain Vrignaud
158 15 Romain Vrignaud
159 15 Romain Vrignaud
160 15 Romain Vrignaud
h1. How do I fix this error: "Unable to find directory $confdir/environments/$environment/modules in environment master" ?
161 15 Romain Vrignaud
162 15 Romain Vrignaud
If you've followed http://docs.puppetlabs.com/guides/environment.html#configuring-environments-on-the-puppet-master to 
163 15 Romain Vrignaud
create a multi-environment puppet configuration, you're initial rake may look something like this:
164 15 Romain Vrignaud
<pre>
165 15 Romain Vrignaud
rake puppet:import:puppet_classes RAILS_ENV=production
166 15 Romain Vrignaud
(in /usr/share/foreman)
167 15 Romain Vrignaud
Evaluating possible changes to your installation
168 15 Romain Vrignaud
Problems were detected during the evaluation phase
169 15 Romain Vrignaud
170 15 Romain Vrignaud
Unable to find directory $confdir/environments/$environment/modules in environment master
171 15 Romain Vrignaud
172 15 Romain Vrignaud
Please fix these issues and try again
173 15 Romain Vrignaud
</pre>
174 15 Romain Vrignaud
This is because Foreman does not currently evaluate $confdir.  There is a feature request open on this issue.  In the 
175 15 Romain Vrignaud
meantime, hard code your $confdir in your puppet.conf.
176 15 Romain Vrignaud
177 21 Romain Vrignaud
h1. Smart-proxy do not recognize my puppet environment
178 23 Romain Vrignaud
179 21 Romain Vrignaud
If I query smart-proxy and it return empty puppet environment :
180 22 Romain Vrignaud
<pre>
181 21 Romain Vrignaud
curl -k -H "Content-Type:application/json" -H "Accept:application/json" http://puppet:8443/puppet/environments
182 21 Romain Vrignaud
=> []
183 22 Romain Vrignaud
</pre>
184 21 Romain Vrignaud
185 21 Romain Vrignaud
You may have to add in your puppetmaster's puppet.conf environment definition like :
186 21 Romain Vrignaud
187 22 Romain Vrignaud
<pre>
188 21 Romain Vrignaud
[production]
189 21 Romain Vrignaud
    manifest   = /etc/puppet/manifests/site.pp
190 21 Romain Vrignaud
    modulepath = /etc/puppet/modules
191 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/fileserver.conf
192 21 Romain Vrignaud
    templatedir=/etc/puppet/templates/
193 21 Romain Vrignaud
194 21 Romain Vrignaud
[preprod]
195 21 Romain Vrignaud
    manifest   = /etc/puppet/preprod/manifests/site.pp
196 21 Romain Vrignaud
    modulepath = /etc/puppet/preprod/modules
197 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/preprod/fileserver.conf
198 21 Romain Vrignaud
    templatedir=/etc/puppet/preprod/templates
199 21 Romain Vrignaud
200 21 Romain Vrignaud
[development]
201 21 Romain Vrignaud
    manifest   = /etc/puppet/development/manifests/site.pp
202 21 Romain Vrignaud
    modulepath = /etc/puppet/development/modules
203 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/development/fileserver.conf
204 21 Romain Vrignaud
    templatedir=/etc/puppet/development/templates/
205 21 Romain Vrignaud
206 21 Romain Vrignaud
[test]
207 21 Romain Vrignaud
    manifest   = /etc/puppet/test/manifests/site.pp
208 21 Romain Vrignaud
    modulepath = /etc/puppet/test/modules
209 21 Romain Vrignaud
    fileserverconfig = /etc/puppet/test/fileserver.conf
210 21 Romain Vrignaud
    templatedir=/etc/puppet/test/templates/
211 22 Romain Vrignaud
</pre>
212 21 Romain Vrignaud
213 21 Romain Vrignaud
214 15 Romain Vrignaud
h1. How do I fix the puppet class import problem with multiple puppet environment ?
215 15 Romain Vrignaud
216 15 Romain Vrignaud
If Foreman don't seems to be aware of all your environment and is able to import only your "production" environment (see http://theforeman.org/issues/1159),
217 34 Christoph Witzany
try to add ":puppet_conf: /etc/puppet/puppet.conf" in your settings.yaml.
218 17 Martin Hovmöller
219 17 Martin Hovmöller
h1. Unable to connect to Hypervisor?
220 18 Martin Hovmöller
221 74 Neil Katin
Make sure the user that's actually running foreman can connect to your remote hypervisor (for instance by running sudo -u foreman virsh -c qemu+ssh://root@<host>/system list).
222 19 Martin Hovmöller
223 17 Martin Hovmöller
If you are running foreman through phusion passenger, ps auxwwf | grep "R[a]ils: /usr/share/foreman" | awk '{ print $1 }' will give you the user that's running foreman. If you find that it's not being run by foreman, do the following:
224 17 Martin Hovmöller
cd ~foreman
225 17 Martin Hovmöller
chown foreman config/environment.rb
226 17 Martin Hovmöller
touch ~foreman/tmp/restart.txt
227 24 chris alfonso
228 24 chris alfonso
h1. Routing errors when running rake test?
229 24 chris alfonso
230 24 chris alfonso
Edit the config/settings.yaml and set the :login: setting to true
231 25 Ohad Levy
232 25 Ohad Levy
h1. cannot load such file -- facter / puppet (LoadError)
233 25 Ohad Levy
234 52 Dominic Cleal
If you running foreman under RVM or custom ruby installation, you would need to get facter installed as a gem (as facter packages are not in your ruby search path).
235 25 Ohad Levy
236 1 Ohad Levy
simply add another file to FOREMAN_DIR/bundler.d
237 52 Dominic Cleal
<pre>
238 52 Dominic Cleal
echo "gem 'facter'" > bundler.d/Gemfile.local.rb
239 52 Dominic Cleal
</pre>
240 52 Dominic Cleal
241 52 Dominic Cleal
h2. Pre-Foreman 1.3
242 52 Dominic Cleal
243 52 Dominic Cleal
Prior to 1.3, puppet was also a dependency, so instead use:
244 52 Dominic Cleal
245 25 Ohad Levy
<pre>
246 37 Dominic Cleal
echo "gem 'puppet'" > bundler.d/Gemfile.local.rb
247 1 Ohad Levy
</pre>
248 37 Dominic Cleal
249 37 Dominic Cleal
Edit the config/settings.yaml and add: @:puppetgem: true@
250 28 Bash Shell
251 28 Bash Shell
h1. Causes of "Error 400 Bad Request", and other smart-proxy related errors in the Foreman interface:
252 28 Bash Shell
253 29 Bash Shell
* Check the sudoers file on the proxy, if the user "foreman-proxy" can run "puppetca". The command puppetca is un-available in puppet 3.0. Workaround is to create a wrapper script.
254 29 Bash Shell
* From the cmd line, check if the user can run "puppetca" and "nsupdate" properly.
255 28 Bash Shell
* Check if Bind is listening on 127.0.0.1. The proxy connects to localhost only, so this is required. 
256 28 Bash Shell
* Check if the foreman-proxy user "foreman-proxy" can read the Bind rndc keys.
257 29 Bash Shell
* In Ubuntu, you will have to tell apparmor to allow Bind to write to zone files and journals. If your zone files are in /etc/bind/zones/, add "/etc/bind/zones/** rw," to /etc/apparmor.d/usr.sbin.named.
258 28 Bash Shell
* If using Ubuntu Libvirt, and the "Virtual Machine" tab is empty, then you most likely need to create a default storage pool:
259 28 Bash Shell
<pre>
260 28 Bash Shell
cat /etc/libvirt/storage/default.xml 
261 28 Bash Shell
<pool type='dir'>
262 28 Bash Shell
  <name>default</name>
263 28 Bash Shell
  <uuid>35e0bbce-4019-ca88-6dec-b1a0c1be774b</uuid>
264 28 Bash Shell
  <capacity>0</capacity>
265 28 Bash Shell
  <allocation>0</allocation>
266 28 Bash Shell
  <available>0</available>
267 28 Bash Shell
  <source>
268 28 Bash Shell
  </source>
269 28 Bash Shell
  <target>
270 28 Bash Shell
    <path>/var/lib/libvirt/images</path>
271 28 Bash Shell
    <permissions>
272 28 Bash Shell
      <mode>0700</mode>
273 28 Bash Shell
      <owner>-1</owner>
274 28 Bash Shell
      <group>-1</group>
275 28 Bash Shell
    </permissions>
276 28 Bash Shell
  </target>
277 28 Bash Shell
</pool>
278 28 Bash Shell
</pre>
279 28 Bash Shell
* On Ubuntu Libvirt, you may have to change /etc/libvirt/libvirtd.conf to listen on TCP.
280 31 Benjamin Papillon
281 31 Benjamin Papillon
h1. Using Webrick you get : Error 400 on SERVER: Could not find node '<node fqdn>'; cannot compile
282 31 Benjamin Papillon
283 32 Benjamin Papillon
If you are using Foreman with webrick and you get an error downloading your catalog, maybe you encouter the issue noted in bug #1507
284 31 Benjamin Papillon
Edit the node.rb script and replace the following :
285 31 Benjamin Papillon
286 31 Benjamin Papillon
<pre>
287 31 Benjamin Papillon
<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000*/*",<br />
288 31 Benjamin Papillon
</pre>
289 31 Benjamin Papillon
290 31 Benjamin Papillon
by this :
291 31 Benjamin Papillon
292 31 Benjamin Papillon
<pre>
293 31 Benjamin Papillon
<br />SETTINGS = {<br /> :url => "http://<node fqdn>:3000",<br />
294 31 Benjamin Papillon
</pre>
295 35 Charles Gomes
296 35 Charles Gomes
h1. DHCP Provisioning Fails with: "dhcpctl_connect: not found"
297 35 Charles Gomes
298 35 Charles Gomes
Depending on the version of ISC DHCP you are using a wrong key will return "dhcpctl_connect: not found" , this misleads to a connection problem.
299 35 Charles Gomes
To solve basically check if your KEY NAME matches your proxy settings.yml and also matches on your dhcpd.conf
300 38 Charles Gomes
301 38 Charles Gomes
h1. Foreman is showing : Error message: Could not find json-1.5.5 in any of the sources
302 38 Charles Gomes
303 38 Charles Gomes
If you have a problem after doing a yum update ruby bundler needs to re-run.
304 38 Charles Gomes
This could happen after an update or chane of ruby packages.
305 39 Charles Gomes
306 39 Charles Gomes
<pre>
307 38 Charles Gomes
rm ~foreman/Gemfile.lock
308 1 Ohad Levy
cd ~foreman 
309 38 Charles Gomes
bundle install --local
310 39 Charles Gomes
</pre>
311 40 Greg Sutcliffe
312 40 Greg Sutcliffe
h1. VNC console in the Foreman UI refuses to connect to my VM: Server Disconnected
313 40 Greg Sutcliffe
314 42 Greg Sutcliffe
Check what version of python you have installed - noVNC requires python2. If you have multiple python versions installed, you can edit `extras/noVNC/websockify.py` and change the shebang on line 1
315 44 Dominic Cleal
316 44 Dominic Cleal
h1. Unable to upgrade to Puppet 3.2 due to rubygem-net-ldap dependency issues
317 44 Dominic Cleal
318 44 Dominic Cleal
There are conflicting version requirements with rubygem-net-ldap between the Foreman and Puppet Labs dependency repos since Puppet 3.2.  Since Foreman 1.1, the rubygem-net-ping package is no longer required for the proxy, so first remove this and you should either be able to upgrade or remove rubygem-net-ldap altogether if there's no dependency any more.
319 45 Dominic Cleal
320 45 Dominic Cleal
h1. How do I change the FQDN of the Foreman host?
321 45 Dominic Cleal
322 93 Dominic Cleal
This documentation is now part of the manual at https://theforeman.org/manuals/latest/index.html#5.5.2Recovery
323 47 Stephen Hoekstra
324 47 Stephen Hoekstra
h1. My deleted host keeps reappearing in the Hosts tab
325 47 Stephen Hoekstra
326 47 Stephen Hoekstra
After a period of time a deleted host reappears in the Host tab although the host no longer exists.  A potential reason for this could be the --push-facts cron job is pushing the facts to the Foreman and the Foreman expects the host to check in but never does.
327 47 Stephen Hoekstra
328 47 Stephen Hoekstra
To fix:
329 47 Stephen Hoekstra
<pre>
330 47 Stephen Hoekstra
rm /var/lib/puppet/yaml/facts/$hostname.yaml
331 47 Stephen Hoekstra
rm /var/lib/puppet/yaml/node/$hostname.yaml
332 47 Stephen Hoekstra
</pre>
333 48 Samarendra M Hedaoo
334 48 Samarendra M Hedaoo
h1. My node's environment is being reset to 'X' even though the puppet.conf on the host has environment = 'Y'. The Foreman settings have default_puppet_environment set as 'X' and enc_environment as False. I am using Puppet 3 on the node.
335 48 Samarendra M Hedaoo
336 48 Samarendra M Hedaoo
Foreman expects the node to specify it's own environment, if enc_environment is False. If no environment is returned by the node, it will override it with the value of default_puppet_environment. This was good till Puppet 3. But Puppet 3 no longer returns environment as a fact and hence Foreman assumes that the environment is not set.
337 49 Dominic Cleal
338 49 Dominic Cleal
h1. No A and/or PTR records are created in DNS for new hosts
339 49 Dominic Cleal
340 49 Dominic Cleal
First ensure you have a smart proxy registered with the DNS feature, with a DNS server installed and configured for the appropriate forward and reverse DNS zones.
341 49 Dominic Cleal
342 49 Dominic Cleal
In Foreman, under More, Provisioning, Domains, edit your domain, change the DNS proxy setting to the appropriate proxy server.  This will enable A records to be created for hosts built in that domain.
343 49 Dominic Cleal
344 49 Dominic Cleal
Next, under More, Provisioning, Subnets, edit your subnet, change the DNS proxy to the appropriate proxy for the reverse DNS zone.  This will cause PTR records to be added for hosts with NICs in that subnet.
345 49 Dominic Cleal
346 49 Dominic Cleal
Both parts are needed for a complete DNS setup.
347 49 Dominic Cleal
348 49 Dominic Cleal
h1. No TFTP menus or files are created for new hosts
349 49 Dominic Cleal
350 89 Dominic Cleal
This requires a proxy server registered with the TFTP feature, and a TFTP daemon running on it.  Foreman will write directly to the TFTP root directory, as configured in the proxy settings.  This should create a file per provisioning interface under /var/lib/tftpboot/pxelinux.cfg/ or similar.
351 1 Ohad Levy
352 89 Dominic Cleal
Do ensure the "tftp_servername" setting in @/etc/foreman-proxy/settings.d/tftp.yml@ is also correct, this is the IP address that gets passed to the DHCP server for the @next-server@ option.
353 89 Dominic Cleal
354 89 Dominic Cleal
In Foreman, there are a number of conditions for this to happen.  After fixing any of them, you either create a new host, or rebuild configs on an existing one from the host list or cycle the build mode (Cancel Build, then Build) to trigger an update.
355 89 Dominic Cleal
356 89 Dominic Cleal
Check the following
357 89 Dominic Cleal
358 89 Dominic Cleal
# Host must be managed, not discovered from Puppet etc.  Edit the host and click the _Manage host_ button.
359 89 Dominic Cleal
# Host must have a provisioning network interface, check the icons on the left under the host's _Interfaces_ tab.
360 89 Dominic Cleal
# The provisioning interface must have a subnet set.
361 90 Dominic Cleal
# The provisioning interface must be marked as _Managed_.
362 89 Dominic Cleal
# The subnet must have a TFTP proxy set: under Infrastructure > Subnets, select the subnet, set the TFTP Proxy under the Proxies tab.
363 89 Dominic Cleal
# The host must have an operating system set.
364 89 Dominic Cleal
# The host must have network build selected as its provisioning method, under the OS tab.  This can't be changed on an existing host.
365 89 Dominic Cleal
# Unattended mode must be enabled in @/etc/foreman/settings.yaml@
366 50 Dominic Cleal
367 91 Dominic Cleal
h1. No DHCP reservations are created for new hosts
368 91 Dominic Cleal
369 91 Dominic Cleal
This requires a proxy server registered with the DHCP feature, and a DHCP daemon running on it. Foreman will typically create DHCP reservations for every managed network interface on a host.
370 91 Dominic Cleal
371 91 Dominic Cleal
When using the ISC DHCP server (dhcpd), Foreman will use its OMAPI interface to request that the daemon writes a reservation, which it then writes to the leases file, e.g. /var/lib/dhcpd/dhcpd.leases or /var/lib/dhcp3/dhcpd.leases. Do note that the smart proxy never writes to config or leases files itself, this is all through the dhcpd - it only _reads_ them. The leases file is maintained by dhcpd and regularly flushed and cleaned up, so it may not exactly reflect reality.
372 91 Dominic Cleal
373 91 Dominic Cleal
In Foreman, there are a number of conditions required for reservations to be created.  After fixing any of them, you either create a new host, or rebuild configs on an existing one from the host list or cycle the build mode (Cancel Build, then Build) to trigger an update.
374 91 Dominic Cleal
375 91 Dominic Cleal
Check the following
376 91 Dominic Cleal
377 91 Dominic Cleal
# Host must be managed, not discovered from Puppet etc.  Edit the host and click the _Manage host_ button.
378 91 Dominic Cleal
# The interface must have a subnet set.
379 91 Dominic Cleal
# The interface must be marked as _Managed_.
380 91 Dominic Cleal
# The interface must have an IP address set, or a compute resource that allocates IPs (e.g. EC2) must be used.
381 91 Dominic Cleal
# The interface must have a MAC address address set, or a compute resource that allocates MACs (e.g. Libvirt, VMware) must be used.
382 91 Dominic Cleal
# The subnet must have a DHCP proxy set: under Infrastructure > Subnets, select the subnet, set the DHCP Proxy under the Proxies tab.
383 91 Dominic Cleal
# The subnet must have DHCP set as the boot mode: under Infrastructure > Subnets, select the subnet, set Boot Mode to DHCP (see also #14905)
384 91 Dominic Cleal
# The host must have an operating set if it has a provisioning network interface
385 91 Dominic Cleal
# Unattended mode must be enabled in @/etc/foreman/settings.yaml@
386 91 Dominic Cleal
387 50 Dominic Cleal
h1. After Foreman 1.3 upgrade, reports and ENC fact uploads no longer work
388 50 Dominic Cleal
389 51 Lukas Zapletal
We changed how the fact and report uploads work in the 1.3 release. During 1.3 upgrade, read "release notes":http://theforeman.org/manuals/1.3/index.html#Upgradenotes and update both the report processor (foreman.rb) and ENC script (node.rb), otherwise errors will be logged in the production.log (return code 400):
390 51 Lukas Zapletal
    
391 51 Lukas Zapletal
    DEPRECATION: /reports/create is deprecated, update your report processor to POST to /api/reports
392 51 Lukas Zapletal
    See the Foreman 1.3 release notes for a new example report processor
393 51 Lukas Zapletal
394 51 Lukas Zapletal
In the release candidates, the error was:
395 50 Dominic Cleal
396 50 Dominic Cleal
    ActionController::RoutingError (No route matches [POST] "/fact_values/create")
397 50 Dominic Cleal
    ActionController::RoutingError (No route matches [POST] "/reports/create")
398 53 Lukas Zapletal
399 53 Lukas Zapletal
h1. Unprocessable Entity error during installation
400 53 Lukas Zapletal
401 53 Lukas Zapletal
Foreman installer registers proxy which is being deployed automatically. If it is, for any reason, already present but under different name, Foreman reject to register the proxy with HTTP 422 error. Proxy registration can be either turned off, or name can be changed using @--foreman-proxy-register-in-foreman@ or @--foreman-proxy-registered-name@ installer options.
402 54 Dominic Cleal
403 54 Dominic Cleal
h1. Installation errors on Puppet 3.4.x or 2.7.24
404 54 Dominic Cleal
405 54 Dominic Cleal
When using Puppet 3.4.0 or 3.4.1, installation errors will be seen and running "puppet --version" after install will show the following:
406 54 Dominic Cleal
407 54 Dominic Cleal
    Could not intialize global default settings: undefined method `mode=' for #<Puppet::Settings::AutosignSetting:0x7fa026ad44e0>
408 54 Dominic Cleal
409 58 Dominic Cleal
This is caused by Puppet bug "PUP-1015":https://tickets.puppetlabs.com/browse/PUP-1015.  On 3.4.1 and 2.7.24, additionally the /etc/puppet/manifests/site.pp file will be set to mode 0600 rather than the default 0644.  This is "PUP-1255":https://tickets.puppetlabs.com/browse/PUP-1255.
410 57 Dominic Cleal
411 58 Dominic Cleal
Puppet 3.4.2 and 2.7.25 fix both of these issues, please ensure you use these versions instead of 3.4.0/1 or 2.7.24.
412 59 Dominic Cleal
413 59 Dominic Cleal
h1. Yum fails to install foreman-installer, Requires: rubygem(rest-client)
414 59 Dominic Cleal
415 59 Dominic Cleal
If you get this error, it indicates the EPEL repository is missing:
416 59 Dominic Cleal
417 59 Dominic Cleal
    Error: Package: rubygem-foreman_api-0.1.9-1.el6.noarch (foreman)
418 59 Dominic Cleal
           Requires: rubygem(rest-client) >= 1.6.1
419 59 Dominic Cleal
420 59 Dominic Cleal
This is a requirement for the installer and Foreman.  Download and install the epel-release RPM from here: http://dl.fedoraproject.org/pub/epel/6/x86_64/repoview/epel-release.html
421 60 Lukas Zapletal
422 60 Lukas Zapletal
h1. Foreman proxy fails to start with Are the values correct in settings.yml and do permissions allow reading?: Permission denied
423 60 Lukas Zapletal
424 60 Lukas Zapletal
If you are using puppet CA see here: http://theforeman.org/manuals/latest/index.html#4.3.7SSL
425 61 Stephen Benjamin
426 77 Tommy McNeely
h1. FreeIPA realm proxy fails with SSLv3 read server certificate B: certificate verify failed
427 77 Tommy McNeely
428 77 Tommy McNeely
If you see error messages like this in proxy.log:
429 77 Tommy McNeely
430 77 Tommy McNeely
    E, [2014-11-21T06:09:59.630126 #5631] ERROR -- : SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
431 77 Tommy McNeely
432 77 Tommy McNeely
The certificate authority (CA) that signed your IPA CA cert is not trusted by the proxy host. This is fairly common if the "foreman proxy" host is not an IPA server. You can trust the CA certificate of the IPA server...
433 77 Tommy McNeely
434 77 Tommy McNeely
    cp /etc/ipa/ca.crt /etc/pki/ca-trust/source/source
435 77 Tommy McNeely
    update-ca-trust
436 77 Tommy McNeely
437 77 Tommy McNeely
438 61 Stephen Benjamin
h1. FreeIPA realm proxy fails with "generic preauthentication failure"
439 61 Stephen Benjamin
440 61 Stephen Benjamin
If you see error messages like this in your proxy logs:
441 61 Stephen Benjamin
 
442 61 Stephen Benjamin
    E, [2014-05-15T19:28:08.211121 #3595] ERROR -- : Failed to initialise credential cache from keytab: krb5_get_init_creds_keytab:  Generic preauthentication failure
443 61 Stephen Benjamin
    E, [2014-05-15T19:28:08.211515 #3595] ERROR -- : Failed to initailize credentials cache from keytab: krb5_get_init_creds_keytab: Generic preauthentication failure
444 61 Stephen Benjamin
    D, [2014-05-15T19:28:08.211614 #3595] DEBUG -- : /usr/share/foreman-proxy/bin/../lib/proxy/kerberos.rb:13:in `init_krb5_ccache'
445 61 Stephen Benjamin
446 76 Tommy McNeely
First, check permissions, the /etc/foreman-proxy/freeipa.keytab file should be owned by foreman-proxy and mode 600.
447 76 Tommy McNeely
448 61 Stephen Benjamin
You may be running a much newer version of FreeIPA than the client which provides some unknown encryption types.
449 61 Stephen Benjamin
450 61 Stephen Benjamin
Run `klist -etk /etc/foreman-proxy/freeipa.keytab` and you'll see some unnamed enryption types like this:
451 61 Stephen Benjamin
452 61 Stephen Benjamin
    1 05/14/14 21:14:17 realm-proxy@EXAMPLE.COM (etype 25)
453 61 Stephen Benjamin
    1 05/14/14 21:14:17 realm-proxy@EXAMPLE.COM (etype 26)
454 61 Stephen Benjamin
455 61 Stephen Benjamin
To fix it, delete /etc/foreman-poxy/freeipa.keytab and refetch it, specifying only the enctypes your system knows about:
456 61 Stephen Benjamin
457 62 Stephen Benjamin
    ipa-getkeytab -s ipa.example.com -p realm-proxy@EXAMPLE.COM -k /etc/foreman-proxy/freeipa.keytab --enctypes=aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96,des3-cbc-sha1,arcfour-hmac
458 62 Stephen Benjamin
459 62 Stephen Benjamin
h1. FreeIPA realm proxy fails with "LoadError: no such file to load -- ffi_c"
460 62 Stephen Benjamin
461 62 Stephen Benjamin
This is due to a broken rubygem-ffi that ships from Puppetlabs: https://jira-web03-dev.puppetlabs.com/browse/CPR-26
462 62 Stephen Benjamin
463 63 Stephen Benjamin
We need the EPEL one (although with the exact same version).
464 62 Stephen Benjamin
465 62 Stephen Benjamin
Ensure you have the latest (rubygem-ffi-1.0.9-11.el6.x86_64):
466 62 Stephen Benjamin
467 62 Stephen Benjamin
    yum upgrade rubygem-ffi
468 62 Stephen Benjamin
469 62 Stephen Benjamin
Now downgrade, but disable the puppetlabs-dependecies repo so we grab the package from EPEL:
470 62 Stephen Benjamin
471 62 Stephen Benjamin
    yum downgrade --disablerepo=puppetlabs-dependencies rubygem-ffi-1.0.9-9.el6
472 62 Stephen Benjamin
473 62 Stephen Benjamin
Restart the proxy.
474 65 Lukas Zapletal
475 65 Lukas Zapletal
h1. How to log REST client calls
476 65 Lukas Zapletal
477 65 Lukas Zapletal
Sometimes it is useful to see content of calls that are being made by Foreman using rest-client library (compute resources, proxy communication etc). This is as easy as dropping the following file and restarting Foreman:
478 65 Lukas Zapletal
479 65 Lukas Zapletal
<pre>
480 65 Lukas Zapletal
$ cat >/usr/share/foreman/config/initializers/00_rest_client.rb <<'EOT'
481 65 Lukas Zapletal
require 'rest_client'
482 65 Lukas Zapletal
RestClient.log =
483 65 Lukas Zapletal
  Object.new.tap do |proxy|
484 65 Lukas Zapletal
    def proxy.<<(message)
485 65 Lukas Zapletal
      Rails.logger.info message
486 65 Lukas Zapletal
    end
487 65 Lukas Zapletal
  end
488 65 Lukas Zapletal
EOT
489 65 Lukas Zapletal
</pre>
490 66 Fergus Nelson
491 66 Fergus Nelson
h1. I'm getting connection timeouts when Foreman tries to connect to ec2 when running behind a http proxy
492 66 Fergus Nelson
493 66 Fergus Nelson
The error can look like this.
494 66 Fergus Nelson
<pre>
495 66 Fergus Nelson
<div id="backtrace" class="alert alert-block alert-danger base in fade hide">
496 66 Fergus Nelson
  <em>Excon::Errors::Timeout</em><br>
497 66 Fergus Nelson
  <strong>connect timeout reached</strong><br>
498 66 Fergus Nelson
  app/models/compute_resources/foreman/model/ec2.rb:59:in `regions'
499 66 Fergus Nelson
<br>app/models/compute_resources/foreman/model/ec2.rb:72:in `test_connection'
500 66 Fergus Nelson
<br>app/models/compute_resource.rb:120:in `new_vm'
501 66 Fergus Nelson
<br>app/views/hosts/_compute.html.erb:1:in `_app_views_hosts__compute_html_erb__2073528223_70084372690700'
502 66 Fergus Nelson
<br>app/controllers/hosts_controller.rb:139:in `compute_resource_selected'
503 66 Fergus Nelson
<br>app/models/taxonomy.rb:48:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:143:in `as_location'
504 66 Fergus Nelson
<br>app/models/taxonomy.rb:47:in `as_taxonomy'<br>app/models/concerns/foreman/thread_session.rb:108:in `as_org'<br>app/models/taxonomy.rb:46:in `as_taxonomy'
505 66 Fergus Nelson
<br>app/controllers/hosts_controller.rb:135:in `compute_resource_selected'<br>app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
506 66 Fergus Nelson
<br>lib/middleware/catch_json_parse_errors.rb:9:in `call'
507 66 Fergus Nelson
</div>
508 66 Fergus Nelson
<p><a href="/" data-id="aid_not_defined">Back</a></p>
509 66 Fergus Nelson
</pre>
510 66 Fergus Nelson
511 68 Dominic Cleal
The problem is that fog isn't picking up your environment's proxy settings, so the solution is for apache to pass them to it.
512 1 Ohad Levy
513 68 Dominic Cleal
On Debian/Ubuntu, edit /etc/apache2/envvars to include the following
514 68 Dominic Cleal
515 66 Fergus Nelson
export http_proxy=http://<host>:<port>
516 66 Fergus Nelson
export https_proxy=https://<host>:<port>
517 1 Ohad Levy
518 68 Dominic Cleal
On Red Hat OSes, edit /etc/sysconfig/httpd instead.
519 68 Dominic Cleal
520 1 Ohad Levy
Restart apache.
521 68 Dominic Cleal
522 68 Dominic Cleal
More information at https://www.phusionpassenger.com/documentation/Users%20guide%20Apache.html#_apache.
523 73 Dominic Cleal
524 73 Dominic Cleal
h1. How to take a thread dump of running Foreman processes
525 73 Dominic Cleal
526 73 Dominic Cleal
It's possible to trigger running Foreman applications to log thread dumps - if they're not responding, this can help determine which piece of code they're running.
527 73 Dominic Cleal
528 73 Dominic Cleal
If you're using Passenger (default setup with foreman-installer), run @pkill -ABRT -f RackApp@ to send an abort signal to Passenger.  This will be written to /var/log/httpd/error_log (EL/Fedora) or /var/log/apache2/error.log (Debian/Ubuntu).
529 73 Dominic Cleal
530 73 Dominic Cleal
If you're using Foreman standalone (WEBrick or other), @pkill -TTIN -f rails@ (or TTIN signal to whatever container you're using) will print the thread dump to stdout - you may have to run Foreman in the foreground to see this.
531 75 Stephen Benjamin
532 75 Stephen Benjamin
h1. "ERROR -- : Wrong size. Was 315, should be 196" when using Realm Proxy on EL7
533 75 Stephen Benjamin
534 75 Stephen Benjamin
A bug in Ruby 2.0 prevents the Realm proxy from working correctly: https://bugs.ruby-lang.org/issues/8182
535 75 Stephen Benjamin
536 75 Stephen Benjamin
Commenting these out on lines 505-506 in /usr/share/ruby/xmlrpc/client.rb is a temporary workaround:
537 75 Stephen Benjamin
538 75 Stephen Benjamin
      <pre>
539 75 Stephen Benjamin
      #elsif expected != "<unknown>" and expected.to_i != data.bytesize and resp["Transfer-Encoding"].nil?
540 75 Stephen Benjamin
      #  raise "Wrong size. Was #{data.bytesize}, should be #{expected}"
541 75 Stephen Benjamin
      </pre>
542 75 Stephen Benjamin
543 75 Stephen Benjamin
This is expected to be fixed in one of the next releases of EL7.
544 87 Dominic Cleal
545 87 Dominic Cleal
h1. What are the default credentials?
546 87 Dominic Cleal
547 87 Dominic Cleal
When using foreman-installer, the first admin account will be set up and credentials are printed at the end, e.g.
548 87 Dominic Cleal
549 87 Dominic Cleal
<pre>
550 87 Dominic Cleal
  * Foreman is running at https://theforeman.example.com
551 87 Dominic Cleal
      Initial credentials are admin / 3ekw5xtyXCoXxS29
552 87 Dominic Cleal
</pre>
553 87 Dominic Cleal
554 87 Dominic Cleal
Visit the web UI and log in with the username "admin" and password "3ekw5xtyXCoXxS29" (in this example).
555 87 Dominic Cleal
556 87 Dominic Cleal
You can customise these when doing the initial installation by using @foreman-installer --foreman-admin-password=example@.  Note that this only works during the first installation - it won't change the password if run again.
557 87 Dominic Cleal
558 87 Dominic Cleal
If you are installing from source, the random password will be printed when running @rake db:seed@.
559 87 Dominic Cleal
560 87 Dominic Cleal
h1. How do I reset the admin password?
561 87 Dominic Cleal
562 87 Dominic Cleal
Run @foreman-rake permissions:reset@ which will create a new random password for the admin account.
563 87 Dominic Cleal
564 87 Dominic Cleal
You can specify a particular password by adding @password=example@ to the end of the command, or change a different user with @username=exuser@.  Note that this command is only designed to reset the admin account, so will give the specified user admin privileges and will create the account if it's missing.
565 92 Lukas Zapletal
566 92 Lukas Zapletal
h1. SELinux denials
567 92 Lukas Zapletal
568 92 Lukas Zapletal
When encountering SELinux denials, it is recommended to re-run @foreman-selinux-enable@ followed by @foreman-selinux-relabel@ and then restart all services (httpd, foreman-proxy, puppet, ISC services). If it does not help, force full filesystem relabel with @touch /.autorelabel@ and rebooting the server.
569 92 Lukas Zapletal
570 92 Lukas Zapletal
Only after that, use @foreman-debug -u@ to upload log files and SELinux configuration and contact developers providing the download link (it's readable only to Foreman Core developers). If you prefer pastebin into the list, provide output of the following commands:
571 92 Lukas Zapletal
 
572 92 Lukas Zapletal
* rpm -q foreman foreman-selinux foreman-proxy-selinux
573 92 Lukas Zapletal
* ps auxZ
574 92 Lukas Zapletal
* semodule -l
575 92 Lukas Zapletal
* semanage boolean -l
576 92 Lukas Zapletal
* ausearch -m AVC -m USER_AVC -m SELINUX_ERR | head -n 100
577 94 Lukas Zapletal
578 94 Lukas Zapletal
h1. Template or "built" request returns Net::HTTPMethodNotAllowed
579 94 Lukas Zapletal
580 94 Lukas Zapletal
When host associated with the token or IP address is not in build mode, Foreman returns: Failed to retrieve built template for hostname - Net::HTTPMethodNotAllowed. This can happen when rendering a template, or at the end of the provisioning script when "built" call is sent via curl/wget.
581 95 Lukas Zapletal
582 95 Lukas Zapletal
h1. No PXELinux/PXEGrub/PXEGrub2 templates were found for this host
583 95 Lukas Zapletal
584 95 Lukas Zapletal
If you encounter "No PXELinux/PXEGrub/PXEGrub2 templates were found for this host, make sure you define at least one in your OS settings or change PXE loader" error make sure all associated templates do belong to the Organization and Location of the host that is being created or edited.
585 96 Lukas Zapletal
586 96 Lukas Zapletal
h1. Enable detailed SQL logger for orchestration messages
587 96 Lukas Zapletal
588 96 Lukas Zapletal
In @/etc/foreman/settings.yaml@ set this:
589 96 Lukas Zapletal
590 96 Lukas Zapletal
<pre>
591 96 Lukas Zapletal
:loggers:
592 96 Lukas Zapletal
  :level: debug
593 96 Lukas Zapletal
  :sql:
594 96 Lukas Zapletal
    :enabled: true
595 96 Lukas Zapletal
</pre>
596 96 Lukas Zapletal
597 96 Lukas Zapletal
This will ensure the SQL logger is enabled (it is turned off on production environmenty by default).
598 97 Joel Kåberg
599 97 Joel Kåberg
h1. Troubleshooting multi CA environment
600 97 Joel Kåberg
601 97 Joel Kåberg
_Note: If you're not running CentOS, adapt the following to your environment._
602 97 Joel Kåberg
When installing the Foreman in an multi CA environment it's usually better to store the CA certs within the host's CA trust. On CentOS when the command *update-ca-trust* is run, the file /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt is created. When installing the Foreman, point *--foreman-proxy-foreman-ssl-ca* *--foreman-proxy-ssl-ca* and optionally (if you're not using the Puppet CA) *--foreman-proxy-puppet-ssl* to the mentioned file. This will make the Foreman trust the CA certs you add in /etc/pki/ca-trust/source/anchors.