Project

General

Profile

Actions
Copy link

Bug #19986

closed

puppetserver fails to restart after installation

Added by Evgeni Golov almost 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
Difficulty:
Triaged:
Bugzilla link:
1470119
Pull request:
Fixed in Releases:
Found in Releases:
Foreman - 1.15.0
Red Hat JIRA:

Description

Ohai,

after installing a fresh 1.15 (using forklift) everything is working fine:

[root@centos7-foreman-1-15 ~]# systemctl status puppetserver.service 
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-06-13 06:13:28 UTC; 1min 19s ago
 Main PID: 4996 (java)
   CGroup: /system.slice/puppetserver.service
           └─4996 /usr/bin/java -Xms2G -Xmx2G -XX:MaxPermSize=256m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /...

Jun 13 06:12:54 centos7-foreman-1-15.example.com systemd[1]: Starting puppetserver Service...
Jun 13 06:12:54 centos7-foreman-1-15.example.com puppetserver[4987]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jun 13 06:13:28 centos7-foreman-1-15.example.com systemd[1]: Started puppetserver Service.

However, when I try to restart the puppetserver, it errors out:

[root@centos7-foreman-1-15 ~]# systemctl restart puppetserver
Job for puppetserver.service failed because the control process exited with error code. See "systemctl status puppetserver.service" and "journalctl -xe" for details.

[root@centos7-foreman-1-15 ~]# journalctl -xe
…
-- Unit puppetserver.service has begun starting up.
Jun 13 06:15:28 centos7-foreman-1-15.example.com puppetserver[5697]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: Exception in thread "main" java.io.FileNotFoundException: /etc/puppetlabs/puppet/ssl/crl.pem (Permission denied)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.io.FileOutputStream.open0(Native Method)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.java.io$fn__9570.invokeStatic(io.clj:355)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.java.io$fn__9570.invoke(io.clj:354)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.lang.MultiFn.invoke(MultiFn.java:238)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.java.io$copy.invokeStatic(io.clj:406)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.java.io$copy.doInvoke(io.clj:391)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.lang.RestFn.invoke(RestFn.java:425)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at me.raynes.fs$copy.invokeStatic(fs.clj:293)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at me.raynes.fs$copy.invoke(fs.clj:289)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.puppetserver.certificate_authority$eval16660$retrieve_ca_crl_BANG___16665$fn__16666.invoke(certificate_authority.clj:752)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.puppetserver.certificate_authority$eval16660$retrieve_ca_crl_BANG___16665.invoke(certificate_authority.clj:744)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.services.ca.certificate_authority_service$reify__24897$service_fnk__5222__auto___positional$reify__24908.retrieve_ca_crl_BANG_(certificate_authority_service.clj:52)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.services.protocols.ca$eval24835$fn__24836$G__24825__24839.invoke(ca.clj:3)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.services.protocols.ca$eval24835$fn__24836$G__24824__24843.invoke(ca.clj:3)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core$partial$fn__4759.invoke(core.clj:2515)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.services.master.master_service$reify__33186$service_fnk__5222__auto___positional$reify__33207.init(master_service.clj:52)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.services$eval5024$fn__5025$G__5012__5028.invoke(services.clj:8)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.services$eval5024$fn__5025$G__5011__5032.invoke(services.clj:8)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13792$run_lifecycle_fn_BANG___13799$fn__13800.invoke(internal.clj:204)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13792$run_lifecycle_fn_BANG___13799.invoke(internal.clj:187)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13821$run_lifecycle_fns__13826$fn__13827.invoke(internal.clj:238)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13821$run_lifecycle_fns__13826.invoke(internal.clj:215)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval14291$build_app_STAR___14300$fn$reify__14310.init(internal.clj:588)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval14337$boot_services_for_app_STAR__STAR___14344$fn__14345$fn__14347.invoke(internal.clj:616)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval14337$boot_services_for_app_STAR__STAR___14344$fn__14345.invoke(internal.clj:615)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval14337$boot_services_for_app_STAR__STAR___14344.invoke(internal.clj:609)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core$partial$fn__4761.invoke(core.clj:2521)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13860$initialize_lifecycle_worker__13871$fn__13872$fn__13958$state_machine__11832__auto____13959$fn__13961.invoke(internal.clj:255)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at puppetlabs.trapperkeeper.internal$eval13860$initialize_lifecycle_worker__13871$fn__13872$fn__13958$state_machine__11832__auto____13959.invoke(internal.clj:255)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:1012)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:1011)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:1016)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:1014)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async$ioc_alts_BANG_$fn__12000.invoke(async.clj:383)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async$do_alts$fn__11946$fn__11949.invoke(async.clj:252)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.core.async.impl.channels.ManyToManyChannel$fn__6756$fn__6757.invoke(channels.clj:95)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at clojure.lang.AFn.run(AFn.java:22)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: at java.lang.Thread.run(Thread.java:748)
Jun 13 06:15:57 centos7-foreman-1-15.example.com puppetserver[5697]: Background process 5704 exited before start had completed
Jun 13 06:15:57 centos7-foreman-1-15.example.com systemd[1]: puppetserver.service: control process exited, code=exited status=1
Jun 13 06:15:57 centos7-foreman-1-15.example.com systemd[1]: Failed to start puppetserver Service.
-- Subject: Unit puppetserver.service has failed
…

The exception is slightly misleading, as the file is readable by Puppet just fine:

[root@centos7-foreman-1-15 ~]# ls -alh /etc/puppetlabs/puppet/ssl/
total 4.0K
drwxrwx--x. 8 root   puppet 126 Jun 13 06:12 .
drwxr-xr-x. 3 root   root   127 Jun 13 06:12 ..
drwxr-xr-x. 5 puppet puppet 158 Jun 13 06:12 ca
drwxr-xr-x. 2 root   puppet   6 Jun 13 06:12 certificate_requests
drwxr-xr-x. 2 root   puppet  64 Jun 13 06:12 certs
-rw-r--r--. 1 root   puppet 987 Jun 13 06:13 crl.pem
drwxr-x---. 2 root   puppet   6 Jun 13 06:05 private
drwxr-x---. 2 root   puppet  50 Jun 13 06:12 private_keys
drwxr-xr-x. 2 root   puppet  50 Jun 13 06:12 public_keys

But what it actually wants, is being able to write that file, and thus changing the owner to "puppet" fixes the issue:

[root@centos7-foreman-1-15 ~]# chown puppet /etc/puppetlabs/puppet/ssl/crl.pem 
[root@centos7-foreman-1-15 ~]# systemctl restart puppetserver
[root@centos7-foreman-1-15 ~]# systemctl status puppetserver.service
● puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-06-13 06:22:01 UTC; 50s ago
  Process: 5679 ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop (code=exited, status=0/SUCCESS)
  Process: 7760 ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start (code=exited, status=0/SUCCESS)
 Main PID: 7767 (java)
   CGroup: /system.slice/puppetserver.service
           └─7767 /usr/bin/java -Xms2G -Xmx2G -XX:MaxPermSize=256m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /...

Jun 13 06:21:35 centos7-foreman-1-15.example.com systemd[1]: Starting puppetserver Service...
Jun 13 06:21:35 centos7-foreman-1-15.example.com puppetserver[7760]: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jun 13 06:22:01 centos7-foreman-1-15.example.com systemd[1]: Started puppetserver Service.

Note, setting the perms to 664 does not help, as Puppet itself tries to enforce the 644 perms.

Actions
Copy link
 #1

Updated by Evgeni Golov almost 7 years ago

The correct permissions seem to be

$ sudo ls -lah /etc/puppetlabs/puppet/ssl/
total 4.0K
drwxrwx--x. 8 puppet puppet 126 Jun 14 08:45 .
drwxr-xr-x. 3 root   root   127 Jun 14 08:45 ..
drwxr-xr-x. 5 puppet puppet 158 Jun 14 08:45 ca
drwxr-xr-x. 2 puppet puppet   6 Jun 14 08:45 certificate_requests
drwxr-xr-x. 2 puppet puppet  72 Jun 14 08:45 certs
-rw-r--r--. 1 puppet puppet 999 Jun 14 08:46 crl.pem
drwxr-x---. 2 puppet puppet   6 Jun 14 08:36 private
drwxr-x---. 2 puppet puppet  58 Jun 14 08:45 private_keys
drwxr-xr-x. 2 puppet puppet  58 Jun 14 08:45 public_keys

Wonder what changes them, though.

Actions
Copy link
 #2

Updated by Evgeni Golov almost 7 years ago 

# grep "etc/puppetlabs/puppet/.*root" /var/log/foreman-installer/katello.log 
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'certdir': 'File[/etc/puppetlabs/puppet/ssl/certs]{:path=>"/etc/puppetlabs/puppet/ssl/certs", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'ssldir': 'File[/etc/puppetlabs/puppet/ssl]{:path=>"/etc/puppetlabs/puppet/ssl", :mode=>"771", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'publickeydir': 'File[/etc/puppetlabs/puppet/ssl/public_keys]{:path=>"/etc/puppetlabs/puppet/ssl/public_keys", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'requestdir': 'File[/etc/puppetlabs/puppet/ssl/certificate_requests]{:path=>"/etc/puppetlabs/puppet/ssl/certificate_requests", :mode=>"755", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'privatekeydir': 'File[/etc/puppetlabs/puppet/ssl/private_keys]{:path=>"/etc/puppetlabs/puppet/ssl/private_keys", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'privatedir': 'File[/etc/puppetlabs/puppet/ssl/private]{:path=>"/etc/puppetlabs/puppet/ssl/private", :mode=>"750", :owner=>"root", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'hostcert': 'File[/etc/puppetlabs/puppet/ssl/certs/centos7-bats-ci.example.com.pem]{:path=>"/etc/puppetlabs/puppet/ssl/certs/centos7-bats-ci.example.com.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'hostprivkey': 'File[/etc/puppetlabs/puppet/ssl/private_keys/centos7-bats-ci.example.com.pem]{:path=>"/etc/puppetlabs/puppet/ssl/private_keys/centos7-bats-ci.example.com.pem", :mode=>"640", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'hostpubkey': 'File[/etc/puppetlabs/puppet/ssl/public_keys/centos7-bats-ci.example.com.pem]{:path=>"/etc/puppetlabs/puppet/ssl/public_keys/centos7-bats-ci.example.com.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'localcacert': 'File[/etc/puppetlabs/puppet/ssl/certs/ca.pem]{:path=>"/etc/puppetlabs/puppet/ssl/certs/ca.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  Using settings: adding file resource 'hostcrl': 'File[/etc/puppetlabs/puppet/ssl/crl.pem]{:path=>"/etc/puppetlabs/puppet/ssl/crl.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/certs]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/public_keys]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/certificate_requests]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/private_keys]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/private]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/certs/centos7-bats-ci.example.com.pem]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/private_keys/centos7-bats-ci.example.com.pem]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/public_keys/centos7-bats-ci.example.com.pem]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/certs/ca.pem]/owner: owner changed 'puppet' to 'root'
[DEBUG 2017-06-14 09:46:50 main]  /File[/etc/puppetlabs/puppet/ssl/crl.pem]/owner: owner changed 'puppet' to 'root'
Actions
Copy link
 #3

Updated by Evgeni Golov over 6 years ago

  • Bugzilla link set to 1470119
Actions
Copy link
 #4

Updated by Eric Helms over 6 years ago

I noticed that if you re-run it, it puts the permissions back to puppet/puppet and everything works fine. It appears to do this on initial runs only.

Actions
Copy link
 #5

Updated by Eric Helms over 6 years ago

A little more context to #2, that output happens after this output indicating its after the catalog run from the puppet apply:

[DEBUG 2017-08-11 12:23:18 main]  Evicting cache entry for environment 'production'
[DEBUG 2017-08-11 12:23:18 main]  Caching environment 'production' (ttl = 0 sec)
[DEBUG 2017-08-11 12:23:19 main]  Finishing transaction 39447760
[DEBUG 2017-08-11 12:23:19 main]  Storing state
[DEBUG 2017-08-11 12:23:19 main]  Stored state in 0.18 seconds
[ WARN 2017-08-11 12:23:19 main]  Applied catalog in 254.14 seconds
[DEBUG 2017-08-11 12:23:19 main]  Applying settings catalog for sections main, reporting, metrics
Actions
Copy link
 #6

Updated by Ewoud Kohl van Wijngaarden over 6 years ago

It's also good to know that I can reproduce this on a plain Foreman + Puppet server.

On the first run:

[DEBUG 2017-08-11 23:28:38 main]  Using settings: adding file resource 'hostcrl': 'File[/etc/puppetlabs/puppet/ssl/crl.pem]{:path=>"/etc/puppetlabs/puppet/ssl/crl.pem", :mode=>"644", :owner=>"root", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

On the second run:

[DEBUG 2017-08-11 23:56:31 main]  Using settings: adding file resource 'hostcrl': 'File[/etc/puppetlabs/puppet/ssl/crl.pem]{:path=>"/etc/puppetlabs/puppet/ssl/crl.pem", :mode=>"644", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}'

That leads me to think that puppet automatically manages files that are used for configuration. While initializing the first run the puppet user does not exist, that only happens after the puppetserver package is installed. I think explicitly managing this file resource is the way to go.

Actions
Copy link
 #7

Updated by Ewoud Kohl van Wijngaarden over 6 years ago

  • Project changed from Foreman to Installer
Actions
Copy link
 #8

Updated by Ewoud Kohl van Wijngaarden over 6 years ago

  • Status changed from New to Resolved

It appears it was fixed in puppetserver 2.8.0 with the automatic CRL refresher. See https://docs.puppet.com/puppetserver/2.8/release_notes.html#new-feature-automatic-crl-refresh-on-certificate-revocation

Actions
Copy link

Also available in: Atom PDF