Project

General

Profile

Bug #17629

Puppet Upgrade from 3 - 4

Added by Sven Vogel over 2 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Documentation
Target version:
Difficulty:
trivial
Triaged:
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Hi,

i upgrade puppet with foreman-installer --upgrade-puppet from 3 to 4. CentOS 7.2

the following problems occur.

D, [2016-12-11T17:22:43.597917 #29422] DEBUG -- : Executing /usr/bin/sudo -S /opt/puppetlabs/bin/puppet cert --ssldir /etc/puppetlabs/puppet/ssl --list --all
W, [2016-12-11T17:22:43.623701 #29422] WARN -- : Failed to run puppetca:
E, [2016-12-11T17:22:43.624100 #29422] ERROR -- : Failed to list certificates: Execution of puppetca failed, check log files
D, [2016-12-11T17:22:43.624154 #29422] DEBUG -- : Failed to list certificates: Execution of puppetca failed, check log files
I, [2016-12-11T17:22:43.625078 #29422] INFO -- : 192.168.85.32 - - [11/Dec/2016:17:22:43 +0100] "GET /puppet/ca HTTP/1.1" 406 74 0.0284

my sudeors file looks like
visudo
  1. Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
  1. includedir /etc/sudoers.d

visudo -f /etc/sudoers.d/foreman-proxy
foreman-proxy ALL = (root) NOPASSWD : /opt/puppetlabs/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /opt/puppetlabs/bin/puppet kick *
Defaults:foreman-proxy !requiretty

these dont work so i moved for testing the "visudo -f /etc/sudoers.d/foreman-proxy" directly into the /etc/sudoers file. i think there is a problem with sudoers file and order... thats a other problem which needs to be checked.

after that i tried su - foreman-proxy and run line command again.

/usr/bin/sudo -S /opt/puppetlabs/bin/puppet cert --ssldir /etc/puppetlabs/puppet/ssl --list --all

now i get

+ "katello01.example.local" (SHA256) 66:A2:85:39:B7:1A:62:8C:92:44:6E:03:F4:45:FA:B8:95:B5:59:F4:6B:5F:71:26:C7:4D:83:52:C4:DD:87:E8 (alt names: "DNS:katello01.example.local", "DNS:puppet", "DNS:puppet.example.local")
+ "test01.example.local" (SHA256) 7E:CC:4A:68:18:B8:85:E8:4E:EC:97:DC:47:0F:4D:7C:BE:77:9C:31:CB:24:0C:18:45:F9:CB:DD:F9:23:07:A9
+ "test02.example.local" (SHA256) EA:F6:B4:EF:23:95:CF:3A:BE:DE:75:82:BA:6C:7E:5D:43:C8:56:03:5F:79:D0:48:7E:E8:04:7D:ED:C7:53:C3
+ "test03.example.local" (SHA256) BE:16:E5:FE:1B:EC:30:02:68:9C:94:9D:6E:17:AD:FE:6F:64:78:21:4B:D8:14:1B:AB:BC:38:04:D1:46:BD:AB

BUT

error seems the same. picture.

i checked the smart proxy https://192.168.85.32l:9090/puppet/ca and get the error "could not read client cert from environment"

maybe there are a correlation.

when i restart the smart proxy in debug mode thats my startup parameter.

[root@katello01 code]# D, [2016-12-11T17:35:42.080440 #30122] DEBUG -- : 'pulp' settings: 'enabled': https, 'mongodb_dir': /var/lib/mongodb (default), 'pulp_content_dir': /var/lib/pulp/content (default), 'pulp_dir': /var/lib/pulp (default), 'pulp_url': https://katello01.example.local/pulp, 'puppet_content_dir': /etc/puppetlabs/code/environments
D, [2016-12-11T17:35:42.085155 #30122] DEBUG -- : 'dynflow' settings: 'core_url': https://katello01.example.local:8008, 'database': /var/lib/foreman-proxy/dynflow/dynflow.sqlite, 'enabled': https
D, [2016-12-11T17:35:42.089031 #30122] DEBUG -- : 'ssh' settings: 'enabled': https, 'local_working_dir': /var/tmp (default), 'remote_working_dir': /var/tmp (default), 'ssh_identity_key_file': /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy, 'ssh_user': root (default)
D, [2016-12-11T17:35:42.103614 #30122] DEBUG -- : 'dns' settings: 'dns_ttl': 86400 (default), 'enabled': https, 'use_provider': dns_nsupdate (default)
D, [2016-12-11T17:35:42.107660 #30122] DEBUG -- : 'tftp' settings: 'enabled': https, 'tftp_servername': 192.168.85.32, 'tftproot': /var/lib/tftpboot (default)
D, [2016-12-11T17:35:42.114433 #30122] DEBUG -- : 'dhcp' settings: 'enabled': https, 'server': 127.0.0.1 (default), 'subnets': [] (default), 'use_provider': dhcp_isc (default)
D, [2016-12-11T17:35:42.118180 #30122] DEBUG -- : 'puppetca' settings: 'enabled': https, 'puppetdir': /etc/puppetlabs/puppet, 'ssldir': /etc/puppetlabs/puppet/ssl
D, [2016-12-11T17:35:42.124272 #30122] DEBUG -- : 'puppet' settings: 'enabled': https, 'puppet_version': 4.8.1, 'use_provider': [:puppet_proxy_puppet_api]
D, [2016-12-11T17:35:42.125871 #30122] DEBUG -- : Providers ['dns_nsupdate'] are going to be configured for 'dns'
D, [2016-12-11T17:35:42.126054 #30122] DEBUG -- : Providers ['dhcp_isc'] are going to be configured for 'dhcp'
D, [2016-12-11T17:35:42.126168 #30122] DEBUG -- : Providers ['puppet_proxy_puppet_api'] are going to be configured for 'puppet'
D, [2016-12-11T17:35:42.130651 #30122] DEBUG -- : 'dns_nsupdate' settings: 'dns_key': /etc/rndc.key, 'dns_server': 127.0.0.1, 'dns_ttl': 86400, 'use_provider': dns_nsupdate
D, [2016-12-11T17:35:42.166876 #30122] DEBUG -- : 'dhcp_isc' settings: 'config': /etc/dhcp/dhcpd.conf (default), 'leases': /var/lib/dhcpd/dhcpd.leases (default), 'leases_file_observer': inotify_leases_file_observer, 'omapi_port': 7911, 'server': 127.0.0.1, 'subnets': [], 'use_provider': dhcp_isc
D, [2016-12-11T17:35:42.175322 #30122] DEBUG -- : 'puppet_proxy_puppet_api' settings: 'classes_retriever': apiv3, 'environments_retriever': apiv3, 'puppet_ssl_ca': /etc/puppetlabs/puppet/ssl/certs/ca.pem, 'puppet_ssl_cert': /etc/puppetlabs/puppet/ssl/certs/katello01.example.local.pem, 'puppet_ssl_key': /etc/puppetlabs/puppet/ssl/private_keys/katello01.example.local.pem, 'puppet_url': https://katello01.example.local:8140, 'puppet_version': 4.8.1, 'use_provider': [:puppet_proxy_puppet_api]
I, [2016-12-11T17:35:42.176736 #30122] INFO -- : Successfully initialized 'pulp'
I, [2016-12-11T17:35:42.178597 #30122] INFO -- : Successfully initialized 'dynflow'
I, [2016-12-11T17:35:42.181500 #30122] INFO -- : Successfully initialized 'ssh'
I, [2016-12-11T17:35:42.181765 #30122] INFO -- : Successfully initialized 'foreman_proxy'
I, [2016-12-11T17:35:42.181913 #30122] INFO -- : Successfully initialized 'dns_nsupdate'
I, [2016-12-11T17:35:42.182019 #30122] INFO -- : Successfully initialized 'dns'
I, [2016-12-11T17:35:42.182109 #30122] INFO -- : Successfully initialized 'tftp'
D, [2016-12-11T17:35:42.206426 #30122] DEBUG -- : Added a subnet: 192.168.85.0
D, [2016-12-11T17:35:42.208209 #30122] DEBUG -- : Added a reservation: 192.168.85.14:00:19:99:bc:04:e6:kvm02.example.local
D, [2016-12-11T17:35:42.208378 #30122] DEBUG -- : Added a reservation: 192.168.85.13:00:19:99:cb:c2:e2:kvm01.example.local
D, [2016-12-11T17:35:42.208503 #30122] DEBUG -- : Added a reservation: 192.168.85.15:00:19:99:c5:0b:83:kvm03.example.local
I, [2016-12-11T17:35:42.208764 #30122] INFO -- : Successfully initialized 'dhcp_isc'
I, [2016-12-11T17:35:42.208897 #30122] INFO -- : Successfully initialized 'dhcp'
I, [2016-12-11T17:35:42.209459 #30122] INFO -- : Successfully initialized 'puppetca'
I, [2016-12-11T17:35:42.209636 #30122] INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2016-12-11T17:35:42.209752 #30122] INFO -- : Successfully initialized 'puppet'
I, [2016-12-11T17:35:42.243380 #30122] INFO -- : WEBrick 1.3.1
I, [2016-12-11T17:35:42.243661 #30122] INFO -- : ruby 2.0.0 (2014-11-13) [x86_64-linux]
D, [2016-12-11T17:35:42.244102 #30122] DEBUG -- : TCPServer.new(0.0.0.0, 9090)
D, [2016-12-11T17:35:42.244301 #30122] DEBUG -- : TCPServer.new(::, 9090)
W, [2016-12-11T17:35:42.244482 #30122] WARN -- : TCPServer Error: Address already in use - bind(2)
I, [2016-12-11T17:35:42.245666 #30122] INFO -- :

is there anybody who can help?

thanks

Sven

error_puppet_ca.PNG View error_puppet_ca.PNG 13.7 KB Sven Vogel, 12/11/2016 11:32 AM
Error puppet ca

History

#1 Updated by Edgars Mazurs over 2 years ago

Had the same issue. Solved it by setting symlink:

sudo ln -s /opt/puppetlabs/bin/puppet /usr/bin/puppet

And restart puppet and puppet server. Also remember to revert /etc/sudoers.d/foreman-proxy to default settings:

foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick *
Defaults:foreman-proxy !requiretty

#2 Updated by Sven Vogel over 2 years ago

thanks for answer. i got it working with. all paths are reset and now they are work.

foreman-installer -v\
--reset-foreman-client-ssl-ca \
--reset-foreman-client-ssl-cert \
--reset-foreman-client-ssl-key \
--reset-foreman-puppet-home \
--reset-foreman-puppet-ssldir \
--reset-foreman-server-ssl-ca \
--reset-foreman-server-ssl-cert \
--reset-foreman-server-ssl-chain \
--reset-foreman-server-ssl-crl \
--reset-foreman-server-ssl-key \
--reset-foreman-websockets-ssl-cert \
--reset-foreman-websockets-ssl-key \
--reset-foreman-proxy-puppet-ssl-ca \
--reset-foreman-proxy-puppet-ssl-cert \
--reset-foreman-proxy-puppet-ssl-key \
--reset-foreman-proxy-puppetca-cmd \
--reset-foreman-proxy-puppetdir \
--reset-foreman-proxy-ssl-ca \
--reset-foreman-proxy-ssl-cert \
--reset-foreman-proxy-ssl-key \
--reset-foreman-proxy-ssldir \
--reset-foreman-puppet-home \

my other problem is now that when i run a puppet agent --test and /etc/puppetlabs/puppet/node.rb myhost i get the following error.

Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca

i dont know how to fix that.

#3 Updated by Sven Vogel over 2 years ago

We found the solution.

We debug the node.rb file and checked the certificates which was send to the katello/foreman server.

we have found out that node.rb use the following paths and ca/cert and key file

/etc/puppetlabs/puppet/ssl/client_cert.pem
/etc/puppetlabs/puppet/ssl/client_key.pem
/etc/puppetlabs/puppet/ssl/ssl_ca.pem

after that we checked the /etc/httpd/conf.d/05-foreman-ssl.conf file.

the foreman-installer seems not correct set the paths to the files...

SSLCertificateFile      "/etc/puppetlabs/puppet/ssl/certs/test.example.com.pem" 
SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/test.example.com.pem"
SSLCertificateChainFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
SSLCACertificateFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"

node.rb called files are differen to the 05-foreman-ssl.conf. we change all files into 05-foreman-ssl.conf to

  1. SSL directives
    SSLEngine on
    SSLCertificateFile "/etc/puppetlabs/puppet/ssl/client_cert.pem"
    SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/client_key.pem"
    SSLCertificateChainFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
    SSLCACertificateFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
    SSLCARevocationFile "/etc/puppetlabs/puppet/ssl/crl.pem"
    SSLVerifyClient optional
    SSLVerifyDepth 3
    SSLOptions +StdEnvVars +ExportCertData

after restart tls error was gone.

we checked it on a node with puppet agent --test and got the next error.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node kvm02.oscloud.local: Exception while executing '/etc/puppetlabs/puppet/node.rb': Cannot run program "/etc/puppetlabs/puppet/node.rb" (in directory "."): error=13, Permission denied

we found that the node.rb dont has puppet right correctly. maybe also a error in upgrade. we changes /etc/puppetlabs/puppet and node.rb to puppet user and group

total 48
drwxrwx--x 8 puppet puppet 4096 Dec 14 12:43 ssl
rw-r--r- 1 root root 2687 Dec 14 12:43 puppet.conf
r-xr-x-- 1 root root 11725 Dec 15 14:39 node.rb.changed
r-xr-x-- 1 puppet puppet 11345 Dec 15 13:34 node.rb
rw-r--r- 1 root root 371 Dec 6 01:17 hiera.yaml
rw-r---- 1 root puppet 365 Dec 14 12:43 foreman.yaml
rw-rw-r- 1 puppet puppet 0 Dec 14 12:43 autosign.conf
rw-r--r- 1 root root 4505 Dec 14 12:43 auth.conf

i am open for suggestions if the other files also need puppet rights?! is it a bug and resolvable or foreman-installer problem?

thanks

Sven

#4 Updated by Justin Sherrill about 2 years ago

  • Legacy Backlogs Release (now unused) changed from 162 to 188

#5 Updated by Stephen Benjamin about 2 years ago

  • Status changed from New to Closed
  • Legacy Backlogs Release (now unused) changed from 188 to 166

You might have found a bug somewhere in here, but it's kind of hard to follow along since you did so many different things, can you clarify what the original problem was? What was the output of the sudo command if you ran it directly as foreman-proxy user? I don't see any sudo-related errors in your logs specifically, the proxy just said it failed to list certificates. The sudoers file looks correct.

Later on, the problem was made worse by resetting all those parameters. Those look like they're from the foreman puppet upgrade wiki, but you can't use those instructions with Katello, we have an independent CA. Resetting everything to the puppet defaults will set a bunch of the paths wrong in that case, which cause the problem you had with node.rb. Katello only needs a subset of them reset, and we do it for you in the `--upgrade-puppet` hook (https://github.com/Katello/katello-installer/blob/master/hooks/pre/31-upgrade-puppet.rb#L56-L64).

We've had a bunch of successful puppet 4 upgrades reported - if things are working for you now, I'll close this, unless you can provide some more info to help investigate.

Thanks!

#7 Updated by Jorick Astrego over 1 year ago

  • Category set to Documentation
  • Legacy Backlogs Release (now unused) changed from 166 to 114
  • Difficulty set to trivial

We have the same problem, upgraded Katello, ran katello puppet upgrade script.

I saw the deploy fail and in the proxy log that sudo could not be executed:

W, [2017-08-24T10:06:54.391688 ] WARN -- : Failed to run puppetca: [sudo] password for foreman-proxy:
sudo: pam_authenticate: Conversation error

E, [2017-08-24T10:06:54.392710 ] ERROR -- : Failed to remove certificate(s) for xxx.xxx.xxx.xxx: Execution of puppetca failed, check log files
E, [2017-08-24T10:06:54.392710 ] ERROR -- : Failed to remove certificate(s) for xxx.xxx.xxx.xxx: Execution of puppetca failed, check log files
W, [2017-08-24T10:10:05.565436 ] WARN -- : Failed to initialize puppet class cache, deferring initialization. Is puppetserver running?

Sudoers file looks like this:

cat /etc/sudoers.d/foreman-proxy
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
Defaults:foreman-proxy !requiretty

So I checked my upgrade process again and it looks like I mistakenly did:

foreman-installer --upgrade-puppet

Instead of:

foreman-installer --scenario katello --upgrade-puppet

So I checked the documentation and it clearly states the wrong command:

https://theforeman.org/plugins/katello/3.4/upgrade/puppet.html

In-place migration

If you plan on upgrading an existing Katello server or Smart Proxy to Puppet 4, the the process is straightforward.

Take backup or VM snapshot of server
run katello-service stop to stop all services
run foreman-installer --upgrade-puppet. This will perform the upgrade.

Running "foreman-installer --scenario katello --upgrade-puppet" fixes things.

Also available in: Atom PDF