Bug #17629
closed
Puppet Upgrade from 3 - 4
Description
Hi,
i upgrade puppet with foreman-installer --upgrade-puppet from 3 to 4. CentOS 7.2
the following problems occur.
D, [2016-12-11T17:22:43.597917 #29422] DEBUG -- : Executing /usr/bin/sudo -S /opt/puppetlabs/bin/puppet cert --ssldir /etc/puppetlabs/puppet/ssl --list --all
W, [2016-12-11T17:22:43.623701 #29422] WARN -- : Failed to run puppetca:
E, [2016-12-11T17:22:43.624100 #29422] ERROR -- : Failed to list certificates: Execution of puppetca failed, check log files
D, [2016-12-11T17:22:43.624154 #29422] DEBUG -- : Failed to list certificates: Execution of puppetca failed, check log files
I, [2016-12-11T17:22:43.625078 #29422] INFO -- : 192.168.85.32 - - [11/Dec/2016:17:22:43 +0100] "GET /puppet/ca HTTP/1.1" 406 74 0.0284
visudo
- Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
- includedir /etc/sudoers.d
visudo -f /etc/sudoers.d/foreman-proxy
foreman-proxy ALL = (root) NOPASSWD : /opt/puppetlabs/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /opt/puppetlabs/bin/puppet kick *
Defaults:foreman-proxy !requiretty
these dont work so i moved for testing the "visudo -f /etc/sudoers.d/foreman-proxy" directly into the /etc/sudoers file. i think there is a problem with sudoers file and order... thats a other problem which needs to be checked.
after that i tried su - foreman-proxy and run line command again.
/usr/bin/sudo -S /opt/puppetlabs/bin/puppet cert --ssldir /etc/puppetlabs/puppet/ssl --list --all
now i get
+ "katello01.example.local" (SHA256) 66:A2:85:39:B7:1A:62:8C:92:44:6E:03:F4:45:FA:B8:95:B5:59:F4:6B:5F:71:26:C7:4D:83:52:C4:DD:87:E8 (alt names: "DNS:katello01.example.local", "DNS:puppet", "DNS:puppet.example.local")
+ "test01.example.local" (SHA256) 7E:CC:4A:68:18:B8:85:E8:4E:EC:97:DC:47:0F:4D:7C:BE:77:9C:31:CB:24:0C:18:45:F9:CB:DD:F9:23:07:A9
+ "test02.example.local" (SHA256) EA:F6:B4:EF:23:95:CF:3A:BE:DE:75:82:BA:6C:7E:5D:43:C8:56:03:5F:79:D0:48:7E:E8:04:7D:ED:C7:53:C3
+ "test03.example.local" (SHA256) BE:16:E5:FE:1B:EC:30:02:68:9C:94:9D:6E:17:AD:FE:6F:64:78:21:4B:D8:14:1B:AB:BC:38:04:D1:46:BD:AB
BUT
error seems the same. picture.
i checked the smart proxy https://192.168.85.32l:9090/puppet/ca and get the error "could not read client cert from environment"
maybe there are a correlation.
when i restart the smart proxy in debug mode thats my startup parameter.
[root@katello01 code]# D, [2016-12-11T17:35:42.080440 #30122] DEBUG -- : 'pulp' settings: 'enabled': https, 'mongodb_dir': /var/lib/mongodb (default), 'pulp_content_dir': /var/lib/pulp/content (default), 'pulp_dir': /var/lib/pulp (default), 'pulp_url': https://katello01.example.local/pulp, 'puppet_content_dir': /etc/puppetlabs/code/environments
D, [2016-12-11T17:35:42.085155 #30122] DEBUG -- : 'dynflow' settings: 'core_url': https://katello01.example.local:8008, 'database': /var/lib/foreman-proxy/dynflow/dynflow.sqlite, 'enabled': https
D, [2016-12-11T17:35:42.089031 #30122] DEBUG -- : 'ssh' settings: 'enabled': https, 'local_working_dir': /var/tmp (default), 'remote_working_dir': /var/tmp (default), 'ssh_identity_key_file': /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy, 'ssh_user': root (default)
D, [2016-12-11T17:35:42.103614 #30122] DEBUG -- : 'dns' settings: 'dns_ttl': 86400 (default), 'enabled': https, 'use_provider': dns_nsupdate (default)
D, [2016-12-11T17:35:42.107660 #30122] DEBUG -- : 'tftp' settings: 'enabled': https, 'tftp_servername': 192.168.85.32, 'tftproot': /var/lib/tftpboot (default)
D, [2016-12-11T17:35:42.114433 #30122] DEBUG -- : 'dhcp' settings: 'enabled': https, 'server': 127.0.0.1 (default), 'subnets': [] (default), 'use_provider': dhcp_isc (default)
D, [2016-12-11T17:35:42.118180 #30122] DEBUG -- : 'puppetca' settings: 'enabled': https, 'puppetdir': /etc/puppetlabs/puppet, 'ssldir': /etc/puppetlabs/puppet/ssl
D, [2016-12-11T17:35:42.124272 #30122] DEBUG -- : 'puppet' settings: 'enabled': https, 'puppet_version': 4.8.1, 'use_provider': [:puppet_proxy_puppet_api]
D, [2016-12-11T17:35:42.125871 #30122] DEBUG -- : Providers ['dns_nsupdate'] are going to be configured for 'dns'
D, [2016-12-11T17:35:42.126054 #30122] DEBUG -- : Providers ['dhcp_isc'] are going to be configured for 'dhcp'
D, [2016-12-11T17:35:42.126168 #30122] DEBUG -- : Providers ['puppet_proxy_puppet_api'] are going to be configured for 'puppet'
D, [2016-12-11T17:35:42.130651 #30122] DEBUG -- : 'dns_nsupdate' settings: 'dns_key': /etc/rndc.key, 'dns_server': 127.0.0.1, 'dns_ttl': 86400, 'use_provider': dns_nsupdate
D, [2016-12-11T17:35:42.166876 #30122] DEBUG -- : 'dhcp_isc' settings: 'config': /etc/dhcp/dhcpd.conf (default), 'leases': /var/lib/dhcpd/dhcpd.leases (default), 'leases_file_observer': inotify_leases_file_observer, 'omapi_port': 7911, 'server': 127.0.0.1, 'subnets': [], 'use_provider': dhcp_isc
D, [2016-12-11T17:35:42.175322 #30122] DEBUG -- : 'puppet_proxy_puppet_api' settings: 'classes_retriever': apiv3, 'environments_retriever': apiv3, 'puppet_ssl_ca': /etc/puppetlabs/puppet/ssl/certs/ca.pem, 'puppet_ssl_cert': /etc/puppetlabs/puppet/ssl/certs/katello01.example.local.pem, 'puppet_ssl_key': /etc/puppetlabs/puppet/ssl/private_keys/katello01.example.local.pem, 'puppet_url': https://katello01.example.local:8140, 'puppet_version': 4.8.1, 'use_provider': [:puppet_proxy_puppet_api]
I, [2016-12-11T17:35:42.176736 #30122] INFO -- : Successfully initialized 'pulp'
I, [2016-12-11T17:35:42.178597 #30122] INFO -- : Successfully initialized 'dynflow'
I, [2016-12-11T17:35:42.181500 #30122] INFO -- : Successfully initialized 'ssh'
I, [2016-12-11T17:35:42.181765 #30122] INFO -- : Successfully initialized 'foreman_proxy'
I, [2016-12-11T17:35:42.181913 #30122] INFO -- : Successfully initialized 'dns_nsupdate'
I, [2016-12-11T17:35:42.182019 #30122] INFO -- : Successfully initialized 'dns'
I, [2016-12-11T17:35:42.182109 #30122] INFO -- : Successfully initialized 'tftp'
D, [2016-12-11T17:35:42.206426 #30122] DEBUG -- : Added a subnet: 192.168.85.0
D, [2016-12-11T17:35:42.208209 #30122] DEBUG -- : Added a reservation: 192.168.85.14:00:19:99:bc:04:e6:kvm02.example.local
D, [2016-12-11T17:35:42.208378 #30122] DEBUG -- : Added a reservation: 192.168.85.13:00:19:99:cb:c2:e2:kvm01.example.local
D, [2016-12-11T17:35:42.208503 #30122] DEBUG -- : Added a reservation: 192.168.85.15:00:19:99:c5:0b:83:kvm03.example.local
I, [2016-12-11T17:35:42.208764 #30122] INFO -- : Successfully initialized 'dhcp_isc'
I, [2016-12-11T17:35:42.208897 #30122] INFO -- : Successfully initialized 'dhcp'
I, [2016-12-11T17:35:42.209459 #30122] INFO -- : Successfully initialized 'puppetca'
I, [2016-12-11T17:35:42.209636 #30122] INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2016-12-11T17:35:42.209752 #30122] INFO -- : Successfully initialized 'puppet'
I, [2016-12-11T17:35:42.243380 #30122] INFO -- : WEBrick 1.3.1
I, [2016-12-11T17:35:42.243661 #30122] INFO -- : ruby 2.0.0 (2014-11-13) [x86_64-linux]
D, [2016-12-11T17:35:42.244102 #30122] DEBUG -- : TCPServer.new(0.0.0.0, 9090)
D, [2016-12-11T17:35:42.244301 #30122] DEBUG -- : TCPServer.new(::, 9090)
W, [2016-12-11T17:35:42.244482 #30122] WARN -- : TCPServer Error: Address already in use - bind(2)
I, [2016-12-11T17:35:42.245666 #30122] INFO -- :
is there anybody who can help?
thanks
Sven
Files
Updated by Edgars Mazurs over 7 years ago
Had the same issue. Solved it by setting symlink:
sudo ln -s /opt/puppetlabs/bin/puppet /usr/bin/puppet
And restart puppet and puppet server. Also remember to revert /etc/sudoers.d/foreman-proxy to default settings:
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet kick *
Defaults:foreman-proxy !requiretty
Updated by Sven Vogel over 7 years ago
thanks for answer. i got it working with. all paths are reset and now they are work.
foreman-installer -v\
--reset-foreman-client-ssl-ca \
--reset-foreman-client-ssl-cert \
--reset-foreman-client-ssl-key \
--reset-foreman-puppet-home \
--reset-foreman-puppet-ssldir \
--reset-foreman-server-ssl-ca \
--reset-foreman-server-ssl-cert \
--reset-foreman-server-ssl-chain \
--reset-foreman-server-ssl-crl \
--reset-foreman-server-ssl-key \
--reset-foreman-websockets-ssl-cert \
--reset-foreman-websockets-ssl-key \
--reset-foreman-proxy-puppet-ssl-ca \
--reset-foreman-proxy-puppet-ssl-cert \
--reset-foreman-proxy-puppet-ssl-key \
--reset-foreman-proxy-puppetca-cmd \
--reset-foreman-proxy-puppetdir \
--reset-foreman-proxy-ssl-ca \
--reset-foreman-proxy-ssl-cert \
--reset-foreman-proxy-ssl-key \
--reset-foreman-proxy-ssldir \
--reset-foreman-puppet-home \
my other problem is now that when i run a puppet agent --test and /etc/puppetlabs/puppet/node.rb myhost i get the following error.
Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
i dont know how to fix that.
Updated by Sven Vogel over 7 years ago
We found the solution.
We debug the node.rb file and checked the certificates which was send to the katello/foreman server.
we have found out that node.rb use the following paths and ca/cert and key file
/etc/puppetlabs/puppet/ssl/client_cert.pem
/etc/puppetlabs/puppet/ssl/client_key.pem
/etc/puppetlabs/puppet/ssl/ssl_ca.pem
after that we checked the /etc/httpd/conf.d/05-foreman-ssl.conf file.
the foreman-installer seems not correct set the paths to the files...
SSLCertificateFile "/etc/puppetlabs/puppet/ssl/certs/test.example.com.pem"
SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/test.example.com.pem"
SSLCertificateChainFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
SSLCACertificateFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"node.rb called files are differen to the 05-foreman-ssl.conf. we change all files into 05-foreman-ssl.conf to
- SSL directives
SSLEngine on
SSLCertificateFile "/etc/puppetlabs/puppet/ssl/client_cert.pem"
SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/client_key.pem"
SSLCertificateChainFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
SSLCACertificateFile "/etc/puppetlabs/puppet/ssl/ssl_ca.pem"
SSLCARevocationFile "/etc/puppetlabs/puppet/ssl/crl.pem"
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars +ExportCertData
after restart tls error was gone.
we checked it on a node with puppet agent --test and got the next error.
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node kvm02.oscloud.local: Exception while executing '/etc/puppetlabs/puppet/node.rb': Cannot run program "/etc/puppetlabs/puppet/node.rb" (in directory "."): error=13, Permission denied
we found that the node.rb dont has puppet right correctly. maybe also a error in upgrade. we changes /etc/puppetlabs/puppet and node.rb to puppet user and group
total 48
drwxrwx--x 8 puppet puppet 4096 Dec 14 12:43 sslrw-r--r- 1 root root 2687 Dec 14 12:43 puppet.confr-xr-x-- 1 root root 11725 Dec 15 14:39 node.rb.changedr-xr-x-- 1 puppet puppet 11345 Dec 15 13:34 node.rbrw-r--r- 1 root root 371 Dec 6 01:17 hiera.yamlrw-r---- 1 root puppet 365 Dec 14 12:43 foreman.yamlrw-rw-r- 1 puppet puppet 0 Dec 14 12:43 autosign.confrw-r--r- 1 root root 4505 Dec 14 12:43 auth.conf
i am open for suggestions if the other files also need puppet rights?! is it a bug and resolvable or foreman-installer problem?
thanks
Sven
Updated by Justin Sherrill over 7 years ago
- translation missing: en.field_release changed from 162 to 188
Updated by Stephen Benjamin over 7 years ago
- Status changed from New to Closed
- translation missing: en.field_release changed from 188 to 166
You might have found a bug somewhere in here, but it's kind of hard to follow along since you did so many different things, can you clarify what the original problem was? What was the output of the sudo command if you ran it directly as foreman-proxy user? I don't see any sudo-related errors in your logs specifically, the proxy just said it failed to list certificates. The sudoers file looks correct.
Later on, the problem was made worse by resetting all those parameters. Those look like they're from the foreman puppet upgrade wiki, but you can't use those instructions with Katello, we have an independent CA. Resetting everything to the puppet defaults will set a bunch of the paths wrong in that case, which cause the problem you had with node.rb. Katello only needs a subset of them reset, and we do it for you in the `--upgrade-puppet` hook (https://github.com/Katello/katello-installer/blob/master/hooks/pre/31-upgrade-puppet.rb#L56-L64).
We've had a bunch of successful puppet 4 upgrades reported - if things are working for you now, I'll close this, unless you can provide some more info to help investigate.
Thanks!
Updated by Sven Vogel about 7 years ago
Stephan,
which are important from https://github.com/Katello/katello-installer/blob/master/hooks/pre/31-upgrade-puppet.rb#L56-L64?
thanks
Sven
Updated by Jorick Astrego over 6 years ago
- Category set to Documentation
- translation missing: en.field_release changed from 166 to 114
- Difficulty set to trivial
We have the same problem, upgraded Katello, ran katello puppet upgrade script.
I saw the deploy fail and in the proxy log that sudo could not be executed:
W, [2017-08-24T10:06:54.391688 ] WARN -- : Failed to run puppetca: [sudo] password for foreman-proxy:
sudo: pam_authenticate: Conversation errorE, [2017-08-24T10:06:54.392710 ] ERROR -- : Failed to remove certificate(s) for xxx.xxx.xxx.xxx: Execution of puppetca failed, check log files
E, [2017-08-24T10:06:54.392710 ] ERROR -- : Failed to remove certificate(s) for xxx.xxx.xxx.xxx: Execution of puppetca failed, check log files
W, [2017-08-24T10:10:05.565436 ] WARN -- : Failed to initialize puppet class cache, deferring initialization. Is puppetserver running?
Sudoers file looks like this:
cat /etc/sudoers.d/foreman-proxy
foreman-proxy ALL = (root) NOPASSWD : /usr/bin/puppet cert *
Defaults:foreman-proxy !requiretty
So I checked my upgrade process again and it looks like I mistakenly did:
foreman-installer --upgrade-puppet
Instead of:
foreman-installer --scenario katello --upgrade-puppet
So I checked the documentation and it clearly states the wrong command:
https://theforeman.org/plugins/katello/3.4/upgrade/puppet.html
In-place migration
If you plan on upgrading an existing Katello server or Smart Proxy to Puppet 4, the the process is straightforward.
Take backup or VM snapshot of server
run katello-service stop to stop all services
run foreman-installer --upgrade-puppet. This will perform the upgrade.
Running "foreman-installer --scenario katello --upgrade-puppet" fixes things.