Plugins » Katello

Dylan Baars wrote:

katello-installer --certs-server-cert "/certs/wellkatellotst.niwa.local.crt"\
--certs-server-cert-req "/certs/wellkatellotst.niwa.local.csr"\
--certs-server-key "/certs/wellkatellotst.niwa.local.key"\
--certs-server-ca-cert "/certs/niwa_cacert.pem"\
--certs-update-server --certs-update-server-ca

Sorry, I should add that we have an internal openSSL CA where the above certificate was signed

Some more information. After installing, /etc/foreman-proxy/settings.yaml on the capsule is below

1. File managed with puppet ###

1. Module: 'foreman_proxy'

:settings_directory: /etc/foreman-proxy/settings.d

1. SSL Setup

1. if enabled, all communication would be verified via SSL
2. NOTE that both certificates need to be signed by the same CA in order for this to work
3. see http://theforeman.org/projects/smart-proxy/wiki/SSL for more information
   :ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
   :ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
   :ssl_private_key: /etc/foreman-proxy/ssl_key.pem

1. the hosts which the proxy accepts connections from
2. commenting the following lines would mean every verified SSL connection allowed
   :trusted_hosts:
   - wellkatellotst.niwa.local
   - wellcapsuletst.niwa.co.nz

1. Endpoint for reverse communication
   :foreman_url: https://wellkatellotst.niwa.local

1. SSL settings for client authentication against Foreman. If undefined, the values
2. from general SSL options are used instead. Mainly useful when Foreman uses
3. different certificates for its web UI and for smart-proxy requests.
   :foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
   :foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
   :foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem

1. by default smart_proxy runs in the foreground. To enable running as a daemon, uncomment 'daemon' setting
   :daemon: true
2. Only used when 'daemon' is set to true.
3. Uncomment and modify if you want to change the default pid file '/var/run/foreman-proxy/foreman-proxy.pid'
   #:daemon_pid: /var/run/foreman-proxy/foreman-proxy.pid

1. host and ports configuration
2. Host or IPs to bind on (e.g. , localhost, 0.0.0.0, ::, 192.168.1.20)
   :bind_host: ''
3. http is disabled by default. To enable, uncomment 'http_port' setting
4. https is enabled if certificate, CA certificate, and private key are present in locations specifed by
5. ssl_certificate, ssl_ca_file, and ssl_private_key correspondingly
6. default values for https_port is 8443
   :https_port: 9090
   :http_port: 8000

1. shared options for virsh DNS/DHCP provider
   :virsh_network: default

1. Log configuration
2. Uncomment and modify if you want to change the location of the log file or use STDOUT or SYSLOG values
   :log_file: /var/log/foreman-proxy/proxy.log
3. Uncomment and modify if you want to change the log level
4. WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
   :log_level: ERROR

1. Log buffer size and extra buffer size (for errors). Defaults to 3000 messages in total,
2. which is about 500 kB request.
   :log_buffer: 2000
   :log_buffer_errors: 1000

If I try and use curl to confirm the certificates are all OK, I get the following

[root@wellcapsuletst foreman-proxy]# curl -v --cacert /etc/foreman-proxy/foreman_ssl_ca.pem --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem https://wellcapsuletst.niwa.co.nz:9090/features

• About to connect() to wellcapsuletst.niwa.co.nz port 9090 (#0)
• Trying 192.168.16.3...
• Connected to wellcapsuletst.niwa.co.nz (192.168.16.3) port 9090 (#0)
• Initializing NSS with certpath: sql:/etc/pki/nssdb
• CAfile: /etc/foreman-proxy/foreman_ssl_ca.pem
  CApath: none
• Server certificate:
• subject: CN=wellcapsuletst.niwa.co.nz,OU=SMART_PROXY,O=FOREMAN,ST=North Carolina,C=US
• start date: Jun 20 04:10:15 2016 GMT
• expire date: Jun 22 04:10:15 2036 GMT
• common name: wellcapsuletst.niwa.co.nz
• issuer: CN=wellkatellotst.niwa.local,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US
• NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
• Peer's certificate issuer has been marked as not trusted by the user.
• Closing connection 0
  curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
  More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Running the equivalent command on the katello master works fine

[root@wellkatellotst foreman-proxy]# curl -v --cacert /etc/foreman-proxy/foreman_ssl_ca.pem --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem https://wellkatellotst.niwa.local:9090/features

• About to connect() to wellkatellotst.niwa.local port 9090 (#0)
• Trying 192.168.59.7...
• Connected to wellkatellotst.niwa.local (192.168.59.7) port 9090 (#0)
• Initializing NSS with certpath: sql:/etc/pki/nssdb
• CAfile: /etc/foreman-proxy/foreman_ssl_ca.pem
  CApath: none
• NSS: client certificate from file
• subject: CN=wellkatellotst.niwa.local,OU=FOREMAN_PROXY,O=FOREMAN,ST=North Carolina,C=US
• start date: Jun 13 00:09:52 2016 GMT
• expire date: Jun 15 00:09:52 2036 GMT
• common name: wellkatellotst.niwa.local
• issuer: CN=wellkatellotst.niwa.local,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North Carolina,C=US
• SSL connection using TLS_RSA_WITH_AES_128_GCM_SHA256
• Server certificate:
• subject: CN=wellkatellotst.niwa.local,OU=NIWA IT,O=National Institute of Water and Atmospheric Research Limited,L=Wellington,ST=NI,C=NZ
• start date: Jun 20 01:35:54 2016 GMT
• expire date: Oct 20 01:35:54 2022 GMT
• common name: wellkatellotst.niwa.local
• issuer: CN=NIWA Root Certificate,OU=IT Operations,O=National Institute of Water and Atmospheric Research Limited,L=Wellington,ST=North Island,C=NZ

GET /features HTTP/1.1
User-Agent: curl/7.29.0
Host: wellkatellotst.niwa.local:9090
Accept: */*

< HTTP/1.1 200 OK
< Content-Type: application/json;charset=utf-8
< Content-Length: 35
< X-Content-Type-Options: nosniff
< Server: WEBrick/1.3.1 (Ruby/2.0.0/2014-11-13) OpenSSL/1.0.1e
< Date: Tue, 28 Jun 2016 02:34:42 GMT
< Connection: Keep-Alive
<

• Connection #0 to host wellkatellotst.niwa.local left intact
  ["pulp","puppet","puppetca","salt"]

Inspecting the certificate on wellcapsuletst it looks OK - issued by wellkatellotst.niwa.local, with the correct DNS name

[root@wellcapsuletst foreman-proxy]# openssl x509 -in /etc/foreman-proxy/foreman_ssl_cert.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 10400263719491574059 (0x905528bffaad992b)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=wellkatellotst.niwa.local
Validity
Not Before: Jun 20 04:10:13 2016 GMT
Not After : Jun 22 04:10:13 2036 GMT
Subject: C=US, ST=North Carolina, O=FOREMAN, OU=FOREMAN_PROXY, CN=wellcapsuletst.niwa.co.nz
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:98:30:4f:bb:9f:6e:aa:82:0f:05:66:90:aa:33:
e7:6d:86:06:97:cc:e4:9b:8c:80:dc:30:73:3f:14:
7f:d8:0e:0b:43:0c:5d:ca:79:ff:92:b0:9c:11:48:
bd:1a:d0:37:10:0b:af:4e:6d:fd:ef:1c:ae:ea:f8:
7e:61:ed:3a:62:3c:19:46:07:20:ec:f7:2f:af:eb:
ad:a3:95:60:03:3a:73:3b:21:4b:4c:63:b1:88:98:
55:04:af:dd:65:1d:72:32:59:0c:43:1a:9e:00:ed:
7d:19:76:5f:ce:ec:96:7b:d2:0b:ca:d4:45:23:7c:
cf:31:0a:56:83:59:3f:5a:9e:f6:90:91:e5:b1:51:
58:f4:25:4f:37:24:33:ef:9a:2f:0f:d3:31:35:94:
ee:bc:40:48:7d:42:fe:64:0a:1f:32:1d:40:4c:4f:
56:ba:fc:7f:fc:8b:28:47:00:d9:da:8e:ff:00:06:
44:d7:7b:da:19:ad:2a:e1:bd:b4:3a:46:ff:c6:ef:
46:8b:a1:1a:08:50:8c:d7:f0:0a:f0:64:ec:1d:9a:
b8:c2:ca:1a:f1:be:a2:be:1d:62:87:4d:45:fd:84:
34:a2:45:b7:a1:68:c5:b3:ca:a7:8a:60:57:85:5d:
e0:21:4a:e9:c2:b3:57:08:72:2a:7b:e0:ae:50:a2:
91:29
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
Netscape Cert Type:
SSL Client
Netscape Comment:
Katello SSL Tool Generated Certificate
X509v3 Subject Key Identifier:
0C:FF:3C:5C:83:DD:72:1E:2A:15:B5:8C:E6:2C:FE:6F:44:30:DB:FB
X509v3 Authority Key Identifier:
keyid:B0:50:63:A4:D0:1B:3C:54:8C:F8:26:8A:9B:51:0A:97:8A:57:4A:8D
DirName:/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=wellkatellotst.niwa.local
serial:90:55:28:BF:FA:AD:99:16

X509v3 Subject Alternative Name:
                DNS:wellcapsuletst.niwa.co.nz
    Signature Algorithm: sha256WithRSAEncryption
         1d:64:a9:2a:5a:14:41:cb:cb:a4:99:81:11:8e:06:2b:23:21:
         68:34:bc:bd:53:07:bf:2b:0c:53:16:0c:ca:c9:fe:ab:66:b3:
         c9:14:e1:2f:bb:b3:d3:a4:3c:30:eb:d6:3e:59:4f:b6:69:56:
         1a:d9:57:b3:37:ea:76:47:87:e7:03:6e:18:a1:9c:19:f2:e6:
         53:48:80:27:b7:bc:c4:e4:10:4e:1c:11:d9:54:ee:bf:07:c5:
         23:db:18:e8:8d:e1:ca:6e:80:b3:dc:98:ad:b8:e0:39:d6:3d:
         16:53:37:36:70:50:41:76:d9:c4:80:e0:ea:4b:d5:7e:f6:56:
         8b:96:98:36:65:17:6f:fc:68:f3:eb:2f:10:72:20:37:1f:d7:
         c4:f1:12:c1:d4:29:84:e9:9f:67:d8:86:7f:49:bd:d1:7f:9a:
         7f:e5:ef:91:e4:61:61:31:82:40:19:c5:62:0e:ba:20:8d:ee:
         f3:e9:2a:6d:f7:02:48:05:5f:b9:39:e6:c6:9c:33:77:6a:6e:
         63:5e:0f:56:3c:3e:1c:45:21:34:04:1b:62:0c:70:96:6a:e2:
         5d:3c:de:04:3d:ab:77:75:2d:6f:fe:96:95:80:74:1e:3c:da:
         fd:1d:00:e0:f9:8b:a1:b8:75:25:c6:1e:ad:60:64:00:ac:29:
         55:20:1f:ed

BUT the CA cert is our internal CA cert, not the katello server CA

[root@wellcapsuletst foreman-proxy]# openssl x509 -in foreman_ssl_ca.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 13082579252401664033 (0xb58ea883fd4bf021)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=NZ, ST=North Island, L=Wellington, O=National Institute of Water and Atmospheric Research Limited, OU=IT Operations, CN=NIWA Root Certificate
Validity
Not Before: Oct 23 20:34:24 2012 GMT
Not After : Oct 21 20:34:24 2022 GMT
Subject: C=NZ, ST=North Island, L=Wellington, O=National Institute of Water and Atmospheric Research Limited, OU=IT Operations, CN=NIWA Root Certificate
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c9:dd:af:5b:0d:ad:02:e6:be:31:69:b9:55:d8:
5e:be:8c:3f:f9:a3:ef:2f:a7:f6:cf:13:f7:68:d6:
1e:a1:52:9a:92:0e:57:2e:0b:05:72:f9:39:ee:12:
08:7c:17:56:f2:2b:a4:21:3d:4d:b9:cf:36:08:80:
ae:3a:14:7f:6f:2d:68:87:c8:3c:b7:38:06:c3:85:
49:79:bf:48:a6:bb:08:95:b5:3a:f1:70:9b:e8:35:
fc:0a:1b:45:3f:a3:c9:72:7c:e9:02:c1:3c:df:bc:
ca:37:28:10:88:ce:e4:fe:2e:46:71:da:7a:45:dd:
f1:c7:db:f2:d0:8a:24:30:ca:46:7f:cd:04:23:fc:
f5:c2:15:df:b5:60:c3:9f:40:e9:1e:d7:f7:40:a8:
f0:42:da:bf:62:f5:7c:14:ae:34:a8:72:43:d0:e5:
40:b4:4a:be:d5:ab:c5:f1:aa:e2:ab:36:02:47:56:
17:bb:cc:ab:a8:5f:3f:e3:27:23:64:40:74:3a:6b:
03:0a:b2:fd:42:03:8e:6c:b8:c7:98:e0:c1:fa:d6:
1f:0d:45:14:7e:e2:c4:0f:b8:db:31:7d:f8:c0:23:
bb:6e:a6:72:f0:a7:8c:ab:ed:3a:72:e2:ed:f4:75:
52:75:67:db:a9:70:9b:cd:9f:56:63:9c:ec:b5:96:
1d:ad
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
6C:AD:CC:AA:66:08:82:7B:43:14:BD:9F:ED:8D:F4:EE:A2:52:59:74
X509v3 Authority Key Identifier:
keyid:6C:AD:CC:AA:66:08:82:7B:43:14:BD:9F:ED:8D:F4:EE:A2:52:59:74
DirName:/C=NZ/ST=North Island/L=Wellington/O=National Institute of Water and Atmospheric Research Limited/OU=IT Operations/CN=NIWA Root Certificate
serial:B5:8E:A8:83:FD:4B:F0:21

X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha1WithRSAEncryption
         ad:f7:d6:e0:c5:19:50:44:97:c2:dc:d7:6f:19:9a:fc:5d:40:
         39:93:9e:39:08:2b:92:ce:9e:96:c1:bd:f2:3a:dd:a2:ae:e7:
         f2:b0:84:1c:00:d4:66:ec:66:b5:12:1d:cd:f3:cf:12:c2:c9:
         bc:46:42:45:f4:9f:a2:8d:f3:25:5b:d2:92:fe:41:a9:d5:0d:
         81:ef:e7:a2:67:2d:69:2e:40:ce:df:ed:dd:ff:da:07:f5:b7:
         4f:19:4a:53:22:7f:2d:5b:09:be:c2:ac:63:c1:49:b6:5c:6b:
         0f:47:a0:12:bb:16:dc:06:c0:7c:fe:85:65:f2:b1:eb:87:7f:
         3d:9e:5c:b9:a8:bc:ad:cc:68:7c:03:42:71:28:de:1d:46:40:
         96:b5:a1:db:75:c9:47:ca:b1:49:ba:1d:c0:c7:cb:03:ba:84:
         20:7e:ec:be:fd:70:16:c3:b4:db:bb:41:7b:b3:e1:70:ef:26:
         3f:cf:f2:7f:4c:fb:97:20:1c:8a:4e:02:6e:da:d0:a1:05:23:
         ab:eb:1b:d2:3a:15:09:50:f4:2c:fe:31:3e:ea:5d:f5:be:08:
         b2:ad:dc:04:2f:2a:ad:c3:2e:26:ec:ba:37:f6:8e:3e:53:55:
         71:9d:38:ce:96:da:dc:ed:d4:3a:d4:8b:39:1e:27:d0:05:bb:
         06:3f:9f:b7

This is actually the same configuration as our katello master, and that's working - so not sure?

I restarted foreman-proxy on the capsule after editing settings.yml and this is the log /var/log/foreman-proxy/proxy.log while trying to add the proxy via the katello GUI > Infrastructure > Smart Proxies > New Smart Proxy

I, [2016-06-28T03:03:08.409379 #7224] INFO -- : Finished initialization of module 'pulpnode'
I, [2016-06-28T03:03:08.409618 #7224] INFO -- : 'foreman_proxy' settings were initialized with default values: :enabled: true
I, [2016-06-28T03:03:08.411227 #7224] INFO -- : Finished initialization of module 'foreman_proxy'
I, [2016-06-28T03:03:08.412875 #7224] INFO -- : Finished initialization of module 'templates'
I, [2016-06-28T03:03:08.966438 #7224] INFO -- : Finished initialization of module 'puppetca'
I, [2016-06-28T03:03:08.966729 #7224] INFO -- : 'puppet' settings were initialized with default values: :puppet_provider: puppetrun, :salt_puppetrun_cmd: puppet.run, :use_cache: true
I, [2016-06-28T03:03:08.971030 #7224] INFO -- : Initializing from Puppet config file: /etc/puppet/puppet.conf
I, [2016-06-28T03:03:09.167832 #7224] INFO -- : Finished initialization of module 'puppet'
I, [2016-06-28T03:03:09.185579 #7229] INFO -- : WEBrick 1.3.1
I, [2016-06-28T03:03:09.185949 #7229] INFO -- : ruby 2.0.0 (2014-11-13) [x86_64-linux]
I, [2016-06-28T03:03:09.186993 #7229] INFO -- : WEBrick 1.3.1
I, [2016-06-28T03:03:09.187213 #7229] INFO -- : ruby 2.0.0 (2014-11-13) [x86_64-linux]
D, [2016-06-28T03:03:09.187475 #7229] DEBUG -- : TCPServer.new(0.0.0.0, 8000)
D, [2016-06-28T03:03:09.187946 #7229] DEBUG -- : TCPServer.new(::, 8000)
D, [2016-06-28T03:03:09.188136 #7229] DEBUG -- : TCPServer.new(0.0.0.0, 9090)
D, [2016-06-28T03:03:09.188430 #7229] DEBUG -- : TCPServer.new(::, 9090)
W, [2016-06-28T03:03:09.188901 #7229] WARN -- : TCPServer Error: Address already in use - bind(2)
I, [2016-06-28T03:03:09.192008 #7229] INFO -- :
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 10400263719491574060 (0x905528bffaad992c)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=wellkatellotst.niwa.local
Validity
Not Before: Jun 20 04:10:15 2016 GMT
Not After : Jun 22 04:10:15 2036 GMT
Subject: C=US, ST=North Carolina, O=FOREMAN, OU=SMART_PROXY, CN=wellcapsuletst.niwa.co.nz
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:ad:5b:41:b8:a0:f8:3c:66:21:67:51:82:47:78:
20:e8:07:62:17:17:d0:7b:cc:c3:45:5f:10:e0:b6:
0c:11:7b:18:be:db:4e:12:bd:e6:5f:0f:37:40:34:
71:b4:00:ce:34:43:8a:0e:93:8f:bc:60:d2:d5:6d:
60:fb:54:63:e2:f1:5d:b2:4e:9d:52:35:f9:c1:1a:
de:f8:bf:bd:16:23:5d:d1:3a:14:d3:22:a7:82:04:
b1:2c:65:35:c4:a8:75:e6:7c:a2:a5:c5:07:25:33:
85:75:1e:b1:66:c6:59:08:e1:3a:cb:b1:83:57:a2:
b9:9b:47:7b:ed:64:af:c1:61:a9:53:f3:7d:01:3f:
e1:63:21:cd:ee:d8:b6:dc:a2:e2:78:c0:a2:18:d9:
72:bd:1c:a6:2f:85:7c:cf:f5:a4:08:df:99:e7:95:
44:cb:fc:77:9f:a4:a1:53:bf:8a:94:02:3d:65:78:
53:c6:54:54:3a:8e:83:cf:0b:44:56:a1:1f:f5:6e:
4d:62:06:76:3e:f3:09:fa:a5:2a:17:d9:7b:46:86:
b4:bd:da:57:1b:79:f1:96:03:58:40:8d:91:1d:0f:
a7:d1:7a:2c:af:69:37:08:72:dd:e2:64:01:11:c9:
c1:f0:9f:42:0c:cb:8c:e2:c3:83:3a:3f:f9:53:2e:
39:23
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
Netscape Cert Type:
SSL Server
Netscape Comment:
Katello SSL Tool Generated Certificate
X509v3 Subject Key Identifier:
1D:7E:1F:06:9F:CD:2E:72:73:29:97:AB:74:6E:49:17:D2:CC:AD:C0
X509v3 Authority Key Identifier:
keyid:B0:50:63:A4:D0:1B:3C:54:8C:F8:26:8A:9B:51:0A:97:8A:57:4A:8D
DirName:/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=wellkatellotst.niwa.local
serial:90:55:28:BF:FA:AD:99:16

X509v3 Subject Alternative Name:
                DNS:wellcapsuletst.niwa.co.nz
    Signature Algorithm: sha256WithRSAEncryption
         56:d0:35:4e:ed:87:53:89:a5:be:50:5e:14:7c:4d:62:96:09:
         84:65:d5:aa:fe:ee:0e:65:9d:48:21:a8:7f:3c:d6:1a:0d:2f:
         4f:9d:33:da:aa:43:1d:56:e7:d7:83:97:94:2c:e3:66:4b:a5:
         72:24:96:b8:f2:43:09:ed:0a:87:89:10:61:ba:d8:de:c2:03:
         e9:55:f7:a6:46:e7:a9:28:da:a7:1c:6e:97:c6:6e:c5:33:8f:
         6a:4e:60:98:16:7f:3c:19:ef:c5:90:d5:a4:4f:3b:55:67:1f:
         30:ba:68:1c:97:fe:7a:98:ab:ff:3c:22:3c:b7:58:a3:e5:3f:
         3a:5a:c6:cb:a8:46:8c:7a:1c:9d:b3:ed:d9:63:a7:de:42:e3:
         8f:7c:13:d5:83:93:bd:bd:e9:5f:29:86:9e:7d:45:bd:f1:ba:
         9c:ff:7b:25:88:e0:a4:de:3a:9f:4f:76:2b:27:32:00:fa:70:
         0f:c2:8e:cc:c9:74:6f:51:27:fc:1b:1f:fb:ab:e2:c9:a9:ef:
         fd:f3:30:31:c2:ce:79:89:01:67:79:de:fc:ad:c1:22:8f:78:
         3f:3d:a1:87:38:f6:b0:c2:f3:90:90:49:6a:78:e6:44:3f:0c:
         b4:21:20:65:c5:b0:6b:4a:a6:fe:d3:b6:49:be:0c:06:a1:83:
         e6:fc:4c:c4

W, [2016-06-28T03:03:09.192283 #7229] WARN -- : TCPServer Error: Address already in use - bind(2)
D, [2016-06-28T03:03:09.240565 #7229] DEBUG -- : Rack::Handler::WEBrick is mounted on /.
D, [2016-06-28T03:03:09.240798 #7229] DEBUG -- : Rack::Handler::WEBrick is mounted on /.
I, [2016-06-28T03:03:09.241190 #7229] INFO -- : WEBrick::HTTPServer#start: pid=7229 port=9090
I, [2016-06-28T03:03:09.241578 #7229] INFO -- : WEBrick::HTTPServer#start: pid=7229 port=8000
E, [2016-06-28T03:04:09.708758 #7229] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
/usr/share/ruby/openssl/ssl.rb:226:in `accept'

So I turned SELinux from enforcing to permissive, and everything started working. I can't find anything useful in /var/log/messages or /var/log/audit/audit.log to suggest what selinux is stopping (nothing comes up when trying to add the smart proxy or refresh features if its added already)

We want to be able to run selinux, and I imagine other will as well :-)

Dylan Baars wrote:

So I turned SELinux from enforcing to permissive, and everything started working. I can't find anything useful in /var/log/messages or /var/log/audit/audit.log to suggest what selinux is stopping (nothing comes up when trying to add the smart proxy or refresh features if its added already)

We want to be able to run selinux, and I imagine other will as well :-)

Opps incorrect - it starting working for HTTP - on port 8080. HTTPS still is broken :(

Good morning,

I have solved it (amazing what sleeping on a problem can do!). I created a private key and CSR on the capsule -

wget -N -P . http://wellkatellotst.niwa.local/pub/ks/niwa_cacert.pem
openssl genrsa -out wellcapsuletst.niwa.co.nz.key 2048
openssl req -nodes -newkey rsa:2048 -keyout wellcapsuletst.niwa.co.nz.key -out wellcapsuletst.niwa.co.nz.csr -subj "/C=NZ/ST=NI/L=Wellington/O=National Institute of Water and Atmospheric Research Limited/OU=NIWA IT/CN=wellcapsuletst.niwa.co.nz"

Signed the cert on our internal CA, then copied the private key and cert to /etc/foreman-proxy on the capsule
chmod 400 /etc/foreman-proxy/wellcapsuletst.niwa.co.nz.key
chown foreman-proxy:root /etc/foreman-proxy/wellcapsuletst.niwa.co.nz.key
chmod 644 /etc/foreman/proxy/wellcapsuletst.niwa.co.nz.crt

Edited /etc/foreman-proxy/settings.yml and updated the ssl settings as below - the cert and private key became the internal CA signed cert and matching private key, but the ssl_ca_file remained as the katello CA cert. This matches the main katello servers foreman-proxy configuration

:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
#:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_certificate: /etc/foreman-proxy/wellcapsuletst.niwa.co.nz.crt
:ssl_private_key: /etc/foreman-proxy/wellcapsuletst.niwa.co.nz.key
#:ssl_private_key: /etc/foreman-proxy/ssl_key.pem

Restarted foreman-proxy on the capsule - "systemctl restart foreman-proxy"

Then in the GUI, updated the smart proxy to HTTPS (https://wellcapsuletst.niwa.co.nz:9090) and refreshed settings - all OK! Just to be sure, I removed the smart proxy from the GUI, rebooted the capsule, then added it back in using HTTPS - worked fine

Hi, some more (not sure if its related) information - now with the capsule running, and content synchronising to it, I installed a new VM into our DMZ attached to this capsule for content. Yum update gives the following

[root@hamldmztest01 yum.repos.d]# yum update
Loaded plugins: changelog, fastestmirror, product-id, search-disabled-repos, subscription-manager, versionlock
https://wellcapsuletst.niwa.co.nz/pulp/repos/NIWA/Prod-Server/CentOS7_Server/custom/CentOS7/centosplus_x86_64/repodata/repomd.xml: [Errno 14] curl#60 - "Peer's certificate issuer has been marked as not trusted by the user."
Trying other mirror.
It was impossible to connect to the CentOS servers.
This could mean a connectivity issue in your environment, such as the requirement to configure a proxy,
or a transparent proxy that tampers with TLS security, or an incorrect system clock.
Please collect information about the specific failure that occurs in your environment,
using the instructions in: https://access.redhat.com/solutions/1527033 and create a bug on https://bugs.centos.org/
One of the configured repositories failed (centosplus x86_64),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then
just ignore the repository until you permanently enable it again or use
--enablerepo for temporary usage:
yum-config-manager --disable NIWA_CentOS7_centosplus_x86_64
4. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=NIWA_CentOS7_centosplus_x86_64.skip_if_unavailable=true
failure: repodata/repomd.xml from NIWA_CentOS7_centosplus_x86_64: [Errno 256] No more mirrors to try.
https://wellcapsuletst.niwa.co.nz/pulp/repos/NIWA/Prod-Server/CentOS7_Server/custom/CentOS7/centosplus_x86_64/repodata/repomd.xml: [Errno 14] curl#60 - "Peer's certificate issuer has been marked as not trusted by the user."
[root@hamldmztest01 yum.repos.d]#

Initially I thought this was a time issue, as it was incorrectly set to UTC instead of NZST. Fixed that, but still the same problem. If I edit /etc/yum.repos.d/redhat.repo and change the sslverify from 1 to 0, it works fine.

Some investigation:
In /etc/yum.repos.d/redhat.repo there is (as an example) this repository configured

[NIWA_CentOS7_extras_x86_64]
metadata_expire = 1
sslclientcert = /etc/pki/entitlement/4388655353131018046.pem
baseurl = https://wellcapsuletst.niwa.co.nz/pulp/repos/NIWA/Prod-Server/CentOS7_Server/custom/CentOS7/extras_x86_64
sslverify = 0
name = extras x86_64
sslclientkey = /etc/pki/entitlement/4388655353131018046-key.pem
gpgkey = https://wellcapsuletst.niwa.co.nz/katello/api/repositories/4/gpg_key_content
enabled = 1
sslcacert = /etc/rhsm/ca/katello-server-ca.pem
gpgcheck = 1

The client certificate that has been installed (sslclientcert) -

[root@hamldmztest01 yum.repos.d]# openssl x509 -text -noout -in /etc/pki/entitlement/4388655353131018046.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4388655353131018046 (0x3ce7a2caa4fda73e)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=wellkatellotst.niwa.local
Validity
Not Before: Jun 20 02:08:57 2016 GMT
Not After : Jun 13 02:08:57 2046 GMT
Subject: CN=4028bb8755d6c9a20155e744079e012f
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:9c:c1:e8:0f:be:b1:a2:03:10:f9:4e:70:3e:c0:
01:b9:4e:b6:0b:4b:f3:f8:2c:2c:3f:df:ec:a2:12:
b1:65:ba:d6:a4:f5:3e:c0:ed:66:7e:ef:73:cd:c9:
c3:17:1c:b4:7c:b6:66:13:04:b3:5c:c7:de:94:12:
b0:1d:e7:b2:d0:34:d1:01:e1:e4:8e:7f:c6:35:b7:
5f:24:27:92:bd:01:c0:39:10:10:c8:51:9d:7c:41:
de:32:2c:e3:ed:cd:a1:1b:4e:d8:c5:b4:84:de:2c:
2b:e0:98:dc:be:21:21:e4:2e:56:dd:17:89:2d:32:
f7:89:84:65:03:fe:5a:de:6f:bd:3b:ff:fe:ef:13:
cf:69:b3:5d:71:3d:62:87:0b:8b:c6:16:de:73:f9:
87:62:1a:aa:72:b2:3f:ea:eb:e3:12:dd:7d:52:22:
00:60:78:f3:f0:5b:ee:d9:b5:d5:55:e2:7b:79:ca:
a3:04:3f:41:82:6a:7a:b9:f4:f4:e4:fc:5c:e8:54:
f6:ef:63:05:55:00:d7:ee:58:10:76:98:9e:f7:b1:
cf:54:1d:33:1a:cc:1f:2f:ba:71:30:c4:4e:df:52:
c2:ea:6e:cd:b2:20:c0:13:d4:4a:2b:28:dc:72:8e:
a7:10:dd:3d:26:5b:22:49:39:54:c2:ae:43:c6:49:
30:bf
Exponent: 65537 (0x10001)
X509v3 extensions:
Netscape Cert Type:
SSL Client, S/MIME
X509v3 Key Usage:
Digital Signature, Key Encipherment, Data Encipherment
X509v3 Authority Key Identifier:
keyid:B0:50:63:A4:D0:1B:3C:54:8C:F8:26:8A:9B:51:0A:97:8A:57:4A:8D
DirName:/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=wellkatellotst.niwa.local
serial:90:55:28:BF:FA:AD:99:16

X509v3 Subject Key Identifier:
                2A:03:8F:BF:79:69:25:DE:43:EB:8E:C1:D7:CE:03:3D:A7:DD:4A:12
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            1.3.6.1.4.1.2312.9.6:
                ..3.2
            1.3.6.1.4.1.2312.9.7:
                .jx../...0.73aHK,.))JL....y.;28.............1.....B.......R..P........E0ViAJbI*B...$?...=.+K.......|...a..
    Signature Algorithm: sha1WithRSAEncryption
         47:97:53:05:40:c1:e8:2f:0a:cc:15:c7:e5:0a:ee:56:0f:f0:
         72:4d:5a:40:2f:62:b0:cb:28:8d:35:70:c2:9d:c8:bc:eb:c1:
         3c:0b:af:d3:37:f9:01:6d:c1:e6:c1:b2:98:8c:5d:bd:fa:c4:
         ad:5b:c1:54:24:6d:43:d6:19:80:34:df:a1:8c:2b:9f:8a:01:
         d4:a5:7d:0b:eb:f5:ea:14:f7:9d:92:49:19:b5:2d:30:71:d4:
         3d:59:83:69:40:23:41:60:ba:11:d5:75:e3:b9:68:fb:bc:fd:
         85:9e:3a:a8:1c:f6:2b:55:24:b6:7e:f1:a8:36:3f:90:d9:29:
         ae:67:61:de:15:e6:88:31:2e:60:e4:80:e1:0c:fb:68:ee:9f:
         94:cf:3c:69:ec:e4:a6:68:9a:89:f3:71:4e:cc:d6:03:67:f4:
         14:68:23:13:18:f8:bb:f0:b9:35:86:12:0f:e7:6f:61:75:a6:
         ea:98:16:97:a9:6a:4e:4d:d8:53:32:01:52:22:9a:e4:cb:be:
         a7:5c:b3:19:d5:f1:6b:c5:d1:dd:c8:bb:02:1f:e0:e5:70:e1:
         ea:9b:f6:24:e0:f2:c4:3f:87:17:da:28:13:b9:40:4b:0e:cd:
         3f:b2:a9:37:a0:51:13:fa:32:c1:b0:d1:ab:a1:09:c7:cf:b2:
         4b:da:13:db
[root@hamldmztest01 yum.repos.d]#

so its issued by C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=wellkatellotst.niwa.local

Interestingly, the sslcacert, /etc/rhsm/ca/katello-server-ca.pem is our internal NIWA CA

[root@hamldmztest01 yum.repos.d]# openssl x509 -noout -text -in /etc/rhsm/ca/katello-server-ca.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 13082579252401664033 (0xb58ea883fd4bf021)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=NZ, ST=North Island, L=Wellington, O=National Institute of Water and Atmospheric Research Limited, OU=IT Operations, CN=NIWA Root Certificate
Validity
Not Before: Oct 23 20:34:24 2012 GMT
Not After : Oct 21 20:34:24 2022 GMT
Subject: C=NZ, ST=North Island, L=Wellington, O=National Institute of Water and Atmospheric Research Limited, OU=IT Operations, CN=NIWA Root Certificate
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c9:dd:af:5b:0d:ad:02:e6:be:31:69:b9:55:d8:
5e:be:8c:3f:f9:a3:ef:2f:a7:f6:cf:13:f7:68:d6:
1e:a1:52:9a:92:0e:57:2e:0b:05:72:f9:39:ee:12:
08:7c:17:56:f2:2b:a4:21:3d:4d:b9:cf:36:08:80:
ae:3a:14:7f:6f:2d:68:87:c8:3c:b7:38:06:c3:85:
49:79:bf:48:a6:bb:08:95:b5:3a:f1:70:9b:e8:35:
fc:0a:1b:45:3f:a3:c9:72:7c:e9:02:c1:3c:df:bc:
ca:37:28:10:88:ce:e4:fe:2e:46:71:da:7a:45:dd:
f1:c7:db:f2:d0:8a:24:30:ca:46:7f:cd:04:23:fc:
f5:c2:15:df:b5:60:c3:9f:40:e9:1e:d7:f7:40:a8:
f0:42:da:bf:62:f5:7c:14:ae:34:a8:72:43:d0:e5:
40:b4:4a:be:d5:ab:c5:f1:aa:e2:ab:36:02:47:56:
17:bb:cc:ab:a8:5f:3f:e3:27:23:64:40:74:3a:6b:
03:0a:b2:fd:42:03:8e:6c:b8:c7:98:e0:c1:fa:d6:
1f:0d:45:14:7e:e2:c4:0f:b8:db:31:7d:f8:c0:23:
bb:6e:a6:72:f0:a7:8c:ab:ed:3a:72:e2:ed:f4:75:
52:75:67:db:a9:70:9b:cd:9f:56:63:9c:ec:b5:96:
1d:ad
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
6C:AD:CC:AA:66:08:82:7B:43:14:BD:9F:ED:8D:F4:EE:A2:52:59:74
X509v3 Authority Key Identifier:
keyid:6C:AD:CC:AA:66:08:82:7B:43:14:BD:9F:ED:8D:F4:EE:A2:52:59:74
DirName:/C=NZ/ST=North Island/L=Wellington/O=National Institute of Water and Atmospheric Research Limited/OU=IT Operations/CN=NIWA Root Certificate
serial:B5:8E:A8:83:FD:4B:F0:21

X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha1WithRSAEncryption
         ad:f7:d6:e0:c5:19:50:44:97:c2:dc:d7:6f:19:9a:fc:5d:40:
         39:93:9e:39:08:2b:92:ce:9e:96:c1:bd:f2:3a:dd:a2:ae:e7:
         f2:b0:84:1c:00:d4:66:ec:66:b5:12:1d:cd:f3:cf:12:c2:c9:
         bc:46:42:45:f4:9f:a2:8d:f3:25:5b:d2:92:fe:41:a9:d5:0d:
         81:ef:e7:a2:67:2d:69:2e:40:ce:df:ed:dd:ff:da:07:f5:b7:
         4f:19:4a:53:22:7f:2d:5b:09:be:c2:ac:63:c1:49:b6:5c:6b:
         0f:47:a0:12:bb:16:dc:06:c0:7c:fe:85:65:f2:b1:eb:87:7f:
         3d:9e:5c:b9:a8:bc:ad:cc:68:7c:03:42:71:28:de:1d:46:40:
         96:b5:a1:db:75:c9:47:ca:b1:49:ba:1d:c0:c7:cb:03:ba:84:
         20:7e:ec:be:fd:70:16:c3:b4:db:bb:41:7b:b3:e1:70:ef:26:
         3f:cf:f2:7f:4c:fb:97:20:1c:8a:4e:02:6e:da:d0:a1:05:23:
         ab:eb:1b:d2:3a:15:09:50:f4:2c:fe:31:3e:ea:5d:f5:be:08:
         b2:ad:dc:04:2f:2a:ad:c3:2e:26:ec:ba:37:f6:8e:3e:53:55:
         71:9d:38:ce:96:da:dc:ed:d4:3a:d4:8b:39:1e:27:d0:05:bb:
         06:3f:9f:b7

In /etc/rhsm/ca/ there is a "katello-default-ca.pem" which is the CA for katellotst.niwa.local, so I did the following

cd /etc/rhsm/ca/
mv katello-server-ca.pem katello-server-ca.pem.orig
cp katello-default-ca.pem katello-server-ca.pem

Edited /etc/yum.repos.d/redhat.repo and set sslverify back to 1.

yum clean all
yum update

This time it works perfectly!

In /etc/rhsm/rhsm.conf, there is this line

1. Default CA cert to use when generating yum repo configs:
   repo_ca_cert = %(ca_cert_dir)skatello-server-ca.pem

I edited it as below

1. Default CA cert to use when generating yum repo configs:
   #repo_ca_cert = %(ca_cert_dir)skatello-server-ca.pem
   repo_ca_cert = %(ca_cert_dir)skatello-default-ca.pem

which of course does the same thing as the file replacement above. This also works - all repositories in redhat.repo are updated with the following sslcacert line

sslcacert = /etc/rhsm/ca/katello-default-ca.pem

and all yum commands begin to work

Is there a way to update the katello-ca-consumer-latest.noarch.rpm rpm on the capsule so it deploys the correct certificate and/or correct configuration of rhsm.conf?

Happy to create a separate ticket for this if you think it needs one

Cheers,
Dylan