Hammer CLI

Not sure if this is related, but I'm also struggling a bit with something that smells a lot like this topic. Setup is Freeipa 4.1 + Foreman 1.8.3 on Centos7 (fully patched).

• Kerberos setup
  

  [root@puppet01 ~]# kdestroy
  [root@puppet01 ~]# kinit earsdown
  Password for earsdown@EXAMPLE.COM: 
  [root@puppet01 ~]# klist
  Ticket cache: KEYRING:persistent:0:0
  Default principal: earsdown@EXAMPLE.COM
  
  Valid starting       Expires              Service principal
  09/07/2015 09:25:30  09/08/2015 09:25:27  krbtgt/EXAMPLE.COM@EXAMPLE.COM
  [root@puppet01 ~]# 
  

• First attempt without session_id cookie or auth - which should (and does) fail
  

  [root@puppet01 ~]# foreman_url=https://puppet01.example.com
  [root@puppet01 ~]# curl -vv -b /root/cookies.txt -c /root/cookies.txt -s $foreman_url/api/hosts | jq .
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  > GET /api/hosts HTTP/1.1
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  >
  < HTTP/1.1 401 Unauthorized
  < Date: Mon, 07 Sep 2015 09:29:41 GMT
  < Server: Apache/2.4.6 (CentOS)
  < Apipie-Checksum: 4dc90ee161b2f69d513ec4065c3fe739
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: no-cache
  < X-Request-Id: 200e84a99465d4cb0454c48b93cf2be0
  < X-Runtime: 0.022239
  < X-Rack-Cache: miss
  < X-Powered-By: Phusion Passenger 4.0.18
  * Added cookie request_method="" for domain puppet01.example.com, path /, expire 1
  < Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  < Status: 401 Unauthorized
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: application/json; charset=utf-8
  <
  { [data not shown]
  * Closing connection 0
  {
    "message": "Unable to authenticate user " 
  }
  

• This time we're going to hit /users/extlogin with "--negotiate -u :", which will log us in and give us a session_id cookie.
  

  [root@puppet01 ~]#   curl  -vv --negotiate  -u : -b /root/cookies.txt -c /root/cookies.txt -s $foreman_url/users/extlogin
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  > GET /users/extlogin HTTP/1.1
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  >
  < HTTP/1.1 401 Unauthorized
  < Date: Mon, 07 Sep 2015 09:29:59 GMT
  < Server: Apache/2.4.6 (CentOS)
  < WWW-Authenticate: Negotiate
  < Content-Length: 120
  < Connection: close
  < Content-Type: text/html; charset=iso-8859-1
  <
  * Closing connection 0
  * Issue another request to this URL: 'https://puppet01.example.com/users/extlogin'
  * About to connect() to puppet01.example.com port 443 (#1)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#1)
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  * Server auth using GSS-Negotiate with user ''
  > GET /users/extlogin HTTP/1.1
  > Authorization: Negotiate <FILTERED>
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  >
  < HTTP/1.1 302 Found
  < Date: Mon, 07 Sep 2015 09:29:59 GMT
  < Server: Apache/2.4.6 (CentOS)
  < WWW-Authenticate: Negotiate <FILTERED>
  < Strict-Transport-Security: max-age=631152000; includeSubdomains
  < X-Frame-Options: SAMEORIGIN
  < Content-Security-Policy: default-src 'self'; connect-src 'self' ws: wss:; font-src 'self'; frame-src 'self'; img-src 'self' *.gravatar.com data:; media-src 'self'; object-src 'self'; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self';
  < X-XSS-Protection: 1; mode=block
  < X-Content-Type-Options: nosniff
  < X-Download-Options: noopen
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: no-cache, private
  < X-Request-Id: d0c49e289a4f945717f8445a1e1741bd
  < X-Runtime: 0.742713
  < X-Rack-Cache: miss
  < X-Powered-By: Phusion Passenger 4.0.18
  * Added cookie _session_id="09f9771105c20aebedad6a5fbbd8af13" for domain puppet01.example.com, path /, expire 0
  < Set-Cookie: _session_id=09f9771105c20aebedad6a5fbbd8af13; path=/; secure; HttpOnly
  * Replaced cookie request_method="" for domain puppet01.example.com, path /, expire 1
  < Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  < Location: https://puppet01.example.com/hosts
  < Status: 302 Found
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: text/html; charset=utf-8
  <
  * Closing connection 1
  <html><body>You are being <a href="https://puppet01.example.com/hosts">redirected</a>.</body></html>
  

• Now, armed with our session_id cookie, we can hit /api/hosts and it works without the need for auth.
  

  [root@puppet01 ~]# curl  -vv -b /root/cookies.txt -c /root/cookies.txt -s "$foreman_url/api/hosts?per_page=999&search=proxyvpngw" | jq .
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  > GET /api/hosts?per_page=999&search=proxyvpngw HTTP/1.1
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  > Cookie: _session_id=09f9771105c20aebedad6a5fbbd8af13
  > 
  < HTTP/1.1 200 OK
  < Date: Mon, 07 Sep 2015 09:44:11 GMT
  < Server: Apache/2.4.6 (CentOS)
  < Foreman_version: 1.8.3
  < Foreman_api_version: 1
  < Apipie-Checksum: 4dc90ee161b2f69d513ec4065c3fe739
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: must-revalidate, private, max-age=0
  < X-Request-Id: d38e7d2816708bb51f08963c868c735a
  < X-Runtime: 0.184627
  < X-Rack-Cache: miss
  < X-Powered-By: Phusion Passenger 4.0.18
  * Replaced cookie request_method="" for domain puppet01.example.com, path /, expire 1 
  < Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
  < ETag: "239b2e5e3ac4a9827bba8c05e735214c" 
  < Status: 200 OK
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: application/json; charset=utf-8
  < 
  { [data not shown]
  * Closing connection 0
  [
    {
      "host": {
        "operatingsystem_id": 1,
        "hostgroup_id": 6,
        "id": 17,
        "name": "proxyvpngw-i-1f7c8fc0.example.com" 
      }
    },
    {
      "host": {
        "operatingsystem_id": 1,
        "hostgroup_id": 6,
        "id": 14,
        "name": "proxyvpngw-i-790cfca6.example.com" 
      }
    },
    {
      "host": {
        "operatingsystem_id": 1,
        "hostgroup_id": 6,
        "id": 15,
        "name": "proxyvpngw-i-cb50a014.example.com" 
      }
    }
  ]
  

• But when I try to delete a host (via DELETE /api/hosts/<hostid>), foreman freaks out and invalidates our credentials:
  

  [root@puppet01 ~]# curl  -vv -b /root/cookies.txt -c /root/cookies.txt -s -X DELETE $foreman_url/api/hosts/15 | jq .
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  > DELETE /api/hosts/15 HTTP/1.1
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  > Cookie: _session_id=09f9771105c20aebedad6a5fbbd8af13
  > 
  < HTTP/1.1 401 Unauthorized
  < Date: Mon, 07 Sep 2015 09:48:36 GMT
  < Server: Apache/2.4.6 (CentOS)
  < Apipie-Checksum: 4dc90ee161b2f69d513ec4065c3fe739
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: no-cache
  < X-Request-Id: b81193ba08de9cbd609957dfc9a2b918
  < X-Runtime: 0.024028
  < X-Rack-Cache: invalidate, pass
  < X-Powered-By: Phusion Passenger 4.0.18
  * Replaced cookie _session_id="d57d0690d37af112761afda6695ac4a1" for domain puppet01.example.com, path /, expire 0 
  < Set-Cookie: _session_id=d57d0690d37af112761afda6695ac4a1; path=/; secure; HttpOnly
  * Replaced cookie request_method="DELETE" for domain puppet01.example.com, path /, expire 0 
  < Set-Cookie: request_method=DELETE; path=/
  < Status: 401 Unauthorized
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: application/json; charset=utf-8
  < 
  { [data not shown]
  * Closing connection 0
  {
    "message": "Unable to authenticate user " 
  }
  

Interestingly, it throws a 401 but doesn't offer a WWW-Authenticate header.

• ...and here's what appears in the production.log:
  

  Started DELETE "/api/hosts/15" for 10.239.12.20 at 2015-09-07 09:48:36 +0000
  2015-09-07 09:48:36 [I] Processing by Api::V1::HostsController#destroy as JSON
  2015-09-07 09:48:36 [I]   Parameters: {"apiv"=>"v1", "id"=>"15"}
  2015-09-07 09:48:36 [W] WARNING: Can't verify CSRF token authenticity
  2015-09-07 09:48:36 [I]   Rendered api/v1/errors/unauthorized.json.rabl (0.6ms)
  2015-09-07 09:48:36 [I] Filter chain halted as :authorize rendered or redirected
  2015-09-07 09:48:36 [I] Completed 401 Unauthorized in 13ms (Views: 1.8ms | ActiveRecord: 3.0ms)
  

Foreman does actually invalidate the previously-working credentials - i.e., if I now try to HTTP GET /api/hosts I'll get the same "unable to authenticate user " error - I would have to re-auth against /users/extlogin and get a new session_id.

Calling HTTP DELETE on /api/hosts DOES work if I pass "-u earsdown:<password>", ie basic auth with the credentials of a FreeIPA user...BUT it takes ages because I can see foreman is re-authenticating me against LDAP even though I'm supplying a valid session_id cookie. When passing "-u earsdown:<password>" (basic auth) for other HTTP GET APIs, it's quick, and it doesn't re-auth me against LDAP - which means it must be using the session_id cookie.

• Here's the output of what happens when calling HTTP DELETE on /api/hosts using basic auth creds (and a previously working session_id - which gets ignored) of a FreeIPA user:
  

  [root@puppet01 ~]# curl  -vv  -u earsdown:<password> -b /root/cookies.txt -c /root/cookies.txt -s -X DELETE $foreman_url/api/v2/hosts/15 | jq .
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  * Server auth using Basic with user 'earsdown'
  > DELETE /api/v2/hosts/15 HTTP/1.1
  > Authorization: Basic <FILTERED>
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  > Cookie: _session_id=7f0d4f8d07225ef1e0843f6419e12990
  >
  < HTTP/1.1 200 OK
  < Date: Mon, 07 Sep 2015 11:00:57 GMT
  < Server: Apache/2.4.6 (CentOS)
  < Foreman_version: 1.8.3
  < Foreman_api_version: 2
  < Apipie-Checksum: 4dc90ee161b2f69d513ec4065c3fe739
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: max-age=0, private, must-revalidate
  < X-Request-Id: bddf2c807912140b391694e55e74dd01
  < X-Runtime: 5.440107
  < X-Rack-Cache: invalidate, pass
  < X-Powered-By: Phusion Passenger 4.0.18
  * Replaced cookie request_method="DELETE" for domain puppet01.example.com, path /, expire 0
  < Set-Cookie: request_method=DELETE; path=/
  < ETag: "e95300684778ae24bef975e168f8756b" 
  < Status: 200 OK
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: application/json; charset=utf-8
  <
  { [data not shown]
  * Closing connection 0
  {
    "uuid": null,
    "use_image": null,
    "updated_at": "2015-09-04T10:41:02Z",
    "source_file_id": null,
    "serial": null,
    "root_pass": "",
    "realm_id": null,
    "puppet_status": 6,
    "puppet_proxy_id": 1,
    "installed_at": null,
    "image_id": null,
    "image_file": "",
    "id": 15,
    "hostgroup_id": 6,
    "grub_pass": "",
    "environment_id": 1,
    "enabled": true,
    "architecture_id": 1,
    "build": false,
    "certname": "proxyvpngw-i-cb50a014.example.com",
    "comment": null,
    "compute_profile_id": null,
    "compute_resource_id": null,
    "created_at": "2015-09-04T10:01:58Z",
    "disk": null,
    "last_compile": "2015-09-04T10:39:44Z",
    "last_freshcheck": null,
    "last_report": "2015-09-04T10:39:32Z",
    "location_id": 2,
    "managed": false,
    "medium_id": null,
    "model_id": 1,
    "name": "proxyvpngw-i-cb50a014.example.com",
    "operatingsystem_id": 1,
    "organization_id": 4,
    "otp": null,
    "owner_id": 5,
    "owner_type": "User",
    "provision_method": "build",
    "ptable_id": null,
    "puppet_ca_proxy_id": null
  }
  [root@puppet01 ~]# curl  -vv  -u earsdown:<password> -b /root/cookies.txt -c /root/cookies.txt -s -X DELETE $foreman_url/api/v2/hosts/15 | jq .
  * About to connect() to puppet01.example.com port 443 (#0)
  *   Trying 10.239.12.20...
  * Connected to puppet01.example.com (10.239.12.20) port 443 (#0)
  * Initializing NSS with certpath: sql:/etc/pki/nssdb
  *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  * NSS: client certificate not found (nickname not specified)
  * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  * Server certificate:
  *       subject: CN=puppet01.example.com,O=EXAMPLE.COM
  *       start date: Aug 19 12:19:46 2015 GMT
  *       expire date: Aug 19 12:19:46 2017 GMT
  *       common name: puppet01.example.com
  *       issuer: CN=Certificate Authority,O=EXAMPLE.COM
  * Server auth using Basic with user 'earsdown'
  > DELETE /api/v2/hosts/15 HTTP/1.1
  > Authorization: Basic <FILTERED>
  > User-Agent: curl/7.29.0
  > Host: puppet01.example.com
  > Accept: */*
  > Cookie: request_method=DELETE; _session_id=7f0d4f8d07225ef1e0843f6419e12990
  >
  < HTTP/1.1 404 Not Found
  < Date: Mon, 07 Sep 2015 11:01:21 GMT
  < Server: Apache/2.4.6 (CentOS)
  < Foreman_version: 1.8.3
  < Foreman_api_version: 2
  < Apipie-Checksum: 4dc90ee161b2f69d513ec4065c3fe739
  < X-UA-Compatible: IE=Edge,chrome=1
  < Cache-Control: no-cache
  < X-Request-Id: 91f18369a51c901f045ef64e2f981170
  < X-Runtime: 4.021407
  < X-Rack-Cache: invalidate, pass
  < X-Powered-By: Phusion Passenger 4.0.18
  * Replaced cookie request_method="DELETE" for domain puppet01.example.com, path /, expire 0
  < Set-Cookie: request_method=DELETE; path=/
  < Status: 404 Not Found
  < Connection: close
  < Transfer-Encoding: chunked
  < Content-Type: application/json; charset=utf-8
  <
  { [data not shown]
  * Closing connection 0
  {
    "error": {
      "message": "Resource host not found by id '15'" 
    }
  }
  

Btw, this behavior is reproducible on /api/v2 URLs as well.

My use-case is a host-retire script which cleans up terminated hosts in foreman (and freeipa, and puppetdb, and nagios, ...you get the idea) based on what's running in $cloud. The script will run via cron by unprivileged user account, and the only credential I want to give it is keytab linked to a FreeIPA user principal... thus... I'm keen to get this working :-)

Ideas?

Ears Down wrote:

Not sure if this is related, but I'm also struggling a bit with something that smells a lot like this topic. Setup is Freeipa 4.1 + Foreman 1.8.3 on Centos7 (fully patched).

It would be better to take this to a new ticket or the mailing list, as this is about adding support to Hammer.

• This time we're going to hit /users/extlogin with "--negotiate -u :", which will log us in and give us a session_id cookie.

I'd have thought you could use this against the API, but I haven't tested. (#11317 is about this, but I can't tell if it's PEBKAC or not working.)

• But when I try to delete a host (via DELETE /api/hosts/<hostid>), foreman freaks out and invalidates our credentials:
  [...]

Interestingly, it throws a 401 but doesn't offer a WWW-Authenticate header.

• ...and here's what appears in the production.log:
  [...]

Foreman does actually invalidate the previously-working credentials - i.e., if I now try to HTTP GET /api/hosts I'll get the same "unable to authenticate user " error - I would have to re-auth against /users/extlogin and get a new session_id.

Calling HTTP DELETE on /api/hosts DOES work if I pass "-u earsdown:<password>", ie basic auth with the credentials of a FreeIPA user...BUT it takes ages because I can see foreman is re-authenticating me against LDAP even though I'm supplying a valid session_id cookie. When passing "-u earsdown:<password>" (basic auth) for other HTTP GET APIs, it's quick, and it doesn't re-auth me against LDAP - which means it must be using the session_id cookie.

This behaviour is intended, implemented in #4895. A session ID is only meant to be used by a web browser with web-based forms and not really for scripted API use. The reason support for session IDs are there is to support web frameworks written against Foreman's API.

As such, the CSRF support is kicking in to prevent you making any changes to Foreman without knowing that it came from a trusted form, or authentication was explicitly passed - else a browser with an active session could be tricked into making changes in Foreman.

The solution is probably just to ensure GSSAPI etc works against the API without use of session IDs etc.