Bug #18179
closedsmart proxy refresh throws stackerror
Description
Hi,
I recently upgraded my main foreman/katello server to 3.2.3. Afterwards I reinstalled my secondary puppet smartproxy. Now everytime I refresh the smart proxy there is an error "SystemStackError (stack level too deep)". I have other smart proxies that work fine. Even though it throws an error the refresh seems to work because I can turn off/on features.
CLI:
# hammer proxy refresh-features --name smartproxypuppet.tld Refresh of smart proxy features failed: 500 Internal Server Error
Production Log:
2017-01-20 16:13:21 7344a677 [app] [I] Started GET "/api/smart_proxies?search=name+%3D+%22smartproxypuppet.tld%22" for 1.1.1.20 at 2017-01-20 16:13:21 +0100 2017-01-20 16:13:21 7344a677 [app] [I] Processing by Api::V2::SmartProxiesController#index as JSON 2017-01-20 16:13:21 7344a677 [app] [I] Parameters: {"search"=>"name = \"smartproxypuppet.tld\"", "apiv"=>"v2", "smart_proxy"=>{}} 2017-01-20 16:13:21 7344a677 [app] [I] Authorized user admin(Admin User) 2017-01-20 16:13:21 7344a677 [app] [I] Rendered api/v2/smart_proxies/index.json.rabl within api/v2/layouts/index_layout (44.5ms) 2017-01-20 16:13:21 7344a677 [app] [I] Completed 200 OK in 113ms (Views: 60.4ms | ActiveRecord: 12.3ms) 2017-01-20 16:13:22 713e4c44 [app] [I] Started PUT "/api/smart_proxies/2/refresh" for 1.1.1.20 at 2017-01-20 16:13:22 +0100 2017-01-20 16:13:22 713e4c44 [app] [I] Processing by Api::V2::SmartProxiesController#refresh as JSON 2017-01-20 16:13:22 713e4c44 [app] [I] Parameters: {"apiv"=>"v2", "id"=>"2", "smart_proxy"=>{}} 2017-01-20 16:13:22 713e4c44 [app] [I] Authorized user admin(Admin User) 2017-01-20 16:14:00 713e4c44 [app] [I] Completed 500 Internal Server Error in 38278ms (ActiveRecord: 1116.1ms) 2017-01-20 16:14:00 713e4c44 [app] [F] | SystemStackError (stack level too deep): | app/models/concerns/audit_extensions.rb:56:in `ensure_auditable_and_associated_name' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' [...many many many more...] | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:51:in `puppet_path' | katello (3.2.1.1) app/lib/katello/lazy_accessor.rb:80:in `save' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:60:in `update_puppet_path' | katello (3.2.1.1) app/models/katello/concerns/smart_proxy_extensions.rb:66:in `refresh_with_puppet_path' | foreman_remote_execution (1.2.2) app/models/concerns/foreman_remote_execution/smart_proxy_extensions.rb:21:in `refresh_with_remote_execution' | app/controllers/api/v2/smart_proxies_controller.rb:60:in `refresh' | app/controllers/api/v2/base_controller.rb:152:in `disable_json_root' | app/controllers/concerns/application_shared.rb:14:in `set_timezone' | app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' | lib/middleware/catch_json_parse_errors.rb:9:in `call' | lib/middleware/tagged_logging.rb:18:in `call'
Foreman Installer for Capsule:
foreman-installer --scenario capsule\ --capsule-parent-fqdn "foreman.tld"\ --foreman-proxy-register-in-foreman "true"\ --foreman-proxy-foreman-base-url "https://foreman.tld"\ --foreman-proxy-trusted-hosts "foreman.tld"\ --foreman-proxy-trusted-hosts "smartproxypuppet.tld"\ --foreman-proxy-oauth-consumer-key "KEY"\ --foreman-proxy-oauth-consumer-secret "KEY"\ --capsule-pulp-oauth-secret "KEY"\ --capsule-certs-tar "~/smartproxypuppet.tld-certs.tar"\ --capsule-puppet-server-implementation puppetserver\ --capsule-puppet-ca-proxy "https://foreman.tld"\ --foreman-proxy-puppetca false\ --foreman-proxy-templates false\ --foreman-proxy-plugin-remote-execution-ssh-enabled false\ --no-enable-foreman-proxy-plugin-pulp\ --foreman-proxy-plugin-pulp-enabled false
I need to disable pulp in /etc/foreman-proxy/settings.d/pulp.yml afterwards and install httpd before running installer but for that I'll open a separate bug :)
Greetings
Klaas
Updated by Anonymous about 8 years ago
Could you post smart-proxy log for when this failure happens? It's not clear from Foreman's log where the failure is occurring.
Updated by Klaas D about 8 years ago
Hi,
on the puppet smart proxy there are logs like this:
I, [2017-01-20T17:15:19.681161 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:15:19 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:15:19.766234 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:15:19 +0100] "GET /features HTTP/1.1" 200 10 0.0092 I, [2017-01-20T17:15:19.851935 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:15:19 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:15:19.932268 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:15:19 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:15:20.014565 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:15:20 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.056454 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.126090 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:16:45.220527 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.309505 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:16:45.400620 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.485953 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0003 I, [2017-01-20T17:16:45.575194 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.667581 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005 I, [2017-01-20T17:16:45.750023 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:16:45.835060 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0004 I, [2017-01-20T17:16:45.907984 #50530] INFO -- : 1.1.1.20 - - [20/Jan/2017:17:16:45 +0100] "GET /features HTTP/1.1" 200 10 0.0005
(lots of them) no errors
the proxy log on the foreman server is not showing anything while this request is happening
the request itsself seems to work:
# curl -vks --cacert /etc/foreman-proxy/foreman_ssl_ca.pem --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem "https://smartproxypuppet.tld:9090/features" * About to connect() to smartproxypuppet.tld port 9090 (#0) * Trying 10.165.1.22... * Connected to smartproxypuppet.tld (10.165.1.22) port 9090 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * NSS: client certificate from file * subject: CN=foreman.tld,OU=FOREMAN_PROXY,O=FOREMAN,ST=State,C=TLD * start date: Jun 15 15:58:33 2016 GMT * expire date: Jun 17 15:58:33 2036 GMT * common name: foreman.tld * issuer: CN=foreman.tld,OU=Name,O=Name,L=Munich,ST=State,C=TLD * SSL connection using TLS_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=smartproxypuppet.tld,OU=SMART_PROXY,O=FOREMAN,ST=North Carolina,C=US * start date: Jun 24 10:23:07 2016 GMT * expire date: Jun 26 10:23:07 2036 GMT * common name: smartproxypuppet.tld * issuer: CN=foreman.tld,OU=Name,O=Name,L=Munich,ST=State,C=TLD > GET /features HTTP/1.1 > User-Agent: curl/7.29.0 > Host: smartproxypuppet.tld:9090 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json;charset=utf-8 < Content-Length: 10 < X-Content-Type-Options: nosniff < Server: WEBrick/1.3.1 (Ruby/2.0.0/2015-12-16) OpenSSL/1.0.1e < Date: Fri, 20 Jan 2017 16:19:15 GMT < Connection: Keep-Alive < * Connection #0 to host smartproxypuppet.tld left intact ["puppet"]
Updated by Anonymous about 8 years ago
- Status changed from New to Rejected
Looks like a bug in Katello here: https://github.com/Katello/katello/blob/KATELLO-3.2/app/models/katello/concerns/smart_proxy_extensions.rb#L51. If none of pulp-related features aren't available on smart-proxy puppet_path will end up recursively calling itself. I don't see a way to move the bug to Katello directly, I'll close it here and open a new one in Katello bug tracker.
Updated by Anonymous about 8 years ago
- Related to Bug #18185: smart proxy refresh throws stackerror added
Updated by Anonymous about 8 years ago
FYI: The issue filed under Katello project is in "Related Issues" above.