Foreman » Smart Proxy

Sinatra v. 1.3.5 is fairly old and is going to have issues with rack versions below 1.6.0. Specifically, your issue is due to incompatibility introduced in rack here: https://github.com/rack/rack/commit/893a2c505a3a43b4f224b50ea06567dacb473cae, which was fixed in Sinatra v. 1.4.6 and later (https://github.com/sinatra/sinatra/commit/b39d72d2a5b824b5f607b7825c4b076f527461a1).

Guessing you're using EL7 from the stacktrace. If so, an updated version of rubygem-sintra is available at http://koji.katello.org/koji/taskinfo?taskID=587904 which will be in a future release.

It appears likely to me that this is just error handling, so the underlying error should be correctly shown after updating. My previous questions still stand.

Dominic Cleal wrote:

Can you provide the context of the error in the log, the request being made? Enabling debug logging (log_level, /etc/foreman-proxy/settings.yml) would be helpful.

Exactly which smart-proxy plugins are installed, if any? Is smart_proxy_chef installed?

I had

Templates, TFTP, DHCP, Puppet, BMC, Discovery, Dynflow, and SSH

removing the SSH remote execution it fixed the problem. No Chef plugin installed

And yes, I'm using EL7 (specifically CentOS 7.3). Debug was already enabled but didn't give any useful log

As I said any type of request is failing, even searching the catalog. I can try to update Sinatra though

Well, I tried to update sinatra via gem update sinatra to 1.4.7 but it turns out that the sinatra gem installed is from the Foreman repo. So, what should I do? Anyway after updating sinatra to 1.4.7 (and having so two versions listed by gem list) the problem was still present but this time it doesn't log anything to the proxy log, even with debug logging enabled.

Looks like Foreman Discovery Image 3.3 is suffering from this as well, when error occurs, this is the only message in logs. We are using 1.14 proxy repository.

https://github.com/theforeman/foreman-discovery-image/pull/82

An updated version of Sinatra is available from the link in comment 3, please use that instead of mixing packages and local gems.

the problem was still present but this time it doesn't log anything to the proxy log, even with debug logging enabled.

What problem precisely? If requests are failing, please provide the response body/headers/status information.

In addition to what Dominic said: All foreman repositories (at this time) contain a package with an old version of sinatra. The bottom line: If you are getting the stacktrace above, your combination of rack and sinatra are incompatible.

I think there is another issue here, as sinatra is trying to handle an exception according to the stacktrace above. We'll need smart-proxy logs to troubleshoot this (unless Lukas who appears to be able to reproduce the issue is going to look into this).

It's easy to reproduce in FDI, but the environment is read-only and very unfriendly to any debugging sessions.

One observation, the request which is erroring out is spawning new thread, I don't known if the exception is thrown in the thread or not, but REX might also use threads and this could be the similarity.

Can't we just upgrade Sinatra/Rack in our nightly repos and backport the fix into 1.14? From the links it looks like this is the fix that should do it: https://github.com/sinatra/sinatra/commit/36e5b26d35da68a8dd67951e27b424e5b60099f8

Ok, I think that I overlooked an error message on foreman-proxy startup, related to the remote execution plugin:

E, [2017-01-17T12:57:37.548051 #1126] ERROR -- : Couldn't enable 'ssh': Ssh public key file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy doesn't exist.
You can generate one with `ssh-keygen -t rsa -b 4096 -f /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy -N ''`
D, [2017-01-17T12:57:37.548114 #1126] DEBUG -- : ["/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.1.4/lib/smart_proxy_remote_execution_ssh.rb:14:in `validate!'", "/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.1.4/lib/smart_proxy_remote_execution_ssh/plugin.rb:27:in `block in <class:Plugin>'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:200:in `class_eval'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:200:in `configure_plugin'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `block in configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `block in initialize_plugins'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `initialize_plugins'", "/usr/share/foreman-proxy/lib/launcher.rb:134:in `launch'", "/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'"]

I've created the ssh key and the original problem is fixed. Now foreman-proxy with the remote_execution plugin is usable again. Sorry for didn't catching it before and creating all this noise; I wonder otherwise if the plugin should just disable itself in this error condition instead of bringing all the proxy down.

Ok, I will rebuild the image on 1.14.1 perhaps to avoid error hiding. Thanks.