Bug #18092

foreman-proxy throws an undefined method 'join'

Added by Davide Ferrari 6 months ago. Updated 6 months ago.

Status:Closed
Priority:Normal
Assigned To:Dominic Cleal
Category:Packaging
Target version:-
Difficulty: Bugzilla link:
Found in release: Pull request:
Story points-
Velocity based estimate-
Release1.14.1Release relationshipAuto

Description

I get this with every request, making the proxy unusable

E, [2017-01-16T12:15:59.678835 #32142] ERROR -- : undefined method `join' for #<String:0x007f1a587784a8> (NoMethodError)
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/showexceptions.rb:37:in `rescue in call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/showexceptions.rb:21:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:124:in `call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1417:in `block in call'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1499:in `synchronize'
/usr/share/gems/gems/sinatra-1.3.5/lib/sinatra/base.rb:1417:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/usr/share/gems/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/builder.rb:153:in `call'
/usr/share/gems/gems/rack-1.6.4/lib/rack/handler/webrick.rb:88:in `service'
/usr/share/ruby/webrick/httpserver.rb:138:in `service'
/usr/share/ruby/webrick/httpserver.rb:94:in `run'
/usr/share/ruby/webrick/server.rb:295:in `block in start_thread'


Related issues

Related to Chef - Bug #18087: Error undefined method `join' for #<String:0x007f1f13877b... Closed 01/16/2017

History

#1 Updated by Dominic Cleal 6 months ago

  • Project changed from Foreman to Smart Proxy
  • Status changed from New to Need more information

Can you provide the context of the error in the log, the request being made? Enabling debug logging (log_level, /etc/foreman-proxy/settings.yml) would be helpful.

Exactly which smart-proxy plugins are installed, if any? Is smart_proxy_chef installed?

#2 Updated by Dmitri Dolguikh 6 months ago

Sinatra v. 1.3.5 is fairly old and is going to have issues with rack versions below 1.6.0. Specifically, your issue is due to incompatibility introduced in rack here: https://github.com/rack/rack/commit/893a2c505a3a43b4f224b50ea06567dacb473cae, which was fixed in Sinatra v. 1.4.6 and later (https://github.com/sinatra/sinatra/commit/b39d72d2a5b824b5f607b7825c4b076f527461a1).

#3 Updated by Dominic Cleal 6 months ago

Guessing you're using EL7 from the stacktrace. If so, an updated version of rubygem-sintra is available at http://koji.katello.org/koji/taskinfo?taskID=587904 which will be in a future release.

It appears likely to me that this is just error handling, so the underlying error should be correctly shown after updating. My previous questions still stand.

#4 Updated by Dominic Cleal 6 months ago

  • Related to Bug #18087: Error undefined method `join' for #<String:0x007f1f13877b00> when uploading facts added

#5 Updated by Davide Ferrari 6 months ago

Dominic Cleal wrote:

Can you provide the context of the error in the log, the request being made? Enabling debug logging (log_level, /etc/foreman-proxy/settings.yml) would be helpful.

Exactly which smart-proxy plugins are installed, if any? Is smart_proxy_chef installed?

I had

Templates, TFTP, DHCP, Puppet, BMC, Discovery, Dynflow, and SSH

removing the SSH remote execution it fixed the problem. No Chef plugin installed

#6 Updated by Davide Ferrari 6 months ago

And yes, I'm using EL7 (specifically CentOS 7.3). Debug was already enabled but didn't give any useful log

#7 Updated by Dominic Cleal 6 months ago

  • Project changed from Smart Proxy to Foreman Remote Execution

Moving in case it's related to that plugin then, but without either logs showing what requests it's failing on or logs with a newer Sinatra version showing a different error, it's unlikely that the bug can be identified or fixed.

#8 Updated by Davide Ferrari 6 months ago

As I said any type of request is failing, even searching the catalog. I can try to update Sinatra though

#9 Updated by Davide Ferrari 6 months ago

Well, I tried to update sinatra via gem update sinatra to 1.4.7 but it turns out that the sinatra gem installed is from the Foreman repo. So, what should I do? Anyway after updating sinatra to 1.4.7 (and having so two versions listed by gem list) the problem was still present but this time it doesn't log anything to the proxy log, even with debug logging enabled.

#10 Updated by Lukas Zapletal 6 months ago

Looks like Foreman Discovery Image 3.3 is suffering from this as well, when error occurs, this is the only message in logs. We are using 1.14 proxy repository.

https://github.com/theforeman/foreman-discovery-image/pull/82

#11 Updated by Lukas Zapletal 6 months ago

  • Project changed from Foreman Remote Execution to Smart Proxy

For the record, we are just using Smart Proxy core in discovery, this is not REX issue. Let's keep it in core for now.

#12 Updated by Dominic Cleal 6 months ago

An updated version of Sinatra is available from the link in comment 3, please use that instead of mixing packages and local gems.

the problem was still present but this time it doesn't log anything to the proxy log, even with debug logging enabled.

What problem precisely? If requests are failing, please provide the response body/headers/status information.

#13 Updated by Dmitri Dolguikh 6 months ago

In addition to what Dominic said: All foreman repositories (at this time) contain a package with an old version of sinatra. The bottom line: If you are getting the stacktrace above, your combination of rack and sinatra are incompatible.

I think there is another issue here, as sinatra is trying to handle an exception according to the stacktrace above. We'll need smart-proxy logs to troubleshoot this (unless Lukas who appears to be able to reproduce the issue is going to look into this).

#14 Updated by Lukas Zapletal 6 months ago

It's easy to reproduce in FDI, but the environment is read-only and very unfriendly to any debugging sessions.

One observation, the request which is erroring out is spawning new thread, I don't known if the exception is thrown in the thread or not, but REX might also use threads and this could be the similarity.

Can't we just upgrade Sinatra/Rack in our nightly repos and backport the fix into 1.14? From the links it looks like this is the fix that should do it: https://github.com/sinatra/sinatra/commit/36e5b26d35da68a8dd67951e27b424e5b60099f8

#15 Updated by Davide Ferrari 6 months ago

Ok, I think that I overlooked an error message on foreman-proxy startup, related to the remote execution plugin:

E, [2017-01-17T12:57:37.548051 #1126] ERROR -- : Couldn't enable 'ssh': Ssh public key file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy doesn't exist.
You can generate one with `ssh-keygen -t rsa -b 4096 -f /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy -N ''`
D, [2017-01-17T12:57:37.548114 #1126] DEBUG -- : ["/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.1.4/lib/smart_proxy_remote_execution_ssh.rb:14:in `validate!'", "/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.1.4/lib/smart_proxy_remote_execution_ssh/plugin.rb:27:in `block in <class:Plugin>'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:200:in `class_eval'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:200:in `configure_plugin'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `block in configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:89:in `configure'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `block in initialize_plugins'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `each'", "/usr/share/foreman-proxy/lib/proxy/plugin_initializer.rb:160:in `initialize_plugins'", "/usr/share/foreman-proxy/lib/launcher.rb:134:in `launch'", "/usr/share/foreman-proxy/bin/smart-proxy:6:in `<main>'"]

I've created the ssh key and the original problem is fixed. Now foreman-proxy with the remote_execution plugin is usable again. Sorry for didn't catching it before and creating all this noise; I wonder otherwise if the plugin should just disable itself in this error condition instead of bringing all the proxy down.

#16 Updated by Dominic Cleal 6 months ago

  • Subject changed from foreman-proxy throws and undefined method 'join' to foreman-proxy throws an undefined method 'join'
  • Category set to Packaging
  • Status changed from Need more information to Closed
  • Assigned To set to Dominic Cleal
  • Release set to 1.14.1

Lukas Zapletal wrote:

Can't we just upgrade Sinatra/Rack in our nightly repos and backport the fix into 1.14? From the links it looks like this is the fix that should do it: https://github.com/sinatra/sinatra/commit/36e5b26d35da68a8dd67951e27b424e5b60099f8

Yes, this is already done (see comment 3) and will be in 1.14.1. This is only a fix for the exception handler, there is still an exception or underlying issues (for example, comment 15) that triggers the error.

Thanks for the update Davide, I'll close this issue as resolved now as it only appears to be a packaging issue. I'd suggest filing a ticket against the plugin with some more logs and examples of what isn't working if it causes requests outside of that module to fail.

Lukas, I expect you're hitting a different issue in a plugin you have loaded. The updated RPM from comment 3 may help.

#17 Updated by Lukas Zapletal 6 months ago

Ok, I will rebuild the image on 1.14.1 perhaps to avoid error hiding. Thanks.

Also available in: Atom PDF