Foreman » Smart Proxy

I tried to use https://github.com/shekyan/slowhttptest which is in EPEL7 and it looks like for the basic (slow headers GET) attach our proxy on my system (VM on Ryzen1700 CPU, 14 GB RAM, 4 cores) is able to handle about 200 opened connections and then it starts refusing them, maximum amount of opened connection peaks at 320: slowhttptest -u http://127.0.0.1:8000/features -c 1000 -g -o slow-headers-1000

During the test the service is randomly not available, but once the test is stopped, it is still able to recover and process requests at normal rate which is on my system about 500 per second: ab -c 10 -t 1 http://127.0.0.1:8000/features (this is Apache ab testing utility, present in RHEL7).

So the key question is what do reporter see, if it's the service not being available during slow DoS attack, then this is normal. Every HTTP server has a top number of opened requests it can process before it starts refusing connections. Ability to handle about 200 requests at the same time is enough for foreman-proxy service which was built to serve requests from one foreman server which is typically 1-50 passenger processes (number of CPU cores, each doing typically few concurrent requests), so we are talking about scaling to several dozens of concurrent requests. We are hitting limitation of the Ruby/Rack/Sinatra stack which is one of the slower ones compared to Golang/Java/JavaScript.

Does puma-3.11.4 (last version running on <=2.0) suffer the same issue? Couldn't we get away by swapping webrick for puma?

Puma does seem to close inactive connections after 30 seconds. Since puma has zero gem dependencies just few basic C deps this is doable. Let's add support into launcher.rb for puma and keep webrick as opt-in just in case this turns to be problematic in production as a good fallback mechanism.

https://community.theforeman.org/t/rfc-add-puma-as-the-default-smart-proxy-server/10975

Ido, I started dicussion on the community list and it looks like you have assigned yourself. Do you want to work on this? It's a nice feature to work on, but keep in mind this must be tested in production setup (RPM, daemon started from systemd, Red Hat, Debian and Windows).

Lukas Zapletal wrote:

Ido, I started dicussion on the community list and it looks like you have assigned yourself. Do you want to work on this? It's a nice feature to work on, but keep in mind this must be tested in production setup (RPM, daemon started from systemd, Red Hat, Debian and Windows).

Sure I want to work on this :)
I don't know yet how to test it though, might require some help from you on that to understand the basics of how to setup such environments and then can go from there :)