I have many servers/smart proxies in my foreman environment. All Told, there are
-12 "UI Only" type servers with nothing but foreman installed
-18 "puppet masters" with the foreman UI installed for ENC lookups
-6 additional smart proxy servers with various roles (dns/dhcp/etc)

Foreman releases a "dot release" roughly once per month, and a major release every 3-4. fairly rapid release cycle - which is GREAT from a new features, bugfix/etc. perspective

When i do my upgrades - I follow the upgrade instructions (RHEL7 RPM based in my case). Generally speaking, it just takes forever....And I feel it's probably an area where improvements can be made.

I pick a UI server and do it "first" the database and "apipie cache" portions can take 45 to 60 minutes (Mysql 5.5 or MariaDB 10.1 for reference)

Later in the step(s) it has you manually do a DB seed, which was already done by the RPM process - but might need to be done again for safety. Not quite as long - but still 20 minutes or so...

Last step (for this first node) is to re-run foreman-installer, which again if I'm watching the logs does another db migrate db seed apipie cache combination, another 20-30 minutes.

This ends up being (assuming none fail) three full db migrate -> db seed -> apipie cache generation(s) per server, for every upgrade undertaken.

And this is just for the first node. All other node(s) have the same rough time-frames in general. In our case - with the amount of nodes we have the upgrade process can easily take 12 hour(s) even if i'm doing multiple nodes at a time - which by the end I am - just to finish it up :).

I don't believe im fully in-tune with the upgrade process in detail but, i think some things can be done to speed it up a bit:
1. have db migrate and db seed record completion(s) somewhere in the database. that way if it completes successfully, all future run(s) are "skipped" if you will. This alone will shave a ton of time off of multi-server upgrades and installs. Provide a "force" type switch to foreman-rake to force a full rerun if needed. This leaves the outlying issue of "how to kick off on plugin install, or other scenarios where it's needed - but im sure there are ways to handle this?

2. When doing an upgrade/install - anytime i rerun foreman-installer on a smart proxy it "errors" because the proxy is already registered and working. Add in some kind of check to suppress this "false positive" error within foreman installer, so that an upgrade returns "success" since its already registered. Fixing this would allow me to better "orchestrate" the installation across our multiple nodes -right now it's difficult due to the false-positive showing as failure to the orchestration.