Few extra notes: On 8/25 we upgraded from 3.9 to 3.16. Puma was set to 2 workers; Min 0 and Max 8. We had tasks running long so we updated PUMA on 10/23 to 8 workers; min 8 and max 32. That is when we experienced and exploited the memory leak.

Oh I see this instance has 64GB, so when you had 8 workers you saw 40% utilization, so that is 25GB (= 3 GB per worker). How many plugins do you have? We know that clean Foreman instance takes 300MB and grows up to 500 for core, 900 with Katello plugin and so on and so on.

In your case, to hold 32 workers 3GB each you will need 96GB RAM. I suggest you lower the number of workers.

I don't believe this is a leak per se, however we obviously do have some memory hogs in the codebase. I'd say many. To fix that, file more concrete report.

I see in the top output up to 9GB memory RSS consuption per worker. That's too much, yeah. Are you able to isolate what operation makes it grow? To identify code paths which needs investigation.

That was useful, most of the time spent are facts, which is expected. We plan on improving how we store them, not all of them need to be stored in normal form.

However what I see as more suspicious are reports generation, there were four requests on report_template generate which in total took 30% of all time spent. Can you make sure nobody clicks/starts on report generation, restart the workers and investigate if memory does not spike that hard? Reports are new feature, there could be a memory hog.

To make less stress on the db, I suggest you define some facts to be filtered out. We currently store all of them, if you have some hypervisors or container hosts with many interfaces, disks etc theye can heavily hit our fact tables - try to filter these facts completely (Administer - Settings - Facts exclude).

Get back to us at https://community.theforeman.org/t/possible-memory-leak-in-reports/21178 please.