Massive db deadlocks in postgres from hosts_counter updates with counter_cache_fix.rb
After updating the Foreman 1.8 rc3 I am seeing a massive amount of DB Deadlocks. This issue was not present in 1.7
#2 Updated by Ohad Levy over 5 years ago
quick google shows a few things:
#4 Updated by Tomer Brisker over 5 years ago
- Category set to Database
Which PostgreSQL version are you using?
This sounds like it might be related to a problem that was fixed in 9.3: http://mina.naguib.ca/blog/2010/11/22/postgresql-foreign-key-deadlocks.html
#5 Updated by Lukas Zapletal over 5 years ago
There are three users in the comments complaining that 9.3 version is even worse and it was not fixed for them :-(
Alvaro Herrera describes the solution in introducing new keyword SELECT ... FOR KEY. That would mean you need both new PostgreSQL 9.3 and newer Rails which takes advantage of that approach? Or some change in Foreman would be required I assume.
#10 Updated by Chuck Schweizer over 5 years ago
My environment is a fully updated RHEL 6 install using the foreman installer.
foreman 1.8 rc3
The foreman server is only setup to receive reports and facts from the puppet masters, it is not acting as a puppet server or external node configurator.
From what I can tell the uploading of reports and facts from the 40K nodes, through the puppet masters, is causing the deadlocks. Commenting out the logic that updates the DB in counter_cache_fix.rb made the deadlocks stop.
The only thing that was change going from foreman 1.7.1 to 1.8 rc3 was foreman. Nothing else on the system was updated or changed. The foreman install was run after installing the 1.8 rc3 rpms.
#13 Updated by Chuck Schweizer about 5 years ago
After reducing the number of puppet masters in my environment I have been unable to reproduce the issue.
Daniel Lobato Garcia wrote:
Tomer has prepared a proposed fix for this issue - https://github.com/theforeman/foreman/pull/2362
Could you report if it works for your case?