Project

General

Profile

Actions

Bug #20929

closed

Huge table "fact_names" makes Foreman crawl

Added by Justin Zandbergen about 7 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Database
Target version:
Fixed in Releases:
Found in Releases:

Description

OS: CentOS 7.3
Foreman: 1.15.3
salt-master: 2017.7.1 (Nitrogen)
salt smart proxy: 2.1.8

Scenario:
We have a few docker hosts that spawn lots and lots of short lived docker containers. Each docker container create a veth<random string> interface on linux.
Everytime when "salt '*' state.highstate test.true" runs from cron all these interfaces are added to the "fact_names" and "fact_values" tables in the foreman postgres DB.
Eventually these tables get bloated with thousands and thousands of interfaces that are long gone. (at one point >500.000 network interfaces).
This will result in the behavior that the command "/usr/bin/foreman-node <some-node>" will take >30 seconds to return with data, which in it's turn will cause to lock-up all the salt-master's worker threads which in it's turn will cause foreman to not function.

The query which is responsible for this long delay is:

SELECT MAX("fact_names"."id") AS maximum_id, "fact_names"."name" AS fact_names_name FROM "fact_names" WHERE "fact_names"."type" IN ('ForemanSalt::FactName') GROUP BY "fact_names"."name"  ORDER BY fact_names.name;

Workaround:
Now i regulary clear the unnessecary data from the "fact_names" and "fact_values" with the queries:

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::veth%'
        )
;

delete from fact_names where name like '%interfaces::veth%';

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::br-%'
        )
;

delete from fact_names where name like '%interfaces::br-%';

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::docker%'
        )
;

delete from fact_names where name like '%interfaces::docker%';

What I think what should happen:
I think that Foreman should not save those interfaces in "fact_names" / "fact_values" according to the setting:

"Administer -> Settings -> Provisioning -> 'Ignore interfaces with matching identifier'".

This might be related to bug #20927 however the patch mentioned in that issue does not resolve the issue for the tables "fact_names" / "fact_values".


Related issues 2 (1 open1 closed)

Related to Installer - Bug #19081: node.rb failing with servers with large amount of facts (ips?)New03/29/2017Actions
Related to Katello - Bug #22894: Make RHSM facts compatible with fact name filteringClosedShimon Shtein03/14/2018Actions
Actions #1

Updated by The Foreman Bot about 7 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Shimon Shtein
  • Pull request https://github.com/theforeman/foreman/pull/5021 added
Actions #2

Updated by Ewoud Kohl van Wijngaarden almost 7 years ago

  • Related to Bug #19081: node.rb failing with servers with large amount of facts (ips?) added
Actions #3

Updated by Shimon Shtein almost 7 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #4

Updated by Lukas Zapletal almost 7 years ago

  • Translation missing: en.field_release set to 330
Actions #5

Updated by Ohad Levy almost 7 years ago

  • Translation missing: en.field_release changed from 330 to 296

1.17 wasn't branched yet, so this is part of 1.17 then..

Actions #6

Updated by The Foreman Bot almost 7 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5142 added
Actions #7

Updated by Chris Duryee over 6 years ago

  • Bugzilla link set to 1553263
Actions #8

Updated by Andrew Kofink over 6 years ago

  • Related to Bug #22894: Make RHSM facts compatible with fact name filtering added
Actions

Also available in: Atom PDF