Project

General

Profile

Bug #20929

Huge table "fact_names" makes Foreman crawl

Added by Justin Zandbergen about 3 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Database
Target version:
Difficulty:
Triaged:
Bugzilla link:

Description

OS: CentOS 7.3
Foreman: 1.15.3
salt-master: 2017.7.1 (Nitrogen)
salt smart proxy: 2.1.8

Scenario:
We have a few docker hosts that spawn lots and lots of short lived docker containers. Each docker container create a veth<random string> interface on linux.
Everytime when "salt '*' state.highstate test.true" runs from cron all these interfaces are added to the "fact_names" and "fact_values" tables in the foreman postgres DB.
Eventually these tables get bloated with thousands and thousands of interfaces that are long gone. (at one point >500.000 network interfaces).
This will result in the behavior that the command "/usr/bin/foreman-node <some-node>" will take >30 seconds to return with data, which in it's turn will cause to lock-up all the salt-master's worker threads which in it's turn will cause foreman to not function.

The query which is responsible for this long delay is:

SELECT MAX("fact_names"."id") AS maximum_id, "fact_names"."name" AS fact_names_name FROM "fact_names" WHERE "fact_names"."type" IN ('ForemanSalt::FactName') GROUP BY "fact_names"."name"  ORDER BY fact_names.name;

Workaround:
Now i regulary clear the unnessecary data from the "fact_names" and "fact_values" with the queries:

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::veth%'
        )
;

delete from fact_names where name like '%interfaces::veth%';

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::br-%'
        )
;

delete from fact_names where name like '%interfaces::br-%';

delete from fact_values
where fact_name_id in (
        select id from fact_names 
        where name like '%interfaces::docker%'
        )
;

delete from fact_names where name like '%interfaces::docker%';

What I think what should happen:
I think that Foreman should not save those interfaces in "fact_names" / "fact_values" according to the setting:

"Administer -> Settings -> Provisioning -> 'Ignore interfaces with matching identifier'".

This might be related to bug #20927 however the patch mentioned in that issue does not resolve the issue for the tables "fact_names" / "fact_values".


Related issues

Related to Installer - Bug #19081: node.rb failing with servers with large amount of facts (ips?)New2017-03-29
Related to Katello - Bug #22894: Make RHSM facts compatible with fact name filteringClosed2018-03-14

Associated revisions

Revision 61b35ac1 (diff)
Added by Shimon Shtein almost 3 years ago

Fixes #20929 - Added fact name filtering on import

Added a setting that will filter out fact names, so they won't be
recorded into `fact_names` and `fact_values` tables at all.

Revision c74dd204 (diff)
Added by Timo Goebel almost 3 years ago

refs #20929 - do not stub settings

History

#1 Updated by The Foreman Bot almost 3 years ago

  • Assignee set to Shimon Shtein
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/5021 added

#2 Updated by Ewoud Kohl van Wijngaarden almost 3 years ago

  • Related to Bug #19081: node.rb failing with servers with large amount of facts (ips?) added

#3 Updated by Shimon Shtein almost 3 years ago

  • % Done changed from 0 to 100
  • Status changed from Ready For Testing to Closed

#4 Updated by Lukas Zapletal almost 3 years ago

  • Legacy Backlogs Release (now unused) set to 330

#5 Updated by Ohad Levy almost 3 years ago

  • Legacy Backlogs Release (now unused) changed from 330 to 296

1.17 wasn't branched yet, so this is part of 1.17 then..

#6 Updated by The Foreman Bot almost 3 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5142 added

#7 Updated by Chris Duryee over 2 years ago

  • Bugzilla link set to 1553263

#8 Updated by Andrew Kofink over 2 years ago

  • Related to Bug #22894: Make RHSM facts compatible with fact name filtering added

Also available in: Atom PDF