Bug #20929
closedHuge table "fact_names" makes Foreman crawl
Description
OS: CentOS 7.3
Foreman: 1.15.3
salt-master: 2017.7.1 (Nitrogen)
salt smart proxy: 2.1.8
Scenario:
We have a few docker hosts that spawn lots and lots of short lived docker containers. Each docker container create a veth<random string> interface on linux.
Everytime when "salt '*' state.highstate test.true" runs from cron all these interfaces are added to the "fact_names" and "fact_values" tables in the foreman postgres DB.
Eventually these tables get bloated with thousands and thousands of interfaces that are long gone. (at one point >500.000 network interfaces).
This will result in the behavior that the command "/usr/bin/foreman-node <some-node>" will take >30 seconds to return with data, which in it's turn will cause to lock-up all the salt-master's worker threads which in it's turn will cause foreman to not function.
The query which is responsible for this long delay is:
SELECT MAX("fact_names"."id") AS maximum_id, "fact_names"."name" AS fact_names_name FROM "fact_names" WHERE "fact_names"."type" IN ('ForemanSalt::FactName') GROUP BY "fact_names"."name" ORDER BY fact_names.name;
Workaround:
Now i regulary clear the unnessecary data from the "fact_names" and "fact_values" with the queries:
delete from fact_values where fact_name_id in ( select id from fact_names where name like '%interfaces::veth%' ) ; delete from fact_names where name like '%interfaces::veth%'; delete from fact_values where fact_name_id in ( select id from fact_names where name like '%interfaces::br-%' ) ; delete from fact_names where name like '%interfaces::br-%'; delete from fact_values where fact_name_id in ( select id from fact_names where name like '%interfaces::docker%' ) ; delete from fact_names where name like '%interfaces::docker%';
What I think what should happen:
I think that Foreman should not save those interfaces in "fact_names" / "fact_values" according to the setting:
"Administer -> Settings -> Provisioning -> 'Ignore interfaces with matching identifier'".
This might be related to bug #20927 however the patch mentioned in that issue does not resolve the issue for the tables "fact_names" / "fact_values".
Updated by The Foreman Bot about 7 years ago
- Status changed from New to Ready For Testing
- Assignee set to Shimon Shtein
- Pull request https://github.com/theforeman/foreman/pull/5021 added
Updated by Ewoud Kohl van Wijngaarden almost 7 years ago
- Related to Bug #19081: node.rb failing with servers with large amount of facts (ips?) added
Updated by Shimon Shtein almost 7 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 61b35ac1c5aa4809cd6f72a367e2020e6113d43b.
Updated by Lukas Zapletal almost 7 years ago
- Translation missing: en.field_release set to 330
Updated by Ohad Levy almost 7 years ago
- Translation missing: en.field_release changed from 330 to 296
1.17 wasn't branched yet, so this is part of 1.17 then..
Updated by The Foreman Bot almost 7 years ago
- Pull request https://github.com/theforeman/foreman/pull/5142 added
Updated by Andrew Kofink over 6 years ago
- Related to Bug #22894: Make RHSM facts compatible with fact name filtering added