Bug #9016
Updated by Lukas Zapletal over 8 years ago
This was initially reported on discovery process, but I located performance bottleneck in fact importing. When when importing a large set of discovered hosts, the avg response time is high: <pre> Completed 201 Created in 8403.7ms (Views: 3.6ms | ActiveRecord: 0.0ms) Completed 201 Created in 3546.0ms (Views: 27.6ms | ActiveRecord: 0.0ms) Completed 201 Created in 3260.2ms (Views: 3.4ms | ActiveRecord: 0.0ms) Completed 201 Created in 5449.5ms (Views: 3.1ms | ActiveRecord: 0.0ms) Completed 201 Created in 5356.3ms (Views: 3.5ms | ActiveRecord: 0.1ms) Completed 201 Created in 8333.2ms (Views: 3.7ms | ActiveRecord: 0.1ms) Completed 201 Created in 5271.3ms (Views: 3.2ms | ActiveRecord: 0.1ms) Completed 201 Created in 8462.4ms (Views: 3.2ms | ActiveRecord: 0.1ms) Completed 201 Created in 4223.7ms (Views: 3.2ms | ActiveRecord: 0.2ms) Completed 201 Created in 8827.6ms (Views: 3.9ms | ActiveRecord: 0.1ms) Completed 201 Created in 4510.4ms (Views: 3.1ms | ActiveRecord: 0.1ms) Completed 201 Created in 4038.9ms (Views: 2.9ms | ActiveRecord: 0.1ms) Completed 201 Created in 4075.4ms (Views: 3.2ms | ActiveRecord: 0.1ms) Completed 201 Created in 4007.8ms (Views: 4.5ms | ActiveRecord: 0.2ms) Completed 201 Created in 3886.1ms (Views: 3.6ms | ActiveRecord: 0.1ms) Completed 201 Created in 4845.1ms (Views: 3.5ms | ActiveRecord: 0.2ms) Completed 201 Created in 4626.0ms (Views: 3.6ms | ActiveRecord: 0.1ms) Completed 201 Created in 4751.8ms (Views: 3.7ms | ActiveRecord: 0.1ms) </pre> Observation, the O(n) depends on amount of facts reported. These are performance tests on production installation of Satellite 6.2 RC1 (roughly Foreman 1.11) with five concurrent CURL processes in loop registering discovery hosts (it calls facts import basically): With 10 facts: <pre> 2016-07-21 13:06:20 [app] [I] Completed 201 Created in 386ms (Views: 2.0ms | ActiveRecord: 66.4ms) 2016-07-21 13:06:20 [app] [I] Completed 201 Created in 353ms (Views: 1.9ms | ActiveRecord: 68.8ms) 2016-07-21 13:06:21 [app] [I] Completed 201 Created in 360ms (Views: 2.1ms | ActiveRecord: 72.6ms) 2016-07-21 13:06:21 [app] [I] Completed 201 Created in 316ms (Views: 1.7ms | ActiveRecord: 58.7ms) 2016-07-21 13:06:21 [app] [I] Completed 201 Created in 366ms (Views: 2.0ms | ActiveRecord: 68.5ms) 2016-07-21 13:06:21 [app] [I] Completed 201 Created in 331ms (Views: 1.7ms | ActiveRecord: 59.6ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 322ms (Views: 1.7ms | ActiveRecord: 58.8ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 380ms (Views: 1.9ms | ActiveRecord: 85.0ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 337ms (Views: 2.2ms | ActiveRecord: 61.8ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 385ms (Views: 2.4ms | ActiveRecord: 77.8ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 442ms (Views: 1.7ms | ActiveRecord: 70.3ms) 2016-07-21 13:06:22 [app] [I] Completed 201 Created in 388ms (Views: 1.9ms | ActiveRecord: 86.2ms) 2016-07-21 13:06:23 [app] [I] Completed 201 Created in 349ms (Views: 2.1ms | ActiveRecord: 66.0ms) 2016-07-21 13:06:23 [app] [I] Completed 201 Created in 346ms (Views: 1.7ms | ActiveRecord: 74.3ms) 2016-07-21 13:06:23 [app] [I] Completed 201 Created in 364ms (Views: 1.8ms | ActiveRecord: 61.6ms) 2016-07-21 13:06:23 [app] [I] Completed 201 Created in 488ms (Views: 2.4ms | ActiveRecord: 80.6ms) 2016-07-21 13:06:24 [app] [I] Completed 201 Created in 485ms (Views: 2.7ms | ActiveRecord: 74.3ms) 2016-07-21 13:06:24 [app] [I] Completed 201 Created in 577ms (Views: 2.8ms | ActiveRecord: 76.6ms) </pre> With 500 facts: <pre> 2016-07-21 13:09:29 [app] [I] Completed 201 Created in 1666ms (Views: 1.8ms | ActiveRecord: 376.8ms) 2016-07-21 13:09:30 [app] [I] Completed 201 Created in 1688ms (Views: 1.8ms | ActiveRecord: 380.2ms) 2016-07-21 13:09:31 [app] [I] Completed 201 Created in 1775ms (Views: 1.8ms | ActiveRecord: 387.7ms) 2016-07-21 13:09:32 [app] [I] Completed 201 Created in 1732ms (Views: 1.8ms | ActiveRecord: 436.5ms) 2016-07-21 13:09:33 [app] [I] Completed 201 Created in 1767ms (Views: 1.8ms | ActiveRecord: 410.1ms) 2016-07-21 13:09:34 [app] [I] Completed 201 Created in 1749ms (Views: 1.9ms | ActiveRecord: 404.3ms) 2016-07-21 13:09:34 [app] [I] Completed 201 Created in 1777ms (Views: 1.9ms | ActiveRecord: 416.1ms) 2016-07-21 13:09:35 [app] [I] Completed 201 Created in 1702ms (Views: 1.7ms | ActiveRecord: 398.9ms) 2016-07-21 13:09:36 [app] [I] Completed 201 Created in 1650ms (Views: 1.8ms | ActiveRecord: 358.3ms) 2016-07-21 13:09:37 [app] [I] Completed 201 Created in 1647ms (Views: 1.8ms | ActiveRecord: 330.7ms) 2016-07-21 13:09:38 [app] [I] Completed 201 Created in 1761ms (Views: 1.8ms | ActiveRecord: 404.6ms) 2016-07-21 13:09:39 [app] [I] Completed 201 Created in 1629ms (Views: 1.8ms | ActiveRecord: 347.6ms) 2016-07-21 13:09:40 [app] [I] Completed 201 Created in 1735ms (Views: 1.8ms | ActiveRecord: 420.6ms) 2016-07-21 13:09:40 [app] [I] Completed 201 Created in 1616ms (Views: 2.1ms | ActiveRecord: 329.6ms) 2016-07-21 13:09:41 [app] [I] Completed 201 Created in 1792ms (Views: 2.0ms | ActiveRecord: 418.4ms) 2016-07-21 13:09:42 [app] [I] Completed 201 Created in 1794ms (Views: 1.8ms | ActiveRecord: 454.0ms) 2016-07-21 13:09:43 [app] [I] Completed 201 Created in 1779ms (Views: 2.5ms | ActiveRecord: 424.4ms) 2016-07-21 13:09:44 [app] [I] Completed 201 Created in 1735ms (Views: 1.8ms | ActiveRecord: 411.7ms) 2016-07-21 13:09:45 [app] [I] Completed 201 Created in 1912ms (Views: 1.9ms | ActiveRecord: 461.2ms) 2016-07-21 13:09:46 [app] [I] Completed 201 Created in 1781ms (Views: 1.9ms | ActiveRecord: 463.5ms) 2016-07-21 13:09:47 [app] [I] Completed 201 Created in 1682ms (Views: 1.7ms | ActiveRecord: 377.2ms) 2016-07-21 13:09:47 [app] [I] Completed 201 Created in 1645ms (Views: 2.2ms | ActiveRecord: 336.1ms) 2016-07-21 13:09:49 [app] [I] Completed 201 Created in 1843ms (Views: 2.0ms | ActiveRecord: 435.9ms) 2016-07-21 13:09:49 [app] [I] Completed 201 Created in 1900ms (Views: 2.3ms | ActiveRecord: 470.4ms) 2016-07-21 13:09:51 [app] [I] Completed 201 Created in 1841ms (Views: 1.7ms | ActiveRecord: 406.0ms) 2016-07-21 13:09:51 [app] [I] Completed 201 Created in 1949ms (Views: 2.5ms | ActiveRecord: 443.3ms) 2016-07-21 13:09:52 [app] [I] Completed 201 Created in 1773ms (Views: 1.8ms | ActiveRecord: 453.9ms) </pre> And with 1000 facts reported in each call: <pre> 2016-07-21 13:11:12 [app] [I] Completed 201 Created in 3104ms (Views: 1.8ms | ActiveRecord: 764.2ms) 2016-07-21 13:11:14 [app] [I] Completed 201 Created in 3133ms (Views: 1.9ms | ActiveRecord: 667.6ms) 2016-07-21 13:11:15 [app] [I] Completed 201 Created in 3113ms (Views: 2.0ms | ActiveRecord: 732.8ms) 2016-07-21 13:11:17 [app] [I] Completed 201 Created in 3139ms (Views: 1.8ms | ActiveRecord: 752.2ms) 2016-07-21 13:11:18 [app] [I] Completed 201 Created in 3015ms (Views: 2.2ms | ActiveRecord: 730.2ms) 2016-07-21 13:11:20 [app] [I] Completed 201 Created in 3139ms (Views: 1.9ms | ActiveRecord: 646.0ms) 2016-07-21 13:11:21 [app] [I] Completed 201 Created in 2877ms (Views: 1.8ms | ActiveRecord: 620.0ms) 2016-07-21 13:11:23 [app] [I] Completed 201 Created in 3011ms (Views: 2.4ms | ActiveRecord: 611.1ms) 2016-07-21 13:11:24 [app] [I] Completed 201 Created in 3007ms (Views: 1.8ms | ActiveRecord: 614.2ms) 2016-07-21 13:11:26 [app] [I] Completed 201 Created in 3163ms (Views: 1.9ms | ActiveRecord: 661.9ms) 2016-07-21 13:11:27 [app] [I] Completed 201 Created in 2923ms (Views: 1.8ms | ActiveRecord: 641.7ms) 2016-07-21 13:11:29 [app] [I] Completed 201 Created in 2982ms (Views: 2.1ms | ActiveRecord: 657.7ms) 2016-07-21 13:11:30 [app] [I] Completed 201 Created in 3018ms (Views: 1.8ms | ActiveRecord: 706.9ms) 2016-07-21 13:11:32 [app] [I] Completed 201 Created in 3212ms (Views: 1.8ms | ActiveRecord: 726.9ms) 2016-07-21 13:11:33 [app] [I] Completed 201 Created in 2931ms (Views: 2.1ms | ActiveRecord: 602.9ms) 2016-07-21 13:11:35 [app] [I] Completed 201 Created in 2991ms (Views: 2.5ms | ActiveRecord: 638.4ms) 2016-07-21 13:11:36 [app] [I] Completed 201 Created in 3016ms (Views: 1.8ms | ActiveRecord: 728.8ms) 2016-07-21 13:11:39 [app] [I] Completed 201 Created in 3170ms (Views: 1.7ms | ActiveRecord: 744.1ms) 2016-07-21 13:11:40 [app] [I] Completed 201 Created in 3312ms (Views: 1.9ms | ActiveRecord: 823.8ms) 2016-07-21 13:11:42 [app] [I] Completed 201 Created in 3198ms (Views: 1.7ms | ActiveRecord: 820.9ms) </pre> Width of fact name or value does not matter, what's the bottleneck is our code that does fact importing. It does this in a loop over all facts merging them with database. We need to optimize that.