Bug #1592
closedreport::expire errors with millions of records
Description
When running report::expire against a DB with millions of records in Reports the 'function' will fail (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/kH0ISyjoMhgJ):
Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465,15466,15467,15468,15469,15470,15471,15475,16252,16256,16257,16258,16259,16260,16263,16267,16268,16269,16270,16272,16279,16282,16283,16284,16285,16286,16287,16302,18042,18043,18044,18045,18046,18062,18063,18064,18065,18066,18067,18068,18084,18085,
Which seems to stem from (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/EADjgREX3Z4J):
And since the 'bad' query starts with: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465 my guess is its specifically line: # reports which have logs entries used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).m ap(&:id)
https://groups.google.com/forum/?fromgroups#!topic/foreman-users/_Mn4oxXmP7E
Code to functionally get past this error that I've used is (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/lSKWXpW3JoQJ):
# used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).map(&:id) used_reports = [] tmp_arr=all_reports.dup slice_size = 10000 until tmp_arr.empty? do first = (tmp_arr.size < slice_size) ? 0 : (tmp_arr.size - slice_size) last = tmp_arr.size tmp = tmp_arr[first..last] used_reports << Report.all(:select => :id, :conditions => {:id => tmp}).map(&:id) tmp_arr = tmp_arr - tmp end used_reports.flatten!.sort
Updated by Ori Rabin over 10 years ago
- Has duplicate Bug #1546: report::expire slow with many reports added
Updated by Ori Rabin over 10 years ago
- Description updated (diff)
- Status changed from New to Assigned
- Assignee set to Ori Rabin
Updated by The Foreman Bot over 10 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/1659 added
- Pull request deleted (
)
Updated by Dominic Cleal over 10 years ago
- Translation missing: en.field_release set to 10
Updated by Dominic Cleal over 10 years ago
- Translation missing: en.field_release deleted (
10)
Updated by Ori Rabin over 10 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 4215defb81fc2b469669592ac1094d7c70950192.
Updated by Dominic Cleal over 10 years ago
- Status changed from Closed to Ready For Testing
- % Done changed from 100 to 0
Commit was reverted (https://github.com/theforeman/foreman/commit/bf1d8e08a928a72f879911225afb21d103895e5b) and the PR re-opened as issues were raised.
Updated by Anonymous over 10 years ago
- Target version changed from 1.7.5 to 1.7.4
Updated by Anonymous over 10 years ago
- Target version changed from 1.7.4 to 1.7.3
Updated by Dominic Cleal over 10 years ago
- Translation missing: en.field_release set to 21
Updated by Ori Rabin over 10 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset ab866a3185b132555e1eed393f8de6c5a9a729c4.
Updated by Daniel Lobato Garcia about 10 years ago
Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.
Updated by Dominic Cleal about 10 years ago
Daniel Lobato Garcia wrote:
Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.
That's the wrong way around.. 1.6.1 should be a minimum risk & stable release, giving us time in pre-1.7 testing to find bugs. Point releases are not for experiments.
Given this has already been reverted once, I think it's moderate risk so probably unsuitable.
Updated by Dominic Cleal about 10 years ago
- Related to Bug #8565: report::expire is running very slowly added
Updated by The Foreman Bot almost 9 years ago
- Pull request https://github.com/theforeman/foreman/pull/2994 added
Updated by Ori Rabin almost 9 years ago
- Pull request deleted (
https://github.com/theforeman/foreman/pull/2994)