Bug #1592
report::expire errors with millions of records
Description
When running report::expire against a DB with millions of records in Reports the 'function' will fail (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/kH0ISyjoMhgJ):
Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465,15466,15467,15468,15469,15470,15471,15475,16252,16256,16257,16258,16259,16260,16263,16267,16268,16269,16270,16272,16279,16282,16283,16284,16285,16286,16287,16302,18042,18043,18044,18045,18046,18062,18063,18064,18065,18066,18067,18068,18084,18085,
Which seems to stem from (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/EADjgREX3Z4J):
And since the 'bad' query starts with: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465 my guess is its specifically line: # reports which have logs entries used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).m ap(&:id)
https://groups.google.com/forum/?fromgroups#!topic/foreman-users/_Mn4oxXmP7E
Code to functionally get past this error that I've used is (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/lSKWXpW3JoQJ):
# used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).map(&:id) used_reports = [] tmp_arr=all_reports.dup slice_size = 10000 until tmp_arr.empty? do first = (tmp_arr.size < slice_size) ? 0 : (tmp_arr.size - slice_size) last = tmp_arr.size tmp = tmp_arr[first..last] used_reports << Report.all(:select => :id, :conditions => {:id => tmp}).map(&:id) tmp_arr = tmp_arr - tmp end used_reports.flatten!.sort
Related issues
Associated revisions
Fixes #1592: making report::expire faster and without errors
History
#1
Updated by Greg Sutcliffe over 10 years ago
- Target version set to Bug scrub
#2
Updated by Greg Sutcliffe over 9 years ago
- Target version deleted (
Bug scrub)
#3
Updated by Ohad Levy over 8 years ago
- Target version set to 1.7.5
#4
Updated by Ori Rabin over 8 years ago
- Has duplicate Bug #1546: report::expire slow with many reports added
#5
Updated by Ori Rabin over 8 years ago
- Description updated (diff)
- Status changed from New to Assigned
- Assignee set to Ori Rabin
#6
Updated by The Foreman Bot over 8 years ago
- Status changed from Assigned to Ready For Testing
- Pull request https://github.com/theforeman/foreman/pull/1659 added
- Pull request deleted (
)
#7
Updated by Dominic Cleal over 8 years ago
- Legacy Backlogs Release (now unused) set to 10
#8
Updated by Dominic Cleal over 8 years ago
- Legacy Backlogs Release (now unused) deleted (
10)
#9
Updated by Ori Rabin over 8 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset 4215defb81fc2b469669592ac1094d7c70950192.
#10
Updated by Dominic Cleal over 8 years ago
- Status changed from Closed to Ready For Testing
- % Done changed from 100 to 0
Commit was reverted (https://github.com/theforeman/foreman/commit/bf1d8e08a928a72f879911225afb21d103895e5b) and the PR re-opened as issues were raised.
#11
Updated by Anonymous over 8 years ago
- Target version changed from 1.7.5 to 1.7.4
#12
Updated by Anonymous over 8 years ago
- Target version changed from 1.7.4 to 1.7.3
#13
Updated by Dominic Cleal over 8 years ago
- Legacy Backlogs Release (now unused) set to 21
#14
Updated by Ori Rabin over 8 years ago
- Status changed from Ready For Testing to Closed
- % Done changed from 0 to 100
Applied in changeset ab866a3185b132555e1eed393f8de6c5a9a729c4.
#15
Updated by Daniel Lobato Garcia over 8 years ago
Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.
#16
Updated by Dominic Cleal over 8 years ago
Daniel Lobato Garcia wrote:
Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.
That's the wrong way around.. 1.6.1 should be a minimum risk & stable release, giving us time in pre-1.7 testing to find bugs. Point releases are not for experiments.
Given this has already been reverted once, I think it's moderate risk so probably unsuitable.
#17
Updated by Dominic Cleal over 8 years ago
- Related to Bug #8565: report::expire is running very slowly added
#18
Updated by The Foreman Bot over 7 years ago
- Pull request https://github.com/theforeman/foreman/pull/2994 added
#19
Updated by Ori Rabin over 7 years ago
- Pull request deleted (
https://github.com/theforeman/foreman/pull/2994)
Fixes #1592: making report::expire faster and without errors