Project

General

Profile

Bug #1592

report::expire errors with millions of records

Added by Jacob McCann about 7 years ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Reporting
Target version:
Difficulty:
Triaged:
Bugzilla link:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

When running report::expire against a DB with millions of records in Reports the 'function' will fail (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/kH0ISyjoMhgJ):

Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465,15466,15467,15468,15469,15470,15471,15475,16252,16256,16257,16258,16259,16260,16263,16267,16268,16269,16270,16272,16279,16282,16283,16284,16285,16286,16287,16302,18042,18043,18044,18045,18046,18062,18063,18064,18065,18066,18067,18068,18084,18085,

Which seems to stem from (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/EADjgREX3Z4J):

And since the 'bad' query starts with: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465 my guess is its specifically line:

  # reports which have logs entries
      used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).m
  ap(&:id)

https://groups.google.com/forum/?fromgroups#!topic/foreman-users/_Mn4oxXmP7E

Code to functionally get past this error that I've used is (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/lSKWXpW3JoQJ):

    # used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).map(&:id)
    used_reports = []
    tmp_arr=all_reports.dup
    slice_size = 10000
    until tmp_arr.empty? do
      first = (tmp_arr.size < slice_size) ? 0 : (tmp_arr.size - slice_size)
      last = tmp_arr.size
      tmp = tmp_arr[first..last]
      used_reports << Report.all(:select => :id, :conditions => {:id => tmp}).map(&:id)
      tmp_arr = tmp_arr - tmp
    end
    used_reports.flatten!.sort


Related issues

Related to Foreman - Bug #8565: report::expire is running very slowlyClosed2014-12-03
Has duplicate Foreman - Bug #1546: report::expire slow with many reportsDuplicate

Associated revisions

Revision 4215defb (diff)
Added by Ori Rabin almost 5 years ago

Fixes #1592: making report::expire faster and without errors

Revision ab866a31 (diff)
Added by Ori Rabin over 4 years ago

Fixes #1592: making report::expire faster and without errors

History

#1 Updated by Greg Sutcliffe over 6 years ago

  • Target version set to Bug scrub

#2 Updated by Greg Sutcliffe over 5 years ago

  • Target version deleted (Bug scrub)

#3 Updated by Ohad Levy almost 5 years ago

  • Target version set to 1.7.5

#4 Updated by Ori Rabin almost 5 years ago

  • Has duplicate Bug #1546: report::expire slow with many reports added

#5 Updated by Ori Rabin almost 5 years ago

  • Description updated (diff)
  • Status changed from New to Assigned
  • Assignee set to Ori Rabin

#6 Updated by The Foreman Bot almost 5 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/1659 added
  • Pull request deleted ()

#7 Updated by Dominic Cleal almost 5 years ago

  • Legacy Backlogs Release (now unused) set to 10

#8 Updated by Dominic Cleal almost 5 years ago

  • Legacy Backlogs Release (now unused) deleted (10)

#9 Updated by Ori Rabin almost 5 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#10 Updated by Dominic Cleal almost 5 years ago

  • Status changed from Closed to Ready For Testing
  • % Done changed from 100 to 0

Commit was reverted (https://github.com/theforeman/foreman/commit/bf1d8e08a928a72f879911225afb21d103895e5b) and the PR re-opened as issues were raised.

#11 Updated by Dmitri Dolguikh almost 5 years ago

  • Target version changed from 1.7.5 to 1.7.4

#12 Updated by Dmitri Dolguikh over 4 years ago

  • Target version changed from 1.7.4 to 1.7.3

#13 Updated by Dominic Cleal over 4 years ago

  • Legacy Backlogs Release (now unused) set to 21

#14 Updated by Ori Rabin over 4 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#15 Updated by Daniel Lobato Garcia over 4 years ago

Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.

#16 Updated by Dominic Cleal over 4 years ago

Daniel Lobato Garcia wrote:

Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.

That's the wrong way around.. 1.6.1 should be a minimum risk & stable release, giving us time in pre-1.7 testing to find bugs. Point releases are not for experiments.

Given this has already been reverted once, I think it's moderate risk so probably unsuitable.

#17 Updated by Dominic Cleal over 4 years ago

  • Related to Bug #8565: report::expire is running very slowly added

#18 Updated by The Foreman Bot over 3 years ago

  • Pull request https://github.com/theforeman/foreman/pull/2994 added

#19 Updated by Ori Rabin over 3 years ago

  • Pull request deleted (https://github.com/theforeman/foreman/pull/2994)

Also available in: Atom PDF