Project

General

Profile

Actions

Bug #1592

closed

report::expire errors with millions of records

Added by Jacob McCann almost 12 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Reporting
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

When running report::expire against a DB with millions of records in Reports the 'function' will fail (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/kH0ISyjoMhgJ):

Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465,15466,15467,15468,15469,15470,15471,15475,16252,16256,16257,16258,16259,16260,16263,16267,16268,16269,16270,16272,16279,16282,16283,16284,16285,16286,16287,16302,18042,18043,18044,18045,18046,18062,18063,18064,18065,18066,18067,18068,18084,18085,

Which seems to stem from (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/EADjgREX3Z4J):

And since the 'bad' query starts with: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465 my guess is its specifically line:

  # reports which have logs entries
      used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).m
  ap(&:id)

https://groups.google.com/forum/?fromgroups#!topic/foreman-users/_Mn4oxXmP7E

Code to functionally get past this error that I've used is (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/lSKWXpW3JoQJ):

    # used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).map(&:id)
    used_reports = []
    tmp_arr=all_reports.dup
    slice_size = 10000
    until tmp_arr.empty? do
      first = (tmp_arr.size < slice_size) ? 0 : (tmp_arr.size - slice_size)
      last = tmp_arr.size
      tmp = tmp_arr[first..last]
      used_reports << Report.all(:select => :id, :conditions => {:id => tmp}).map(&:id)
      tmp_arr = tmp_arr - tmp
    end
    used_reports.flatten!.sort


Related issues 2 (0 open2 closed)

Related to Foreman - Bug #8565: report::expire is running very slowlyClosedDominic Cleal12/03/2014Actions
Has duplicate Foreman - Bug #1546: report::expire slow with many reportsDuplicateOri RabinActions
Actions #1

Updated by Greg Sutcliffe over 11 years ago

  • Target version set to Bug scrub
Actions #2

Updated by Greg Sutcliffe over 10 years ago

  • Target version deleted (Bug scrub)
Actions #3

Updated by Ohad Levy over 9 years ago

  • Target version set to 1.7.5
Actions #4

Updated by Ori Rabin over 9 years ago

  • Has duplicate Bug #1546: report::expire slow with many reports added
Actions #5

Updated by Ori Rabin over 9 years ago

  • Description updated (diff)
  • Status changed from New to Assigned
  • Assignee set to Ori Rabin
Actions #6

Updated by The Foreman Bot over 9 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/1659 added
  • Pull request deleted ()
Actions #7

Updated by Dominic Cleal over 9 years ago

  • translation missing: en.field_release set to 10
Actions #8

Updated by Dominic Cleal over 9 years ago

  • translation missing: en.field_release deleted (10)
Actions #9

Updated by Ori Rabin over 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #10

Updated by Dominic Cleal over 9 years ago

  • Status changed from Closed to Ready For Testing
  • % Done changed from 100 to 0

Commit was reverted (https://github.com/theforeman/foreman/commit/bf1d8e08a928a72f879911225afb21d103895e5b) and the PR re-opened as issues were raised.

Actions #11

Updated by Anonymous over 9 years ago

  • Target version changed from 1.7.5 to 1.7.4
Actions #12

Updated by Anonymous over 9 years ago

  • Target version changed from 1.7.4 to 1.7.3
Actions #13

Updated by Dominic Cleal over 9 years ago

  • translation missing: en.field_release set to 21
Actions #14

Updated by Ori Rabin over 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #15

Updated by Daniel Lobato Garcia over 9 years ago

Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.

Actions #16

Updated by Dominic Cleal over 9 years ago

Daniel Lobato Garcia wrote:

Any reasons why this can't go in 1.6.1? Releasing this in 1.6.1 would help us find possible bugs before 1.7.

That's the wrong way around.. 1.6.1 should be a minimum risk & stable release, giving us time in pre-1.7 testing to find bugs. Point releases are not for experiments.

Given this has already been reverted once, I think it's moderate risk so probably unsuitable.

Actions #17

Updated by Dominic Cleal over 9 years ago

  • Related to Bug #8565: report::expire is running very slowly added
Actions #18

Updated by The Foreman Bot over 8 years ago

  • Pull request https://github.com/theforeman/foreman/pull/2994 added
Actions #19

Updated by Ori Rabin over 8 years ago

  • Pull request deleted (https://github.com/theforeman/foreman/pull/2994)
Actions

Also available in: Atom PDF