Project

General

Profile

Bug #687

rake reports:expire abuses memory and network bandwidth

Added by Kal McFate about 8 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Database
Target version:
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

Noticed this specifically from the reports:expire process:

SELECT * FROM `reports` WHERE (created_at < '2011-02-22 15:52:29' and status = 0)

There is no need at all to send the entire row. With thousands of hosts with 50 reports a day, this job sends GB of data over the network from the sql server (and into the rake processes memory) every time it is run just to delete reports. This should be optimized as it can cause some serious memory and network usage problems if processing a large number of reports.

Associated revisions

Revision 0de3b547 (diff)
Added by Ohad Levy almost 8 years ago

fixes #687 - this dramateclly improves expired reports deletion speed and system usage.

History

#1 Updated by Kal McFate about 8 years ago

Just for example:

I ran:
rake reports:expire days=12

followed by:
rake reports:expire days=10

results in:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5830 root 19 0 1300m 1.2g 3484 R 95.7 30.7 28:56.57 /usr/bin/ruby /usr/bin/rake reports:expire days=10

on 2 days of data.

30 minutes, and still going.

#2 Updated by Ohad Levy about 8 years ago

yes you are correct.

the main reason we need to pull all reports is to know if we need to delete the related log, message and source records.

maybe we could improve the finder sql to make it faster.

one thing we could do quickly, is simply to read 1000 records or so each time instead of loading them all into memory.

#3 Updated by Ohad Levy almost 8 years ago

  • Category set to Database
  • Status changed from New to Assigned
  • Assignee set to Ohad Levy
  • Target version set to 0.4

#4 Updated by Ohad Levy almost 8 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF