Project

General

Profile

Actions

Bug #20423

closed

Foreman DB logs index out of range

Added by Chris Baldwin over 6 years ago. Updated over 6 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
Database
Target version:
-
Difficulty:
easy
Triaged:
Fixed in Releases:
Found in Releases:

Description

Hi,

Why is this a problem?
Our Foreman instances stopped recording new puppet logs in the DB.
What caused the problem?
Our investigation suggests that setting the index in the logs table as an integer (a normal serial instead of bigserial/int8) seems to be the root cause. I'm not 100% sure what code actually does it, my best guess is in here: https://github.com/theforeman/foreman/blob/develop/db/migrate/20101018120621_create_logs.rb , but I don't see a reference to the actual index.
Why fix the problem?
Other long-lived or large installs will run in to this issue. Any table that has significant turnover/growth will face this issue at some point. 2^31-1 isn't a huge number when you have scaled out your infrastructure.
Short term fix?
Clear out logs, reset the index, hope things don't break.
Long term fix?
Change the index to a bigserial/int8.

Writeup:
Per https://groups.google.com/forum/#!topic/foreman-users/hsBX88LEUnY

This is in Foreman 1.12, 1.13, & 1.14. I see the same code in 1.15 for this, but haven't tested it.

After we found that our puppet logs weren't updating, we found (lots of) this in our postgres logs:
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]ERROR: integer out of range
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]STATEMENT: INSERT INTO "logs" ("message_id", "source_id", "report_id", "level_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"

So we checked the log id #:
myforemandb=> select * from logs order by id desc limit 1;
id | source_id | message_id | report_id | level_id | created_at | updated_at
------------+-----------+------------+-----------+----------+----------------------------+----------------------------
2147483647 | 321 | 1807107 | 292344055 | 2 | 2017-07-24 12:08:01.668757 | 2017-07-24 12:08:01.668757
(1 row)

That id listed is 2^31-1. It turns out that "id" is a signed int, meaning it's 32-bit. I cannot write more logs to the logs table now.

myforemandb=> \d+ logs
Table "public.logs"
Column | Type | Modifiers | Storage | Stats target | Description
------------+-----------------------------+---------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('logs_id_seq'::regclass) | plain | |
source_id | integer | | plain | |
message_id | integer | | plain | |
report_id | integer | | plain | |
level_id | integer | | plain | |
created_at | timestamp without time zone | not null | plain | |
updated_at | timestamp without time zone | not null | plain | |
Indexes:
"logs_pkey" PRIMARY KEY, btree (id)
"index_logs_on_level_id" btree (level_id)
"index_logs_on_message_id" btree (message_id)
"index_logs_on_report_id" btree (report_id)
"index_logs_on_source_id" btree (source_id)


Related issues 1 (0 open1 closed)

Is duplicate of Foreman - Bug #13810: Reports fail to upload, Mysql2::Error: Out of range value for column 'id' (logs table)ClosedTimo Goebel02/19/2016Actions
Actions #1

Updated by Daniel Lobato Garcia over 6 years ago

That's awful, and the fix should be tiny. I'll make a migration to improve these db column types for 1.15.3.

Actions #2

Updated by Daniel Lobato Garcia over 6 years ago

  • Status changed from New to Duplicate

Or actually - this looks like a dupe of https://github.com/theforeman/foreman/pull/4169/files (#13810), the logs column was increased to bigint there.

Are you able to update?

Actions #3

Updated by Daniel Lobato Garcia over 6 years ago

  • Is duplicate of Bug #13810: Reports fail to upload, Mysql2::Error: Out of range value for column 'id' (logs table) added
Actions #4

Updated by Chris Baldwin over 6 years ago

Daniel Lobato Garcia wrote:

Or actually - this looks like a dupe of https://github.com/theforeman/foreman/pull/4169/files (#13810), the logs column was increased to bigint there.

Are you able to update?

I can't update to 1.15 right now. I'll try to change the column to bigint to help future-proof my current install. Thanks for pointing out the duplicate, not sure why that didn't turn up in my searches.

Actions

Also available in: Atom PDF