Bug #20423
closedForeman DB logs index out of range
Description
Hi,
Why is this a problem?
Our Foreman instances stopped recording new puppet logs in the DB.
What caused the problem?
Our investigation suggests that setting the index in the logs table as an integer (a normal serial instead of bigserial/int8) seems to be the root cause. I'm not 100% sure what code actually does it, my best guess is in here: https://github.com/theforeman/foreman/blob/develop/db/migrate/20101018120621_create_logs.rb , but I don't see a reference to the actual index.
Why fix the problem?
Other long-lived or large installs will run in to this issue. Any table that has significant turnover/growth will face this issue at some point. 2^31-1 isn't a huge number when you have scaled out your infrastructure.
Short term fix?
Clear out logs, reset the index, hope things don't break.
Long term fix?
Change the index to a bigserial/int8.
Writeup:
Per https://groups.google.com/forum/#!topic/foreman-users/hsBX88LEUnY
This is in Foreman 1.12, 1.13, & 1.14. I see the same code in 1.15 for this, but haven't tested it.
After we found that our puppet logs weren't updating, we found (lots of) this in our postgres logs:
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]ERROR: integer out of range
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]STATEMENT: INSERT INTO "logs" ("message_id", "source_id", "report_id", "level_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
So we checked the log id #:
myforemandb=> select * from logs order by id desc limit 1;
id | source_id | message_id | report_id | level_id | created_at | updated_at
------------+-----------+------------+-----------+----------+----------------------------+----------------------------
2147483647 | 321 | 1807107 | 292344055 | 2 | 2017-07-24 12:08:01.668757 | 2017-07-24 12:08:01.668757
(1 row)
That id listed is 2^31-1. It turns out that "id" is a signed int, meaning it's 32-bit. I cannot write more logs to the logs table now.
myforemandb=> \d+ logs
Table "public.logs"
Column | Type | Modifiers | Storage | Stats target | Description
------------+-----------------------------+---------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('logs_id_seq'::regclass) | plain | |
source_id | integer | | plain | |
message_id | integer | | plain | |
report_id | integer | | plain | |
level_id | integer | | plain | |
created_at | timestamp without time zone | not null | plain | |
updated_at | timestamp without time zone | not null | plain | |
Indexes:
"logs_pkey" PRIMARY KEY, btree (id)
"index_logs_on_level_id" btree (level_id)
"index_logs_on_message_id" btree (message_id)
"index_logs_on_report_id" btree (report_id)
"index_logs_on_source_id" btree (source_id)