Project

General

Profile

Actions

Bug #23455

closed

Upgrade and migration 1.15.6 to 1.16.1 (with Katello) - Foreman unstable and unusable

Added by Josh Pavel almost 6 years ago. Updated 8 months ago.

Status:
Rejected
Priority:
Urgent
Assignee:
Category:
Upgrades
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

We are moving/migrating/upgrading our Foreman installation.

The backup/restore/upgrade went well (I hit https://bugzilla.redhat.com/show_bug.cgi?id=1556819, but the workaround offered there worked).

As I began attaching the smart-proxies back to the new Foreman, the first few went without issue. Eventually, though, I began to get flooded with messages like this in the log:

Apr 30 12:41:40 foreman-01 qpidd: 2018-04-30 12:41:40 [Protocol] error Error on attach: Node not found: pulp.agent.c0500867-d215-408e-a1ed-f8ed8b5e4075
Apr 30 12:41:40 foreman-01 qpidd32429: 2018-04-30 12:41:40 [Protocol] error Error on attach: Node not found: pulp.agent.c0500867-d215-408e-a1ed-f8ed8b5e4075
Apr 30 12:41:40 foreman-01 qpidd32429: 2018-04-30 12:41:40 [Protocol] error Error on attach: Node not found: pulp.agent.8278fc78-0b10-4ef5-b2de-b22b3e502007
Apr 30 12:41:40 foreman-01 qpidd: 2018-04-30 12:41:40 [Protocol] error Error on attach: Node not found: pulp.agent.8278fc78-0b10-4ef5-b2de-b22b3e502007
Apr 30 12:41:44 foreman-01 qpidd32429: 2018-04-30 12:41:44 [Protocol] error Error on attach: Node not found: pulp.agent.a3754ba6-6067-4c46-899a-248cabc4a2d8
Apr 30 12:41:44 foreman-01 qpidd: 2018-04-30 12:41:44 [Protocol] error Error on attach: Node not found: pulp.agent.a3754ba6-6067-4c46-899a-248cabc4a2d8
Apr 30 12:41:45 foreman-01 qpidd: 2018-04-30 12:41:45 [Protocol] error Error on attach: Node not found: pulp.agent.033fe747-c608-4207-9176-a17cdedbff49
Apr 30 12:41:45 foreman-01 qpidd32429: 2018-04-30 12:41:45 [Protocol] error Error on attach: Node not found: pulp.agent.033fe747-c608-4207-9176-a17cdedbff49
Apr 30 12:41:45 foreman-01 qpidd32429: 2018-04-30 12:41:45 [Protocol] error Error on attach: Node not found: pulp.agent.af48ce1d-a58d-4b58-bf0f-c005e2c754fc
Apr 30 12:41:45 foreman-01 qpidd: 2018-04-30 12:41:45 [Protocol] error Error on attach: Node not found: pulp.agent.af48ce1d-a58d-4b58-bf0f-c005e2c754fc

...and I notice that the UI is locked up and unresponsive until a restart, in which case it is only operable for a few minutes.

There are constantly 5 or 6 postgres tasks taking up a full core that do not seem to resolve themselves, even when they've been let run for several hours.

33848 | -00:00:01.183384 | foreman   | SELECT  "sources".* FROM "sources" WHERE "sources"."digest" = $1  ORDER BY "sources"."id" ASC LIMIT 1
33963 | -00:00:03.247328 | foreman | SELECT "sources".* FROM "sources" WHERE "sources"."digest" = $1 ORDER BY "sources"."id" ASC LIMIT 1
33928 | -00:00:04.361622 | foreman | SELECT "sources".* FROM "sources" WHERE "sources"."digest" = $1 ORDER BY "sources"."id" ASC LIMIT 1
33889 | -00:00:04.430823 | foreman | SELECT "sources".* FROM "sources" WHERE "sources"."digest" = $1 ORDER BY "sources"."id" ASC LIMIT 1
33780 | -00:00:04.430876 | foreman | SELECT "sources".* FROM "sources" WHERE "sources"."digest" = $1 ORDER BY "sources"."id" ASC LIMIT 1
33814 | -00:00:04.431819 | foreman | SELECT "sources".* FROM "sources" WHERE "sources"."digest" = $1 ORDER BY "sources"."id" ASC LIMIT 1
Actions

Also available in: Atom PDF