Project

General

Profile

Bug #23611

ApplicationJob starts when foreman_tasks does not exist

Added by Jonathon Turel about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Team Backlog:
Fixed in Releases:
Found in Releases:

Description

I discovered this while looking into an issue in Katello on the centos7-katello-nightly forklift VM but I suspect it'll show up after any fresh installation.

ApplicationJobs (using dynflow) are being started when foreman_tasks has not been fully initialized and this error can be seen from several jobs across Foreman and Katello being started:

2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "foreman_tasks_tasks" does not exist
LINE 8:                WHERE a.attrelid = '"foreman_tasks_tasks"'::r...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

I believe it comes down to a timing issue with how these components are installed; hoping to start & track discussion about it here. It seems like even if the jobs do eventually start due to retry attempts, we shouldn't be throwing huge stack traces. I don't mind fixing the problem, but some guidance would be appreciated. To add, it appears that the jobs do eventually start, but after several attempts.

- should we expand ApplicationJob to not start jobs until foreman_tasks and/or dynflow are completely initialized?
- something else?


Related issues

Related to Katello - Bug #23326: multiple paused state "Create Pulpdiskspace" notifications is beng triggered in dynflowClosed2018-04-19
Related to Foreman - Bug #22833: delayed_plan 'CreateRssNotifications' is failing in 15sec intervals Duplicate2018-03-09
Related to Foreman - Bug #23932: ApplicationJob tests fail when run on jenkins from a pluginClosed2018-06-14

Associated revisions

Revision e5b4634f (diff)
Added by Adam Ruzicka about 1 year ago

  • Fixes #23611 - Provide helper method to run "service" active jobs

The root cause was we were triggering the active jobs too early,
sometimes even before the db was migrated. This led to all sorts of
errors. The provided helper method does not trigger the jobs when in
any rake or in test environment. This helper should be reused in
Katello as well.

History

#1 Updated by Jonathon Turel about 1 year ago

  • Related to Bug #23326: multiple paused state "Create Pulpdiskspace" notifications is beng triggered in dynflow added

#2 Updated by Ivan Necas about 1 year ago

  • Category deleted (Dynflow)
  • Project changed from foreman-tasks to Foreman

#3 Updated by The Foreman Bot about 1 year ago

  • Assignee set to Adam Ruzicka
  • Status changed from New to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/5575 added

#4 Updated by Marek Hulán about 1 year ago

  • Related to Bug #22833: delayed_plan 'CreateRssNotifications' is failing in 15sec intervals added

#5 Updated by Marek Hulán about 1 year ago

  • Bugzilla link set to 1581311

#6 Updated by Ivan Necas about 1 year ago

  • Legacy Backlogs Release (now unused) set to 330

#7 Updated by Ivan Necas about 1 year ago

  • Status changed from Ready For Testing to Closed

#8 Updated by The Foreman Bot about 1 year ago

  • Pull request https://github.com/theforeman/foreman/pull/5682 added

#9 Updated by Adam Ruzicka about 1 year ago

  • Related to Bug #23932: ApplicationJob tests fail when run on jenkins from a plugin added

Also available in: Atom PDF