ApplicationJob starts when foreman_tasks does not exist
I discovered this while looking into an issue in Katello on the centos7-katello-nightly forklift VM but I suspect it'll show up after any fresh installation.
ApplicationJobs (using dynflow) are being started when foreman_tasks has not been fully initialized and this error can be seen from several jobs across Foreman and Katello being started:
2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "foreman_tasks_tasks" does not exist LINE 8: WHERE a.attrelid = '"foreman_tasks_tasks"'::r... ^ : SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, c.collname, col_description(a.attrelid, a.attnum) AS comment FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum LEFT JOIN pg_type t ON a.atttypid = t.oid LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum
I believe it comes down to a timing issue with how these components are installed; hoping to start & track discussion about it here. It seems like even if the jobs do eventually start due to retry attempts, we shouldn't be throwing huge stack traces. I don't mind fixing the problem, but some guidance would be appreciated. To add, it appears that the jobs do eventually start, but after several attempts.
- should we expand ApplicationJob to not start jobs until foreman_tasks and/or dynflow are completely initialized?
- something else?
- Fixes #23611 - Provide helper method to run "service" active jobs
The root cause was we were triggering the active jobs too early,
sometimes even before the db was migrated. This led to all sorts of
errors. The provided helper method does not trigger the jobs when in
any rake or in test environment. This helper should be reused in
Katello as well.