I discovered this while looking into an issue in Katello on the centos7-katello-nightly forklift VM but I suspect it'll show up after any fresh installation.

ApplicationJobs (using dynflow) are being started when foreman_tasks has not been fully initialized and this error can be seen from several jobs across Foreman and Katello being started:

2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "foreman_tasks_tasks" does not exist
LINE 8:                WHERE a.attrelid = '"foreman_tasks_tasks"'::r...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

I believe it comes down to a timing issue with how these components are installed; hoping to start & track discussion about it here. It seems like even if the jobs do eventually start due to retry attempts, we shouldn't be throwing huge stack traces. I don't mind fixing the problem, but some guidance would be appreciated. To add, it appears that the jobs do eventually start, but after several attempts.

- should we expand ApplicationJob to not start jobs until foreman_tasks and/or dynflow are completely initialized?
- something else?