multiple paused state "Create Pulpdiskspace" notifications is beng triggered in dynflow
On a nightly install, we see many tasks under dynflow around "CreatePulpDiskSpaceNotifications"
Error: NoMethodError undefined method `storage' for nil:NilClass - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:8:in `block in deliver!'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:7:in `deliver!'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/jobs/create_pulp_disk_space_notifications.rb:7:in `perform'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/execution.rb:37:in `block in perform_now'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/i18n-0.9.1/lib/i18n.rb:265:in `with_locale'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `instance_exec'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/logging.rb:24:in `block (4 levels) in <module:Logging>'" - "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'"
I'm not sure what's causing that so filing under dynflow component for now.
#4 Updated by Jonathon Turel over 1 year ago
I looked into this and I think there's an issue around the timing of the foreman and foreman_tasks being installed and started. I also didn't see this problem exactly as described which also indicates a timing related issue to me.
From foreman production.log:
2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "foreman_tasks_tasks" does not exist LINE 8: WHERE a.attrelid = '"foreman_tasks_tasks"'::r... ^ : SELECT a.attname, format_type(a.atttypid, a.atttypmod), pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, c.collname, col_description(a.attrelid, a.attnum) AS comment FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum LEFT JOIN pg_type t ON a.atttypid = t.oid LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum
From foreman_proxy proxy.log:
# Logfile created on 2018-05-03 00:54:14 +0000 by logger.rb/41954 I, [2018-05-03T00:54:14.761880 ] INFO -- : Successfully initialized 'pulp' I, [2018-05-03T00:54:14.761989 ] INFO -- : Successfully initialized 'foreman_proxy' I, [2018-05-03T00:54:14.762027 ] INFO -- : Successfully initialized 'templates' I, [2018-05-03T00:54:14.762060 ] INFO -- : Successfully initialized 'tftp' I, [2018-05-03T00:54:14.762089 ] INFO -- : Successfully initialized 'puppetca' I, [2018-05-03T00:54:14.762250 ] INFO -- : Started puppet class cache initialization I, [2018-05-03T00:54:14.769301 ] INFO -- : Successfully initialized 'puppet_proxy_puppet_api' I, [2018-05-03T00:54:14.769425 ] INFO -- : Successfully initialized 'puppet' I, [2018-05-03T00:54:14.769486 ] INFO -- : Successfully initialized 'logs' I, [2018-05-03T00:54:14.783040 ] INFO -- : WEBrick 1.3.1 I, [2018-05-03T00:54:14.783123 ] INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux] I, [2018-05-03T00:54:14.783934 ] INFO -- :
I'm not very familiar with the installer side of things but it looks like foreman is starting and kicking off the jobs when foreman_tasks hasn't been installed & started - see that there is a 4 minute difference in the logs. I will think about what the appropriate fix is and update.