Project

General

Profile

Bug #23326

multiple paused state "Create Pulpdiskspace" notifications is beng triggered in dynflow

Added by Sachin Ghai over 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Category:
Notifications
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

On a nightly install, we see many tasks under dynflow around "CreatePulpDiskSpaceNotifications"

Error:

NoMethodError

undefined method `storage' for nil:NilClass

- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:8:in
  `block in deliver!'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in
  `each'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in
  `each'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:7:in
  `deliver!'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/jobs/create_pulp_disk_space_notifications.rb:7:in
  `perform'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/execution.rb:37:in
  `block in perform_now'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in
  `block in run_callbacks'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/i18n-0.9.1/lib/i18n.rb:265:in
  `with_locale'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/translation.rb:7:in
  `block (2 levels) in <module:Translation>'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in
  `instance_exec'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in
  `block in run_callbacks'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/logging.rb:24:in
  `block (4 levels) in <module:Logging>'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in
  `block in instrument'" 

I'm not sure what's causing that so filing under dynflow component for now.


Related issues

Related to Foreman - Bug #23611: ApplicationJob starts when foreman_tasks does not existClosed2018-05-16

Associated revisions

Revision 30203315 (diff)
Added by Adam Ruzicka over 1 year ago

Fixes #23326 - Create active jobs properly

Revision 9ab0d774 (diff)
Added by Adam Ruzicka over 1 year ago

Fixes #23326 - Version bumping

History

#1 Updated by Ivan Necas over 1 year ago

  • Category changed from Dynflow to Notifications
  • Project changed from foreman-tasks to Katello

This belongs to the Katello project

#2 Updated by Andrew Kofink over 1 year ago

  • Legacy Backlogs Release (now unused) set to 349

#3 Updated by Andrew Kofink over 1 year ago

  • Legacy Backlogs Release (now unused) changed from 349 to 338
  • Assignee set to Jonathon Turel

#4 Updated by Jonathon Turel over 1 year ago

I looked into this and I think there's an issue around the timing of the foreman and foreman_tasks being installed and started. I also didn't see this problem exactly as described which also indicates a timing related issue to me.

From foreman production.log:

2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "foreman_tasks_tasks" does not exist
LINE 8:                WHERE a.attrelid = '"foreman_tasks_tasks"'::r...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

From foreman_proxy proxy.log:

# Logfile created on 2018-05-03 00:54:14 +0000 by logger.rb/41954
I, [2018-05-03T00:54:14.761880 ]  INFO -- : Successfully initialized 'pulp'
I, [2018-05-03T00:54:14.761989 ]  INFO -- : Successfully initialized 'foreman_proxy'
I, [2018-05-03T00:54:14.762027 ]  INFO -- : Successfully initialized 'templates'
I, [2018-05-03T00:54:14.762060 ]  INFO -- : Successfully initialized 'tftp'
I, [2018-05-03T00:54:14.762089 ]  INFO -- : Successfully initialized 'puppetca'
I, [2018-05-03T00:54:14.762250 ]  INFO -- : Started puppet class cache initialization
I, [2018-05-03T00:54:14.769301 ]  INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2018-05-03T00:54:14.769425 ]  INFO -- : Successfully initialized 'puppet'
I, [2018-05-03T00:54:14.769486 ]  INFO -- : Successfully initialized 'logs'
I, [2018-05-03T00:54:14.783040 ]  INFO -- : WEBrick 1.3.1
I, [2018-05-03T00:54:14.783123 ]  INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux]
I, [2018-05-03T00:54:14.783934 ]  INFO -- :

I'm not very familiar with the installer side of things but it looks like foreman is starting and kicking off the jobs when foreman_tasks hasn't been installed & started - see that there is a 4 minute difference in the logs. I will think about what the appropriate fix is and update.

#5 Updated by Jonathon Turel over 1 year ago

The proxy log is irrelevant - ignore it :)

#6 Updated by Jonathon Turel over 1 year ago

  • Related to Bug #23611: ApplicationJob starts when foreman_tasks does not exist added

#7 Updated by The Foreman Bot over 1 year ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/7378 added

#8 Updated by Adam Ruzicka over 1 year ago

  • % Done changed from 0 to 100
  • Status changed from Ready For Testing to Closed

Also available in: Atom PDF