Project

General

Profile

Actions

Bug #23326

closed

multiple paused state "Create Pulpdiskspace" notifications is beng triggered in dynflow

Added by Sachin Ghai about 6 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Category:
Notifications
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

On a nightly install, we see many tasks under dynflow around "CreatePulpDiskSpaceNotifications"

Error:

NoMethodError

undefined method `storage' for nil:NilClass

- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:8:in
  `block in deliver!'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in
  `each'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/delegation.rb:39:in
  `each'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/services/katello/ui_notifications/pulp/proxy_disk_space.rb:7:in
  `deliver!'" 
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/jobs/create_pulp_disk_space_notifications.rb:7:in
  `perform'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/execution.rb:37:in
  `block in perform_now'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in
  `block in run_callbacks'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/i18n-0.9.1/lib/i18n.rb:265:in
  `with_locale'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/translation.rb:7:in
  `block (2 levels) in <module:Translation>'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in
  `instance_exec'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in
  `block in run_callbacks'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activejob-5.1.4/lib/active_job/logging.rb:24:in
  `block (4 levels) in <module:Logging>'" 
- "/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in
  `block in instrument'" 

I'm not sure what's causing that so filing under dynflow component for now.


Related issues 1 (0 open1 closed)

Related to Foreman - Bug #23611: ApplicationJob starts when foreman_tasks does not existClosedAdam Ruzicka05/16/2018Actions
Actions #1

Updated by Ivan Necas almost 6 years ago

  • Project changed from foreman-tasks to Katello
  • Category changed from Dynflow to Notifications

This belongs to the Katello project

Actions #2

Updated by Andrew Kofink almost 6 years ago

  • translation missing: en.field_release set to 349
Actions #3

Updated by Andrew Kofink almost 6 years ago

  • Assignee set to Jonathon Turel
  • translation missing: en.field_release changed from 349 to 338
Actions #4

Updated by Jonathon Turel almost 6 years ago

I looked into this and I think there's an issue around the timing of the foreman and foreman_tasks being installed and started. I also didn't see this problem exactly as described which also indicates a timing related issue to me.

From foreman production.log:

2018-05-03T00:50:19 [W|dyn|] Error on on_execution_plan_save event
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "foreman_tasks_tasks" does not exist
LINE 8:                WHERE a.attrelid = '"foreman_tasks_tasks"'::r...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
                     c.collname, col_description(a.attrelid, a.attnum) AS comment
                FROM pg_attribute a
                LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
                LEFT JOIN pg_type t ON a.atttypid = t.oid
                LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
               WHERE a.attrelid = '"foreman_tasks_tasks"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

From foreman_proxy proxy.log:

# Logfile created on 2018-05-03 00:54:14 +0000 by logger.rb/41954
I, [2018-05-03T00:54:14.761880 ]  INFO -- : Successfully initialized 'pulp'
I, [2018-05-03T00:54:14.761989 ]  INFO -- : Successfully initialized 'foreman_proxy'
I, [2018-05-03T00:54:14.762027 ]  INFO -- : Successfully initialized 'templates'
I, [2018-05-03T00:54:14.762060 ]  INFO -- : Successfully initialized 'tftp'
I, [2018-05-03T00:54:14.762089 ]  INFO -- : Successfully initialized 'puppetca'
I, [2018-05-03T00:54:14.762250 ]  INFO -- : Started puppet class cache initialization
I, [2018-05-03T00:54:14.769301 ]  INFO -- : Successfully initialized 'puppet_proxy_puppet_api'
I, [2018-05-03T00:54:14.769425 ]  INFO -- : Successfully initialized 'puppet'
I, [2018-05-03T00:54:14.769486 ]  INFO -- : Successfully initialized 'logs'
I, [2018-05-03T00:54:14.783040 ]  INFO -- : WEBrick 1.3.1
I, [2018-05-03T00:54:14.783123 ]  INFO -- : ruby 2.0.0 (2015-12-16) [x86_64-linux]
I, [2018-05-03T00:54:14.783934 ]  INFO -- :

I'm not very familiar with the installer side of things but it looks like foreman is starting and kicking off the jobs when foreman_tasks hasn't been installed & started - see that there is a 4 minute difference in the logs. I will think about what the appropriate fix is and update.

Actions #5

Updated by Jonathon Turel almost 6 years ago

The proxy log is irrelevant - ignore it :)

Actions #6

Updated by Jonathon Turel almost 6 years ago

  • Related to Bug #23611: ApplicationJob starts when foreman_tasks does not exist added
Actions #7

Updated by The Foreman Bot almost 6 years ago

  • Status changed from New to Ready For Testing
  • Pull request https://github.com/Katello/katello/pull/7378 added
Actions #8

Updated by Adam Ruzicka almost 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions

Also available in: Atom PDF