Project

General

Profile

Bug #25723

Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks

Added by Adam Ruzicka almost 2 years ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Notifications
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1654347

Description of problem:
Foreman may create duplicate CreateRssNotifications, CreatePulpDiskSpaceNotifications or SendExpireSoonNotifications when restart foreman-tasks.

  1. su - postgres c "psql -d foreman -c 'select label,count(label),state from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state;'"
    label | count | state
    ---------------------------------------
    ------------------
    CreateRssNotifications | 2 | scheduled
    CreatePulpDiskSpaceNotifications | 1 | scheduled
    SendExpireSoonNotifications | 1 | scheduled

The "spawn_if_missing" function is looking for task that is in "scheduled" state during the dynflow startup. If no "CreateRssNotifications" is in "schdeduled" state, then create one. I suspect that the task was not in "scheduled" state previously before Dynflow got terminated and cause Dynflow to create a duplicate task during startup.

/usr/share/foreman/app/jobs/application_job.rb

def self.spawn_if_missing(world)
return if (Foreman.in_rake? && !Foreman.in_rake?('dynflow:executor')) || Rails.env.test?
pending_jobs = world.persistence.find_execution_plans(filters: { :state => 'scheduled' })
scheduled_job = pending_jobs.select do |job|
delayed_plan = world.persistence.load_delayed_plan job.id
next unless delayed_plan.present?
arg = delayed_plan.to_hash[:serialized_args].first
arg.is_a?(Hash) && arg['job_class'] == self.to_s
end
  1. Schedule the job only if it doesn't exit yet
    self.perform_later if scheduled_job.blank?
    end

Associated revisions

Revision 668853b9 (diff)
Added by Adam Ruzicka 6 months ago

Fixes #25723 - Prevent AJ.spawn_if_missing from creating duplicates

Previously we only checked execution plans which were in scheduled state. This
left us with a small window where the ActiveJob was actually present, but we
didn't detect it so we spawned a new one.

With this patch, we consider execution plans in states scheduled, planning,
planned and running as present and plans in paused and stopped states as
missing. The only state which is not covered is pending, but in that state the
execution plan doesn't have enough data and we cannot reliably identify it.

This is not perfect, it can still happen that two processes get initialized at
roughly the same time, go through all the checks and then both create a new job.
We cannot solve this with the data we already store and we would have to
introduce some kind of locking, similar to what we did for singleton actions.

History

#1 Updated by Ondřej Ezr 6 months ago

  • Subject changed from Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks to Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks
  • Project changed from foreman-tasks to Foreman

Moving this to foreman as the related code lives in foreman.

#2 Updated by Adam Ruzicka 6 months ago

  • Assignee set to Adam Ruzicka
  • Status changed from New to Ready For Testing

#3 Updated by The Foreman Bot 6 months ago

  • Pull request https://github.com/theforeman/foreman/pull/7618 added

#4 Updated by The Foreman Bot 6 months ago

  • Fixed in Releases 2.1.0 added

#5 Updated by Adam Ruzicka 6 months ago

  • Status changed from Ready For Testing to Closed

#6 Updated by Amit Upadhye 5 months ago

  • Category set to Notifications

Also available in: Atom PDF