Bug #25723
Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks
Pull request:
Fixed in Releases:
Found in Releases:
Description
Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1654347
Description of problem:
Foreman may create duplicate CreateRssNotifications, CreatePulpDiskSpaceNotifications or SendExpireSoonNotifications when restart foreman-tasks.
- su - postgres
c "psql -d foreman -c 'select label,count(label),state from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state;'"------------------
label | count | state
---------------------------------------
CreateRssNotifications | 2 | scheduled
CreatePulpDiskSpaceNotifications | 1 | scheduled
SendExpireSoonNotifications | 1 | scheduled
The "spawn_if_missing" function is looking for task that is in "scheduled" state during the dynflow startup. If no "CreateRssNotifications" is in "schdeduled" state, then create one. I suspect that the task was not in "scheduled" state previously before Dynflow got terminated and cause Dynflow to create a duplicate task during startup.
/usr/share/foreman/app/jobs/application_job.rb
def self.spawn_if_missing(world)
return if (Foreman.in_rake? && !Foreman.in_rake?('dynflow:executor')) || Rails.env.test?
pending_jobs = world.persistence.find_execution_plans(filters: { :state => 'scheduled' })
scheduled_job = pending_jobs.select do |job|
delayed_plan = world.persistence.load_delayed_plan job.id
next unless delayed_plan.present?
arg = delayed_plan.to_hash[:serialized_args].first
arg.is_a?(Hash) && arg['job_class'] == self.to_s
end
- Schedule the job only if it doesn't exit yet
self.perform_later if scheduled_job.blank?
end
Associated revisions
History
#1
Updated by Ondřej Ezr 9 months ago
- Subject changed from Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks to Foreman may create duplicate CreateRssNotifications tasks after restarting foreman tasks
- Project changed from foreman-tasks to Foreman
Moving this to foreman as the related code lives in foreman.
#2
Updated by Adam Ruzicka 9 months ago
- Assignee set to Adam Ruzicka
- Status changed from New to Ready For Testing
#3
Updated by The Foreman Bot 9 months ago
- Pull request https://github.com/theforeman/foreman/pull/7618 added
#4
Updated by The Foreman Bot 9 months ago
- Fixed in Releases 2.1.0 added
#5
Updated by Adam Ruzicka 9 months ago
- Status changed from Ready For Testing to Closed
Applied in changeset 668853b9abf0fc10453e1c2c256caf69d5d92492.
#6
Updated by Amit Upadhye 8 months ago
- Category set to Notifications
Fixes #25723 - Prevent AJ.spawn_if_missing from creating duplicates
Previously we only checked execution plans which were in scheduled state. This
left us with a small window where the ActiveJob was actually present, but we
didn't detect it so we spawned a new one.
With this patch, we consider execution plans in states scheduled, planning,
planned and running as present and plans in paused and stopped states as
missing. The only state which is not covered is pending, but in that state the
execution plan doesn't have enough data and we cannot reliably identify it.
This is not perfect, it can still happen that two processes get initialized at
roughly the same time, go through all the checks and then both create a new job.
We cannot solve this with the data we already store and we would have to
introduce some kind of locking, similar to what we did for singleton actions.