Plugins » foreman-tasks

I am seeing this in Katello on the latest dev environment as of this report. I put some more info here https://gist.github.com/johnpmitsch/e5e7346fc32a86465d4e9031155170cb and am reposting the traceback below.

To reproduce, I will enable and disable RH repositories. It doesn't always happen right away, but once the task gets stuck it stays in this state and no other tasks can be run

versions:
dynflow (1.3.0)
smart_proxy_dynflow (0.2.3)
smart_proxy_dynflow_core (0.2.2)
foreman-tasks (0.17.0)
foreman-tasks-core (0.3.3)

2019-09-04T18:01:58 [E|dyn|f45e65f7] invalid state transition pending >> planned in #<Dynflow::ExecutionPlan:0x00007f8b1a229848> (RuntimeError)
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/stateful.rb:35:in `set_state'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/stateful.rb:28:in `state='
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/execution_plan.rb:122:in `update_state'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:39:in `block in invalidate_planning_lock'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:103:in `with_valid_execution_plan_for_lock'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:35:in `invalidate_planning_lock'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:16:in `block (2 levels) in invalidate'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:15:in `each'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:15:in `block in invalidate'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/coordinator.rb:325:in `acquire'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:13:in `invalidate'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:135:in `block in worlds_validity_check'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:129:in `each'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:129:in `worlds_validity_check'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/world/invalidation.rb:110:in `perform_validity_checks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/rails.rb:45:in `block in initialize!'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/rails.rb:37:in `tap'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/rails.rb:37:in `initialize!'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/dynflow-1.3.0/lib/dynflow/rails.rb:72:in `world'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:23:in `trigger'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:29:in `block in trigger_task'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:27:in `trigger_task'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks.rb:58:in `sync_task'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/foreman-tasks-0.16.0/lib/foreman_tasks/triggers.rb:22:in `sync_task'
 | /home/vagrant/katello/app/controllers/katello/api/v2/repository_sets_controller.rb:44:in `available_repositories'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /home/vagrant/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /home/vagrant/katello/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/static_dispatcher.rb:66:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/bullet-6.0.2/lib/bullet/rack.rb:15:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/extractor/recorder.rb:137:in `call'
 | /home/vagrant/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/apipie-rails-0.5.16/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /home/vagrant/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-openid-1.4.2/lib/rack/openid.rb:98:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /home/vagrant/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/migration.rb:559:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /home/vagrant/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /home/vagrant/katello/lib/katello/prevent_json_parsing.rb:12:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/secure_headers-6.1.1/lib/secure_headers/middleware.rb:11:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/puma-4.1.0/lib/puma/configuration.rb:228:in `call'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/puma-4.1.0/lib/puma/server.rb:664:in `handle_request'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/puma-4.1.0/lib/puma/server.rb:467:in `process_client'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/puma-4.1.0/lib/puma/server.rb:328:in `block in run'
 | /home/vagrant/foreman/.vendor/ruby/2.5.0/gems/puma-4.1.0/lib/puma/thread_pool.rb:135:in `block in spawn_thread'
 | [ concurrent-ruby ]