Bug #18363
closedActions::Pulp::Repository::Destroy (error) after deleting repository
Description
Hi,
after deleting a repository I get errors like these after the next daily sync:
Actions::Pulp::Repository::Destroy (error) Started at: 2017-02-02 00:19:13 UTC Ended at: 2017-02-02 08:25:06 UTC Real time: 29153.60s Execution time (excluding suspended state): 0.30s Input: --- pulp_id: ORG-Library-Product1-Repository1 capsule_id: 3 remote_user: admin remote_cp_user: admin locale: en Output: --- {} Error: RestClient::ResourceNotFound 404 Resource Not Found --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:79:in `block in get_response'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in `process_result'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'" - "/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/resource.rb:94:in `delete'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:78:in `get_response'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:66:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/resources/repository.rb:48:in `delete'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/pulp/repository/destroy.rb:11:in `invoke_external_task'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:83:in `initiate_external_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:18:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/cancellable.rb:9:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:30:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:16:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:16:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:30:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/world.rb:30:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in `block in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `block in with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:499:in `execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:260:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/actor.rb:26:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'"
and pulp_id: ORG-Product1-Repository1
but the rest of the error message is the same
I can skip the erroring operations in dynflow console and resume the task and it seems to work normally again during the next nightly sync.
Greetings
Klaas
Updated by Justin Sherrill almost 8 years ago
- Status changed from New to Need more information
I suspect that this error is actually overwriting another error. If you delete the repository and go to Monitor > tasks and find the repository delete task:
1) is it still running?
2) if not, is there any error here (particular the pulp repo delete step)?
Updated by Klaas D almost 8 years ago
6: Actions::Pulp::Repository::Destroy (error) [ 25638.51s / 3.07s ] Skip Started at: 2017-02-09 01:03:50 UTC Ended at: 2017-02-09 08:11:08 UTC Real time: 25638.51s Execution time (excluding suspended state): 3.07s Input: --- pulp_id: ORG-Foreman-Foreman_1_11_EL7 capsule_id: 7 remote_user: admin remote_cp_user: admin locale: en Output: --- pulp_tasks: - exception: task_type: pulp.server.tasks.repository.delete _href: "/pulp/api/v2/tasks/567ec4af-2f1a-416e-a2ca-fc75fb10fef4/" task_id: 567ec4af-2f1a-416e-a2ca-fc75fb10fef4 tags: - pulp:repository:ORG-Foreman-Foreman_1_11_EL7 - pulp:action:delete finish_time: '2017-02-09T01:03:58Z' _ns: task_status start_time: '2017-02-09T01:03:58Z' traceback: | Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 484, in __call__ return super(Task, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 103, in __call__ return super(PulpTask, self).__call__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 525, in delete repo = model.Repository.objects.get_repo_or_missing_resource(repo_id) File "/usr/lib/python2.7/site-packages/pulp/server/db/querysets.py", line 139, in get_repo_or_missing_resource raise pulp_exceptions.MissingResource(repository=repo_id) MissingResource: Missing resource(s): repository=ORG-Foreman-Foreman_1_11_EL7 spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-3@pulpcapsule1.tld.dq state: error worker_name: reserved_resource_worker-3@pulpcapsule1.tld result: error: code: PLP0009 data: resources: repository: ORG-Foreman-Foreman_1_11_EL7 description: 'Missing resource(s): repository=ORG-Foreman-Foreman_1_11_EL7' sub_errors: [] _id: "$oid": 589bbff77c9cd8ae9e14862a id: 589bbff77c9cd8ae9e14862a poll_attempts: total: 2 failed: 2 Error: RestClient::ResourceNotFound 404 Resource Not Found --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:79:in `block in get_response'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:493:in `process_result'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:421:in `block in transmit'" - "/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:413:in `transmit'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:176:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.8.0/lib/restclient/resource.rb:94:in `delete'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:78:in `get_response'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/base.rb:66:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.9.0/lib/runcible/resources/repository.rb:48:in `delete'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/pulp/repository/destroy.rb:11:in `invoke_external_task'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:83:in `initiate_external_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:89:in `rescue in resume_external_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:87:in `resume_external_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/polling.rb:16:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/cancellable.rb:9:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:506:in `block (3 levels) in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:30:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/remote_action.rb:16:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action/progress.rb:16:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.2.1.1/app/lib/actions/middleware/keep_locale.rb:11:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:26:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:17:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware.rb:30:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/stack.rb:22:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/middleware/world.rb:30:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:504:in `block in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `block in with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:419:in `with_error_handling'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:499:in `execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/action.rb:260:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.15/lib/dynflow/actor.rb:26:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'"
that one is part of the task:
Id: bc962d7f-b264-4d7f-868d-d4545cedb87c Label: Actions::Katello::Repository::CapsuleGenerateAndSync Name: Sync Repository on Capsule(s) Owner: foreman_admin Execution type: Delayed Start at: 2017-02-09 02:03:27 +0100 Start before: - Started at: 2017-02-09 02:03:27 +0100 Ended at: State: paused Result: error Params: {"services_checked"=>["pulp", "pulp_auth"], "locale"=>"en"}
that task was called because of the nightly repository sync (not sure which one actually causes that sync to run, I sync like 20 repos)
the delete task:
Id: cdf4a007-40fd-4482-bd9c-71b699dd787e Label: Actions::Katello::Repository::Destroy Name: Delete Owner: admin Execution type: Delayed Start at: 2017-02-08 12:49:32 +0100 Start before: - Started at: 2017-02-08 12:49:32 +0100 Ended at: 2017-02-08 12:49:36 +0100 State: stopped Result: success Params: repository 'Foreman 1.11 EL7'; product 'Foreman'; organization 'ORG' Dynflow Details: 3: Actions::Pulp::Repository::Destroy (success) [ 1.81s / 0.27s ] Started at: 2017-02-08 11:49:34 UTC Ended at: 2017-02-08 11:49:36 UTC Real time: 1.81s Execution time (excluding suspended state): 0.27s Input: --- pulp_id: ORG-Foreman-Foreman_1_11_EL7 remote_user: admin remote_cp_user: admin locale: en Output: --- pulp_tasks: - exception: task_type: pulp.server.tasks.repository.delete _href: "/pulp/api/v2/tasks/76567c34-a7e3-4c4f-94e2-638f0b71b0ce/" task_id: 76567c34-a7e3-4c4f-94e2-638f0b71b0ce tags: - pulp:repository:ORG-Foreman-Foreman_1_11_EL7 - pulp:action:delete finish_time: '2017-02-08T11:49:35Z' _ns: task_status start_time: '2017-02-08T11:49:35Z' traceback: spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-4@foreman.tld.dq state: finished worker_name: reserved_resource_worker-4@foreman.tld result: error: _id: "$oid": 589b05ce805ea3675455154f id: 589b05ce805ea3675455154f poll_attempts: total: 1 failed: 0 6: Actions::Candlepin::Product::ContentRemove (success) [ 0.10s / 0.10s ] Started at: 2017-02-08 11:49:34 UTC Ended at: 2017-02-08 11:49:34 UTC Real time: 0.10s Execution time (excluding suspended state): 0.10s Input: --- product_id: '1469701488937' content_id: '1469701492484' remote_user: admin remote_cp_user: admin locale: en Output: --- response: 204 8: Actions::Candlepin::Product::ContentDestroy (success) [ 0.10s / 0.10s ] Started at: 2017-02-08 11:49:34 UTC Ended at: 2017-02-08 11:49:35 UTC Real time: 0.10s Execution time (excluding suspended state): 0.10s Input: --- content_id: '1469701492484' remote_user: admin remote_cp_user: admin locale: en Output: --- response: 204
parent task:
Id: d80d116c-a60c-442c-bc58-ea73bbb62065 Label: Actions::BulkAction Name: Delete Owner: admin Execution type: Delayed Start at: 2017-02-08 12:49:30 +0100 Start before: - Started at: 2017-02-08 12:49:31 +0100 Ended at: 2017-02-08 12:49:42 +0100 State: stopped Result: success Params: delete; repository 'Foreman 1.12 EL7'; product 'Foreman'; organization 'ORG'; ...
There was no pulp sync directly after the delete, not sure why that didn't happen :)
There are more than one sync task that ran tonight, here is the one that actually ran successful:
Id: 3074affe-65d5-4af0-840d-bdca90bca7f6 Label: Actions::Katello::Repository::CapsuleGenerateAndSync Name: Sync Repository on Capsule(s) Owner: foreman_admin Execution type: Delayed Start at: 2017-02-09 02:03:22 +0100 Start before: - Started at: 2017-02-09 02:03:23 +0100 Ended at: 2017-02-09 02:08:14 +0100 State: stopped Result: success Params: {"services_checked"=>["pulp", "pulp_auth"], "locale"=>"en"} Dynflow details of first destroy: 6: Actions::Pulp::Repository::Destroy (success) [ 22.27s / 0.26s ] Started at: 2017-02-09 01:03:47 UTC Ended at: 2017-02-09 01:04:09 UTC Real time: 22.27s Execution time (excluding suspended state): 0.26s Input: --- pulp_id: ORG-Foreman-Foreman_1_11_EL7 capsule_id: 7 remote_user: admin remote_cp_user: admin locale: en Output: --- pulp_tasks: - exception: task_type: pulp.server.tasks.repository.delete _href: "/pulp/api/v2/tasks/5c2a09e5-2cee-4ab6-b277-28dec77888c0/" task_id: 5c2a09e5-2cee-4ab6-b277-28dec77888c0 tags: - pulp:repository:ORG-Foreman-Foreman_1_11_EL7 - pulp:action:delete finish_time: '2017-02-09T01:03:55Z' _ns: task_status start_time: '2017-02-09T01:03:54Z' traceback: spawned_tasks: [] progress_report: {} queue: reserved_resource_worker-1@pulpcapsule1.srv.muenchen.de.dq state: finished worker_name: reserved_resource_worker-1@pulpcapsule1.srv.muenchen.de result: error: _id: "$oid": 589bbff37c9cd8ae9e148624 id: 589bbff37c9cd8ae9e148624 poll_attempts: total: 1 failed: 0
Updated by Eric Helms almost 8 years ago
- Category set to Repositories
- Status changed from Need more information to Needs design
- Assignee set to Justin Sherrill
Updated by John Mitsch over 7 years ago
- Translation missing: en.field_release set to 228
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 228 to 258
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release changed from 258 to 267
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release changed from 267 to 281
Updated by Justin Sherrill over 7 years ago
- Translation missing: en.field_release changed from 281 to 286
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 286 to 295
Updated by Ewoud Kohl van Wijngaarden about 6 years ago
- Target version deleted (
Katello 3.4.6)
Updated by Ewoud Kohl van Wijngaarden over 1 year ago
- Status changed from Needs design to Rejected
This is from Pulp 2, so I'm going to close this now.