Plugins » Katello

I have a capsule that had previously run out of space. After I fixed that problem, I clicked the Syncronize button on the Smart Proxy Overview page. After letting it run over the long 3 day weekend, where all but two steps completed, I restarted pulp on the capsule which cancelled the sync.

Here is the output from the sync step that was cancelled:

Started at: 2016-09-02 12:07:00 UTC

Ended at: 2016-09-06 12:00:41 UTC

Real time: 345221.79s

Execution time (excluding suspended state): 5412.99s

Input:

---
capsule_id: 5
repo_pulp_id: Organization-PROD-RHEL_6-EPEL_-_EL6-EPEL_-_EL6
remote_user: admin
remote_cp_user: admin

Output:

---
pulp_tasks:
- exception:
task_type: pulp.server.managers.repo.sync.sync
href: "/pulp/api/v2/tasks/1460795d-aa01-4a05-9f2d-7e28a5c8f44b/"
task_id: 1460795d-aa01-4a05-9f2d-7e28a5c8f44b
tags:
- pulp:repository:Organization-PROD-RHEL_6-EPEL-_EL6-EPEL_-_EL6
- pulp:action:sync
finish_time:
_ns: task_status
start_time: '2016-09-02T12:07:00Z'
traceback:
spawned_tasks: []
progress_report:
yum_importer:
content:
items_total: 0
state: FINISHED
error_details: []
details:
rpm_total: 0
rpm_done: 0
drpm_total: 0
drpm_done: 0
size_total: 0
size_left: 0
items_left: 0
comps:
state: FINISHED
purge_duplicates:
state: FINISHED
distribution:
items_total: 0
state: FINISHED
error_details: []
items_left: 0
errata:
state: FINISHED
metadata:
state: FINISHED
queue: reserved_resource_worker-0@capsule.domain.tld.dq
state: canceled
worker_name: reserved_resource_worker-0@capsule.domain.tld
result:
error:
_id:
"$oid": 57c96b644010ad626e927687
id: 57c96b644010ad626e927687
poll_attempts:
total: 21240
failed: 1

Error:

Katello::Errors::PulpError

Task canceled

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:121:in
`block in external_task='"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in
`external_task='"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in
`poll_external_task_with_rescue'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:45:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in
`block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:40:in
`block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/models/katello/concerns/user_extensions.rb:20:in
`cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:27:in
`as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:39:in
`as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in
`with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:11:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:22:in
`with_locale'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:11:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in
`block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in
`execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
`block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
`block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
`open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in
`block in on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
`block in assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`tap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
`match_value'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
`block in match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
`match'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in
`on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in
`process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
`process_envelopes?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
`process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in
`block in on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
`block (2 levels) in schedule_execution'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`block in synchronize'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in
`block in schedule_execution'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in
`work'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
`block in call_job'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`run_task'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in
`block (3 levels) in create_worker'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`loop'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`block (2 levels) in create_worker'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`catch'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`block in create_worker'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'"

My questions are, 1) What caused this step to fail in the first place, and 2) what is the correct way to clean it up so the next sync doesn't hang also?