Project

General

Profile

Bug #16448

Capsule Sync hang

Added by Brian Shaw over 5 years ago. Updated almost 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Difficulty:
Triaged:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

I have a capsule that had previously run out of space. After I fixed that problem, I clicked the Syncronize button on the Smart Proxy Overview page. After letting it run over the long 3 day weekend, where all but two steps completed, I restarted pulp on the capsule which cancelled the sync.

Here is the output from the sync step that was cancelled:

Started at: 2016-09-02 12:07:00 UTC

Ended at: 2016-09-06 12:00:41 UTC

Real time: 345221.79s

Execution time (excluding suspended state): 5412.99s

Input:

---
capsule_id: 5
repo_pulp_id: Organization-PROD-RHEL_6-EPEL_-_EL6-EPEL_-_EL6
remote_user: admin
remote_cp_user: admin

Output:

---
pulp_tasks:
- exception:
task_type: pulp.server.managers.repo.sync.sync
href: "/pulp/api/v2/tasks/1460795d-aa01-4a05-9f2d-7e28a5c8f44b/"
task_id: 1460795d-aa01-4a05-9f2d-7e28a5c8f44b
tags:
- pulp:repository:Organization-PROD-RHEL_6-EPEL
-_EL6-EPEL_-_EL6
- pulp:action:sync
finish_time:
_ns: task_status
start_time: '2016-09-02T12:07:00Z'
traceback:
spawned_tasks: []
progress_report:
yum_importer:
content:
items_total: 0
state: FINISHED
error_details: []
details:
rpm_total: 0
rpm_done: 0
drpm_total: 0
drpm_done: 0
size_total: 0
size_left: 0
items_left: 0
comps:
state: FINISHED
purge_duplicates:
state: FINISHED
distribution:
items_total: 0
state: FINISHED
error_details: []
items_left: 0
errata:
state: FINISHED
metadata:
state: FINISHED
queue:
state: canceled
worker_name:
result:
error:
_id:
"$oid": 57c96b644010ad626e927687
id: 57c96b644010ad626e927687
poll_attempts:
total: 21240
failed: 1

Error:

Katello::Errors::PulpError

Task canceled

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:121:in
`block in external_task='"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in
`external_task='"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:98:in
`poll_external_task_with_rescue'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/polling.rb:21:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/cancellable.rb:9:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:45:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:506:in
`block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:40:in
`block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/models/katello/concerns/user_extensions.rb:20:in
`cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:27:in
`as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:39:in
`as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in
`with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:11:in
`block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:22:in
`with_locale'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0.rc2.1/app/lib/actions/middleware/keep_locale.rb:11:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in
`run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in
`block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in
`block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in
`with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in
`execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in
`block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in
`with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
`block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in
`open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
`execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in
`block in on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
`block in assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`tap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
`assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
`match_value'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
`block in match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
`match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
`match'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in
`on_message'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in
`process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in
`process_envelopes?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in
`pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
`on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in
`process_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in
`block in on_envelope'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in
`block (2 levels) in schedule_execution'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`block in synchronize'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
`synchronize'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in
`block in schedule_execution'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in
`work'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in
`block in call_job'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`call'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
`run_task'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in
`block (3 levels) in create_worker'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`loop'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
`block (2 levels) in create_worker'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`catch'"
- "/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
`block in create_worker'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'"

My questions are, 1) What caused this step to fail in the first place, and 2) what is the correct way to clean it up so the next sync doesn't hang also?

History

#1 Updated by Justin Sherrill over 5 years ago

  • Tracker changed from Support to Bug
  • Status changed from New to Need more information

Did you notice any errors in /var/log/messages on the capsule?

This seems like it may be a qpid/pulp issue if its workers are loosing work.

#2 Updated by Brian Shaw over 5 years ago

Justin,
This is the only error that is jumping out at me for either qpid or pulp during this time frame.

Sep 2 20:01:51 pcolmgxa15 pulp: pulp.server.db.reaper:INFO: The reaper task is cleaning out old documents from the database.
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688) /usr/lib/python2.7/site-packages/pulp/server/db/model/reaper_base.py:35: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688) collection.remove({'_id': {'$lte': expired_object_id}})
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688)
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688) /usr/lib/python2.7/site-packages/pulp/server/db/model/celery_result.py:30: DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688) collection.remove({'date_done': {'$lt': last_valid_date_done}})
Sep 2 20:01:51 pcolmgxa15 pulp: py.warnings:WARNING: (3382-22688)
Sep 2 20:01:51 pcolmgxa15 pulp: pulp.server.db.reaper:INFO: The reaper task has completed.

Brian

#3 Updated by Justin Sherrill over 5 years ago

Can you try restarting pulp services on the capsule, kick off a new sync, and upload a fresh foreman-debug from the katello server and capsule?

#4 Updated by Eric Helms over 5 years ago

  • Legacy Backlogs Release (now unused) set to 166

#5 Updated by Justin Sherrill over 5 years ago

  • Status changed from Need more information to Rejected

Closing, feel free to reopen if this is still an issue

Also available in: Atom PDF