Bug #20073
closed
After upgrade from 3.4 to 3.4.1 Can't sync redhat repos error [E] RPM1004
Description
2017-06-21 15:03:49 9e204935 [app] [I] Completed 200 OK in 41ms (Views: 0.2ms | ActiveRecord: 4.2ms)
2017-06-21 15:03:49 [foreman-tasks/action] [E] RPM1004: Error retrieving metadata: Forbidden (Katello::Errors::PulpError)
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/pulp/repository/sync.rb:48:in `external_task='
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/polling.rb:98:in `poll_external_task_with_rescue'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/polling.rb:21:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/cancellable.rb:9:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:512:in `block (3 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action/progress.rb:16:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.0.2/app/lib/actions/middleware/keep_locale.rb:11:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:26:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:17:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware.rb:30:in `run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/stack.rb:22:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/middleware/world.rb:30:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:511:in `block (2 levels) in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:510:in `block in execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `block in with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:425:in `with_error_handling'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:505:in `execute_run'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/action.rb:266:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/director.rb:55:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/executors/parallel/worker.rb:11:in `on_message'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/context.rb:46:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.24/lib/dynflow/actor.rb:26:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:161:in `process_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.3/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:18:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:96:in `work'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.3/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-06-21 15:03:50 [app] [I] Current user: foreman_admin (administrator)
2017-06-21 15:03:50 6ab18e03 [app] [I] Started GET "/katello/sync_management/sync_status?repoids%5B%5D=375&_=1498050210915" for 10.210.140.138 at 2017-06-21 15:03:50 +0200
2017-06-21 15:03:50 6ab18e03 [app] [I] Processing by Katello::SyncManagementController#sync_status as JSON
2017-06-21 15:03:50 6ab18e03 [app] [I] Parameters: {"repoids"=>["375"], "_"=>"1498050210915"}
2017-06-21 15:03:50 6ab18e03 [app] [I] Current user: MGamboa (administrator)
2017-06-21 15:03:50 6ab18e03 [app] [I] Completed 200 OK in 44ms (Views: 0.2ms | ActiveRecord: 3.7ms)
2017-06-21 15:03:51 54a446bd [app] [I] Started GET "/katello/sync_management/sync_status?repoids%5B%5D=375&_=1498050210916" for 10.210.140.138 at 2017-06-21 15:03:51 +0200
2017-06-21 15:03:51 54a446bd [app] [I] Processing by Katello::SyncManagementController#sync_status as JSON
2017-06-21 15:03:51 54a446bd [app] [I] Parameters: {"repoids"=>["375"], "_"=>"1498050210916"}
2017-06-21 15:03:51 54a446bd [app] [I] Current user: MGamboa (administrator)
Updated by Justin Sherrill over 7 years ago
Can you upload your manifest to the foreman-debug rsync repo, or email it to
jsherril <~at~> redhat.com
maybe we can reproduce these issues with your manifest.
Updated by Mario Gamboa over 7 years ago
Here is the logs
Jun 22 10:44:35 satellite pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Jun 22 10:44:35 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[288e877d-1639-4161-b9b2-6d5e2b9478e5]
Jun 22 10:44:35 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[dc728f97-4c01-406a-8ec5-31bf5e365cd3]
Jun 22 10:44:35 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[9edeb5df-18e4-4d65-9533-74d430e56a59]
Jun 22 10:44:35 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/extras/os/.
Jun 22 10:44:35 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:35 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[288e877d-1639-4161-b9b2-6d5e2b9478e5] succeeded in 0.0364646410017s: None
Jun 22 10:44:36 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:36 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/extras/os/.
Jun 22 10:44:36 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:37 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:38 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/extras/os/.
Jun 22 10:44:38 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:39 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:39 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/extras/os.
Jun 22 10:44:39 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:40 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Jun 22 10:44:40 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:41 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Jun 22 10:44:42 satellite pulp: pulp.server.async.tasks:INFO: Task failed : [dc728f97-4c01-406a-8ec5-31bf5e365cd3] : Error retrieving metadata: Forbidden
Jun 22 10:44:42 satellite pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[dc728f97-4c01-406a-8ec5-31bf5e365cd3] raised expected: PulpCodedException()
Jun 22 10:44:42 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[9edeb5df-18e4-4d65-9533-74d430e56a59] succeeded in 0.00404729800357s: None
Updated by Mario Gamboa over 7 years ago
Using pulp-admin same result
pulp-admin u admin -p $pulpAdminPassword rpm repo sync run --repo-id=f54312f5-d543-4ccc-a591-4a75d1567d72
---------------------------------------------------------------------
Synchronizing Repository [f54312f5-d543-4ccc-a591-4a75d1567d72]--------------------------------------------------------------------
This command may be exited via ctrl+c without affecting the request.
Downloading metadata...
[\]
... completed
Downloading repository content...
[-]
[==================================================] 100%
RPMs: 0/0 items
Delta RPMs: 0/0 items
... completed
Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
Task Failed
Error retrieving metadata: Forbidden
Updated by Mario Gamboa over 7 years ago
also notice in candlepin log warn when refresh the manifest
2017-06-22 10:53:39,581 [thread=http-bio-8443-exec-1] [req=34d17278-c7ba-4cb3-8812-a7d650d41a2d, org=orgname, csid=] INFO org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=343
2017-06-22 10:54:04,454 [thread=http-bio-8443-exec-4] [req=95b549cf-2a9e-4844-867a-eb752e74de3e, org=, csid=] INFO org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/consumers/8eba76d4-0ddd-486b-b0c1-96c46d5d4fe3
2017-06-22 10:54:04,524 [thread=http-bio-8443-exec-4] [req=95b549cf-2a9e-4844-867a-eb752e74de3e, org=orgname, csid=] WARN org.candlepin.common.resteasy.filter.LinkHeaderResponseFilter - Method marked for pagination, but no page exists in the context.
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 258 to 267
Updated by Eric Helms over 7 years ago
Mario,
Justin never received the manifest you sent, can you please resend it to:
jsherril@redhat.com
Updated by Mario Gamboa over 7 years ago
Eric this also fix it just running again the foreman-installer --upgrade script
Updated by Eric Helms over 7 years ago
- Translation missing: en.field_release changed from 267 to 281