Plugins » Katello

For a while now, all attempts to apply errata fail with error 'Version X does not exist for repository X'.
e.g below "Version 43 does not exist for repository 'Everything_SRPMS-12633641'"

The failed task leaves all the repositories locked. The only way I've found to recover from this is:

• Skip the errored task in dynflow console
• Resume the failed task
• Delete the incomplete minor content version (to remove inconsistent CVV)
• Create a full new content version with all pending updates, not just specific errata

In the case where the logs are captured below, "AlmaLinux 8 Full" is a composite CV currently at version 51. Version 43 was purged some time ago. The composite contains "AlmaLinux 8" content view currently at version 16. There are no content views with version 43 currently in any content view.

2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/af8e3005-63b7-40fd-9105-cd19a9f5958e/
2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on distribution_version , 8
2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on distribution_arch , x86_64
2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on distribution_family , AlmaLinux
2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on distribution_variant , AppStream,BaseOS
2022-01-28T11:47:04 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on distribution_bootable , true
2022-01-28T11:47:05 [I|aud|7e2a9b7b] Katello::Repository (4704) update event on last_indexed 1970-01-01 01:00:00 +0100, 2022-01-28 11:47:05 +0000
2022-01-28T11:47:05 [I|aud|7e2a9b7b] Katello::Repository (4705) update event on version_href /pulp/api/v3/repositories/rpm/rpm/df797d30-07c7-45fc-a21f-fab40fdff8d2/versions/0/, /pulp/api/v3/repositories/rpm/rpm/12945dfa-5d05-4ecb-879d-971609513a5b/versions/83/
2022-01-28T11:47:05 [I|aud|7e2a9b7b] Katello::Repository (4705) update event on publication_href , /pulp/api/v3/publications/rpm/rpm/64e4b70e-b02e-494f-9ee5-aa83874dbc4e/
2022-01-28T11:47:09 [I|aud|7e2a9b7b] Katello::Repository (4705) update event on last_indexed 1970-01-01 01:00:00 +0100, 2022-01-28 11:47:09 +0000
2022-01-28T11:47:09 [E|bac|7e2a9b7b] Error message: the server returns an error
 7e2a9b7b | HTTP status code: 400
 7e2a9b7b | Response headers: {"date"=>"Fri, 28 Jan 2022 11:47:09 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"73", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"7e2a9b7b-eeb1-44fb-bb75-1ef98e024ee4", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 whitefall.jellybean.sihnon.net", "connection"=>"close"}
 7e2a9b7b | Response body: ["Version 43 does not exist for repository 'Everything_SRPMS-12633641'."] (PulpRpmClient::ApiError)
 7e2a9b7b | /usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
 7e2a9b7b | /usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:79:in `copy_content_with_http_info'
 7e2a9b7b | /usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:28:in `copy_content'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/services/katello/pulp3/repository/yum.rb:143:in `copy_content_chunked'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/services/katello/pulp3/repository/yum.rb:116:in `multi_copy_units'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/pulp3/repository/multi_copy_units.rb:49:in `invoke_external_task'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:19:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 7e2a9b7b | /usr/share/gems/gems/katello-4.4.0.pre.master/app/lib/actions/middleware/remote_action.rb:16:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 7e2a9b7b | /usr/share/gems/gems/foreman-tasks-5.3.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 7e2a9b7b | /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 7e2a9b7b | [ sidekiq ]
 7e2a9b7b | [ concurrent-ruby ]
2022-01-28T11:47:11 [I|app|62326bc9] Started POST "/api/config_reports" for 81.187.154.141 at 2022-01-28 11:47:11 +0000
2022-01-28T11:47:11 [I|app|62326bc9] Processing by Api::V2::ConfigReportsController#create as JSON
2022-01-28T11:47:11 [I|app|62326bc9]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2022-01-28T11:47:11 [I|app|62326bc9] Scanning report with: Foreman::PuppetReportScanner, ForemanAnsible::AnsibleReportScanner
2022-01-28T11:47:12 [I|app|62326bc9] Imported report for arvad.sihnon.net in 308.5 ms, status refreshed in 67.5 ms
2022-01-28T11:47:12 [I|app|62326bc9]   Rendering api/v2/config_reports/create.json.rabl
2022-01-28T11:47:12 [I|app|62326bc9]   Rendered api/v2/config_reports/create.json.rabl (Duration: 7.0ms | Allocations: 6673)
2022-01-28T11:47:12 [I|app|62326bc9] Completed 201 Created in 463ms (Views: 8.8ms | ActiveRecord: 371.6ms | Allocations: 34475)
2022-01-28T11:47:15 [I|bac|7e2a9b7b] Task {label: Actions::Katello::ContentView::IncrementalUpdates, id: 5a6a48b0-c65f-4f07-bc9a-ae7a9ad23af4, execution_plan_id: 5f34ad6e-5d8e-4176-b763-ed19e892abc3} state changed: paused  result: error

Installed package versions:

# rpm -qa '*katello*' '*pulp*'
rubygem-pulp_deb_client-2.16.0-1.el8.noarch
rubygem-pulp_file_client-1.10.0-1.el8.noarch
python38-pulp-container-2.9.0-1.el8.noarch
foreman-installer-katello-3.2.0-0.1.develop.20220125164810git72c2858.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
pulpcore-selinux-1.2.6-2.el8.x86_64
katello-server-ca-1.0-1.noarch
rubygem-katello-4.4.0-0.2.pre.master.20220125221425gitbc1eef9.el8.noarch
katello-selinux-4.0.2-1.el8.noarch
pulp-client-1.0-1.noarch
katello-repos-4.4-0.2.nightly.el8.noarch
rubygem-pulp_ansible_client-0.10.1-1.el8.noarch
rubygem-smart_proxy_pulp-3.2.0-2.fm3_2.el8.noarch
python38-pulp-rpm-3.16.1-2.el8.noarch
katello-default-ca-1.0-1.noarch
katello-certs-tools-2.8.2-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
katello-common-4.4.0-0.3.master.el8.noarch
katello-4.4.0-0.3.master.el8.noarch
python38-pulp-deb-2.16.1-1.el8.noarch
katello-ca-consumer-whitefall.jellybean.sihnon.net-1.0-1.noarch
python38-pulp-file-1.10.1-1.el8.noarch
rubygem-pulp_rpm_client-3.16.1-1.el8.noarch
rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
katello-debug-4.4.0-0.3.master.el8.noarch
python38-pulpcore-3.16.1-1.el8.noarch
python38-pulp-cli-0.12.0-1.el8.noarch
rubygem-pulpcore_client-3.16.0-1.el8.noarch
rubygem-pulp_container_client-2.9.0-1.el8.noarch
rubygem-hammer_cli_katello-1.4.0-0.1.pre.master.20220118144255git85dfb44.el8.noarch
python38-pulp-ansible-0.10.1-1.el8.noarch
rubygem-pulp_python_client-3.5.2-1.el8.noarch
katello-client-bootstrap-1.7.7-1.el8.noarch