Actions
Bug #32284
closed
Upgrade from 3.14 -> 3.18, Pulp migration failure
Description
Hi,
I have (had) a 3.14 install and I followed the upgrade process notes on https://theforeman.org/plugins/katello/3.18/upgrade/index.html
It all seemed to go well, no errors encountered.
My next step is to migrate to Pulp3. I followed the instructions for correcting the permissions, and ran the migration routine. It failed with a backtrace
[root@foreman ~]# chmod -R g+rwX /var/lib/pulp/content
[root@foreman ~]# find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} \;
[root@foreman ~]# chgrp -R pulp /var/lib/pulp/content
[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics:
============Migration Summary================
Migrated/Total RPMs: 0/54175
Migrated/Total errata: 0/178581
Migrated/Total repositories: 0/634
Estimated migration time based on yum content: 1 hours, 23 minutes
Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
[OK]
--------------------------------------------------------------------------------
[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-10 16:13:42 +0100: Pre-migrating Pulp 2 ERRATUM content (general info) 5027/5132Migration failed, You will want to investigate: https://foreman.mydom/foreman_tasks/tasks/201e3c25-55d2-4361-a845-699f019cf940
rake aborted!
ForemanTasks::TaskError: Task 201e3c25-55d2-4361-a845-699f019cf940: Katello::Errors::Pulp3Error: PackageReleaseComponent matching query does not exist.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
[root@foreman ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 40K 5.8G 1% /dev/shm
tmpfs 5.8G 98M 5.7G 2% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/mapper/foreman-root 10G 5.6G 4.5G 56% /
/dev/vda1 1014M 329M 686M 33% /boot
/dev/mapper/foreman-var 27G 13G 15G 47% /var
nas1:/volume7/foreman/pulp 576G 180G 397G 32% /var/lib/pulp
tmpfs 1.2G 0 1.2G 0% /run/user/0
here's the full raw output from the task:
{
"pulp_tasks": [
{
"pulp_href": "/pulp/api/v3/tasks/1711b0a4-c05f-4b1e-aaeb-95a94499e802/",
"pulp_created": "2021-04-10T15:13:14.087+00:00",
"state": "failed",
"name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
"started_at": "2021-04-10T15:13:14.180+00:00",
"finished_at": "2021-04-10T15:13:43.901+00:00",
"error": {
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n rv = job.perform()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n self._result = self._execute()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n migrate_content(plan, skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/migrator.py\", line 85, in migrate_content_to_pulp3\n loop.run_until_complete(dm.create())\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n await pipeline\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 105, in run\n d_content.content.q()\n File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in get\n self.model._meta.object_name\n",
"description": "PackageReleaseComponent matching query does not exist."
},
"worker": "/pulp/api/v3/workers/a70acf17-2574-4172-99d8-7545f61a4802/",
"child_tasks": [],
"task_group": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
"progress_reports": [
{
"message": "Processing Pulp 2 repositories, importers, distributors",
"code": "processing.repositories",
"state": "completed",
"total": 7,
"done": 7
},
{
"message": "Pre-migrating Pulp 2 DEB content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DEB content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DEB_RELEASE content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_RELEASE content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_COMPONENT content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_COMPONENT content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 RPM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 RPM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 SRPM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 SRPM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 5751,
"done": 5697
},
{
"message": "Pre-migrating Pulp 2 ERRATUM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 5751,
"done": 5697
},
{
"message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component",
"code": "migrating.deb.content",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component2",
"code": "migrating.deb.content",
"state": "completed",
"total": 367,
"done": 367
},
{
"message": "Pre-migrating Pulp 2 MODULEMD content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Creating repositories in Pulp 3",
"code": "creating.repositories",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating importers to Pulp 3",
"code": "migrating.importers",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating content to Pulp 3",
"code": "migrating.content",
"state": "failed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_release",
"code": "migrating.deb.content",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb",
"code": "migrating.deb.content",
"state": "completed",
"total": 13,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component3",
"code": "migrating.deb.content",
"state": "completed",
"total": 1,
"done": 1
}
],
"created_resources": [
"/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/"
],
"reserved_resources_record": [
"pulp_2to3_migration"
]
}
],
"task_groups": [
{
"pulp_href": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
"description": "Migration Sub-tasks",
"all_tasks_dispatched": false,
"waiting": 0,
"skipped": 0,
"running": 0,
"completed": 0,
"canceled": 0,
"failed": 1,
"group_progress_reports": [
{
"message": "Repo version creation",
"code": "create.repo_version",
"total": 0,
"done": 0
},
{
"message": "Distribution creation",
"code": "create.distribution",
"total": 0,
"done": 0
}
]
}
],
"poll_attempts": {
"total": 18,
"failed": 1
}
}
Updated by 
Updated by 
Updated by 
Updated by
Actions