Actions
Bug #32284
closed
Upgrade from 3.14 -> 3.18, Pulp migration failure
Description
Hi,
I have (had) a 3.14 install and I followed the upgrade process notes on https://theforeman.org/plugins/katello/3.18/upgrade/index.html
It all seemed to go well, no errors encountered.
My next step is to migrate to Pulp3. I followed the instructions for correcting the permissions, and ran the migration routine. It failed with a backtrace
[root@foreman ~]# chmod -R g+rwX /var/lib/pulp/content
[root@foreman ~]# find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} \;
[root@foreman ~]# chgrp -R pulp /var/lib/pulp/content
[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics:
============Migration Summary================
Migrated/Total RPMs: 0/54175
Migrated/Total errata: 0/178581
Migrated/Total repositories: 0/634
Estimated migration time based on yum content: 1 hours, 23 minutes
Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
[OK]
--------------------------------------------------------------------------------
[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-04-10 16:13:42 +0100: Pre-migrating Pulp 2 ERRATUM content (general info) 5027/5132Migration failed, You will want to investigate: https://foreman.mydom/foreman_tasks/tasks/201e3c25-55d2-4361-a845-699f019cf940
rake aborted!
ForemanTasks::TaskError: Task 201e3c25-55d2-4361-a845-699f019cf940: Katello::Errors::Pulp3Error: PackageReleaseComponent matching query does not exist.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
The following steps ended up in failing state:
[content-prepare]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"
[root@foreman ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 5.8G 0 5.8G 0% /dev
tmpfs 5.8G 40K 5.8G 1% /dev/shm
tmpfs 5.8G 98M 5.7G 2% /run
tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/mapper/foreman-root 10G 5.6G 4.5G 56% /
/dev/vda1 1014M 329M 686M 33% /boot
/dev/mapper/foreman-var 27G 13G 15G 47% /var
nas1:/volume7/foreman/pulp 576G 180G 397G 32% /var/lib/pulp
tmpfs 1.2G 0 1.2G 0% /run/user/0
here's the full raw output from the task:
{
"pulp_tasks": [
{
"pulp_href": "/pulp/api/v3/tasks/1711b0a4-c05f-4b1e-aaeb-95a94499e802/",
"pulp_created": "2021-04-10T15:13:14.087+00:00",
"state": "failed",
"name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
"started_at": "2021-04-10T15:13:14.180+00:00",
"finished_at": "2021-04-10T15:13:43.901+00:00",
"error": {
"traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n rv = job.perform()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n self._result = self._execute()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n migrate_content(plan, skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/migrator.py\", line 85, in migrate_content_to_pulp3\n loop.run_until_complete(dm.create())\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n await pipeline\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 105, in run\n d_content.content.q()\n File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n return getattr(self.get_queryset(), name)(*args, **kwargs)\n File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in get\n self.model._meta.object_name\n",
"description": "PackageReleaseComponent matching query does not exist."
},
"worker": "/pulp/api/v3/workers/a70acf17-2574-4172-99d8-7545f61a4802/",
"child_tasks": [],
"task_group": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
"progress_reports": [
{
"message": "Processing Pulp 2 repositories, importers, distributors",
"code": "processing.repositories",
"state": "completed",
"total": 7,
"done": 7
},
{
"message": "Pre-migrating Pulp 2 DEB content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DEB content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DEB_RELEASE content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_RELEASE content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_COMPONENT content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 DEB_COMPONENT content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 1,
"done": 1
},
{
"message": "Pre-migrating Pulp 2 RPM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 RPM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 SRPM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 SRPM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 5751,
"done": 5697
},
{
"message": "Pre-migrating Pulp 2 ERRATUM content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 5751,
"done": 5697
},
{
"message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component",
"code": "migrating.deb.content",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component2",
"code": "migrating.deb.content",
"state": "completed",
"total": 367,
"done": 367
},
{
"message": "Pre-migrating Pulp 2 MODULEMD content (general info)",
"code": "premigrating.content.general",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Pre-migrating Pulp 2 MODULEMD content (detail info)",
"code": "premigrating.content.detail",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Creating repositories in Pulp 3",
"code": "creating.repositories",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating importers to Pulp 3",
"code": "migrating.importers",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating content to Pulp 3",
"code": "migrating.content",
"state": "failed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_release",
"code": "migrating.deb.content",
"state": "completed",
"total": 0,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb",
"code": "migrating.deb.content",
"state": "completed",
"total": 13,
"done": 0
},
{
"message": "Migrating deb content to Pulp 3 deb_component3",
"code": "migrating.deb.content",
"state": "completed",
"total": 1,
"done": 1
}
],
"created_resources": [
"/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/"
],
"reserved_resources_record": [
"pulp_2to3_migration"
]
}
],
"task_groups": [
{
"pulp_href": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
"description": "Migration Sub-tasks",
"all_tasks_dispatched": false,
"waiting": 0,
"skipped": 0,
"running": 0,
"completed": 0,
"canceled": 0,
"failed": 1,
"group_progress_reports": [
{
"message": "Repo version creation",
"code": "create.repo_version",
"total": 0,
"done": 0
},
{
"message": "Distribution creation",
"code": "create.distribution",
"total": 0,
"done": 0
}
]
}
],
"poll_attempts": {
"total": 18,
"failed": 1
}
}
Updated by Chris Roberts over 3 years ago
- Category changed from Upgrades to Repositories
- Target version set to Katello 3.18.4
- Triaged changed from No to Yes
Updated by Justin Sherrill over 3 years ago
Can you confirm what version of python3-pulp-2to3-migration is installed?
rpm -q python3-pulp-2to3-migration
Since this is an error from pulp, i went ahead and filed a pulp bug for this: https://pulp.plan.io/issues/8562
Updated by Sven Solberg over 3 years ago
Justin Sherrill wrote:
Can you confirm what version of python3-pulp-2to3-migration is installed?
Since this is an error from pulp, i went ahead and filed a pulp bug for this: https://pulp.plan.io/issues/8562
Hi Justin, thanks for that.
Err.. no! Its not installed and not in my repositories?!
[root@foreman ~]# rpm -q python3-pulp2to3-migration package python3-pulp2to3-migration is not installed [root@foreman ~]# yum install python3-pulp2to3-migration Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.strencom.net * centos-sclo-rh: mirrors.ukfast.co.uk * epel: mirrors.ukfast.co.uk * extras: mirror.strencom.net * updates: mirror.strencom.net No package python3-pulp2to3-migration available. Error: Nothing to do [root@foreman ~]# yum search pulp2to3 Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * base: mirror.strencom.net * centos-sclo-rh: mirrors.ukfast.co.uk * epel: mirrors.ukfast.co.uk * extras: mirror.strencom.net * updates: mirror.strencom.net Warning: No matches found for: pulp2to3 No matches found
Here's my katello rpm repo list:
[root@foreman yum.repos.d]# cat katello.repo # Place this file in your /etc/yum.repos.d/ directory [katello] name=Katello 3.18 baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.18/katello/el7/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=1 gpgcheck=1 module_hotfixes=1 [pulp] name=Pulp Community Release baseurl=https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.21/$releasever/$basearch gpgkey=https://repos.fedorapeople.org/repos/pulp/pulp/GPG-RPM-KEY-pulp-2 enabled=1 gpgcheck=1 module_hotfixes=1 # Candlepin RPMs as supported by Katello - This is a coordinated # copy of Candlepin's packages in order to ensure compatibility [katello-candlepin] name=Candlepin: an open source entitlement management system. baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.18/candlepin/el7/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=1 gpgcheck=1 module_hotfixes=1 [pulpcore] name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages. baseurl=https://yum.theforeman.org/pulpcore/3.7/el7/$basearch/ gpgkey=https://yum.theforeman.org/pulpcore/3.7/GPG-RPM-KEY-pulpcore enabled=1 gpgcheck=1 module_hotfixes=1 # source repositories [katello-source] name=Katello 3.18 Source baseurl=https://fedorapeople.org/groups/katello/releases/source/srpm/3.18/katello/el7/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=0 gpgcheck=1 [pulp-source] name=Pulp Community Release Source baseurl=https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.21/$releasever/src/ gpgkey=https://repos.fedorapeople.org/repos/pulp/pulp/GPG-RPM-KEY-pulp-2 enabled=0 gpgcheck=1 [katello-candlepin-source] name=Katello Candlepin source baseurl=https://fedorapeople.org/groups/katello/releases/source/srpm/3.18/candlepin/el7/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=0 gpgcheck=1 [pulpcore-source] name=pulpcore source baseurl=https://yum.theforeman.org/pulpcore/3.7/el7/source/ gpgkey=https://yum.theforeman.org/pulpcore/3.7/GPG-RPM-KEY-pulpcore enabled=0 gpgcheck=1
Updated by Sven Solberg over 3 years ago
Actually - I found that I have the package, there's a slight typo in the name: python3-pulp-2to3-migration
Installed Packages Name : python3-pulp-2to3-migration Arch : noarch Version : 0.10.0 Release : 1.el7 Size : 1.1 M Repo : installed From repo : pulpcore Summary : Pulp 2 to Pulp 3 migration tool URL : http://www.pulpproject.org Licence : GPLv2+ Description : Pulp 2 to Pulp 3 migration tool
i'll attempt to update it to 0.11.0 and report back.
Updated by Sven Solberg over 3 years ago
Installed 0.11.0 without issue, updated all of the remaining packages in the system.
Migration was progressing successfully without issue, then the server ran out of memory and an OoM kicked in and stopped everything.
I shut down the VM, increased from 12GB to 16GB memory. Restarted the migration process and:
OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/cf25f398-bfda-4751-9309-a5883f019f5b'
I shut down all the services, cleared out the temporary folders, restarted all foreman services, ran a health check and started again.
More time passed and more of the same folder not empty errors. I've checked the folder after this failure and the folder are empty.
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: c1e4ecd5-234a-45da-a1bc-f0dc325b76d5 Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: 967bc087-ed3f-4632-8e1e-67c03aea864d Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (c1e4ecd5-234a-45da-a1bc-f0dc325b76d5) Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: Job OK (967bc087-ed3f-4632-8e1e-67c03aea864d) Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: 2f502df4-b258-4dfb-8024-a63bb57712ac Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (d8262d2b-e97c-42c1-914c-37860e2632d1) Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: e3edba80-c690-43da-a516-b933df03a298 Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: 107e3757-7a68-4792-8229-f3eb1ee81f6b Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (e3edba80-c690-43da-a516-b933df03a298) Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (107e3757-7a68-4792-8229-f3eb1ee81f6b) Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=cce2d175-f054-4b8c-8519-654a99274232, version=1 Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: 932ec04a-233d-47e2-80e4-78658b850bc5 Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (932ec04a-233d-47e2-80e4-78658b850bc5) Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last): Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Apr 16 09:37:05 foreman pulpcore-worker-3: rv = job.perform() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Apr 16 09:37:05 foreman pulpcore-worker-3: self._result = self._execute() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Apr 16 09:37:05 foreman pulpcore-worker-3: return self.func(*self.args, **self.kwargs) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration Apr 16 09:37:05 foreman pulpcore-worker-3: migrated_repo.pulp3_repository_version Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor Apr 16 09:37:05 foreman pulpcore-worker-3: pulp2dist, repo_version) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3 Apr 16 09:37:05 foreman pulpcore-worker-3: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish Apr 16 09:37:05 foreman pulpcore-worker-3: metadata_signing_service=metadata_signing_service, Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__ Apr 16 09:37:05 foreman pulpcore-worker-3: self.delete() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete Apr 16 09:37:05 foreman pulpcore-worker-3: self._delete() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete Apr 16 09:37:05 foreman pulpcore-worker-3: shutil.rmtree(self.path) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree Apr 16 09:37:05 foreman pulpcore-worker-3: onerror(os.rmdir, path, sys.exc_info()) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree Apr 16 09:37:05 foreman pulpcore-worker-3: os.rmdir(path) Apr 16 09:37:05 foreman pulpcore-worker-3: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14343@foreman.mydom/2f502df4-b258-4dfb-8024-a63bb57712ac' Apr 16 09:37:05 foreman pulpcore-worker-3: Traceback (most recent call last): Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Apr 16 09:37:05 foreman pulpcore-worker-3: rv = job.perform() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Apr 16 09:37:05 foreman pulpcore-worker-3: self._result = self._execute() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Apr 16 09:37:05 foreman pulpcore-worker-3: return self.func(*self.args, **self.kwargs) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration Apr 16 09:37:05 foreman pulpcore-worker-3: migrated_repo.pulp3_repository_version Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor Apr 16 09:37:05 foreman pulpcore-worker-3: pulp2dist, repo_version) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3 Apr 16 09:37:05 foreman pulpcore-worker-3: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish Apr 16 09:37:05 foreman pulpcore-worker-3: metadata_signing_service=metadata_signing_service, Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__ Apr 16 09:37:05 foreman pulpcore-worker-3: self.delete() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete Apr 16 09:37:05 foreman pulpcore-worker-3: self._delete() Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete Apr 16 09:37:05 foreman pulpcore-worker-3: shutil.rmtree(self.path) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree Apr 16 09:37:05 foreman pulpcore-worker-3: onerror(os.rmdir, path, sys.exc_info()) Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree Apr 16 09:37:05 foreman pulpcore-worker-3: os.rmdir(path) Apr 16 09:37:05 foreman pulpcore-worker-3: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14343@foreman.mydom/2f502df4-b258-4dfb-8024-a63bb57712ac' Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: d8349d28-584f-433b-ac58-7386ab122c43 Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: Job OK (d8349d28-584f-433b-ac58-7386ab122c43) Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:ERROR: Traceback (most recent call last): Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Apr 16 09:37:12 foreman pulpcore-worker-2: rv = job.perform() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Apr 16 09:37:12 foreman pulpcore-worker-2: self._result = self._execute() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Apr 16 09:37:12 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration Apr 16 09:37:12 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor Apr 16 09:37:12 foreman pulpcore-worker-2: pulp2dist, repo_version) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3 Apr 16 09:37:12 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish Apr 16 09:37:12 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service, Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__ Apr 16 09:37:12 foreman pulpcore-worker-2: self.delete() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete Apr 16 09:37:12 foreman pulpcore-worker-2: self._delete() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete Apr 16 09:37:12 foreman pulpcore-worker-2: shutil.rmtree(self.path) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree Apr 16 09:37:12 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info()) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree Apr 16 09:37:12 foreman pulpcore-worker-2: os.rmdir(path) Apr 16 09:37:12 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/d221672c-e44c-4e00-bd07-5d25c96a1598' Apr 16 09:37:12 foreman pulpcore-worker-2: Traceback (most recent call last): Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Apr 16 09:37:12 foreman pulpcore-worker-2: rv = job.perform() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Apr 16 09:37:12 foreman pulpcore-worker-2: self._result = self._execute() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Apr 16 09:37:12 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration Apr 16 09:37:12 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor Apr 16 09:37:12 foreman pulpcore-worker-2: pulp2dist, repo_version) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3 Apr 16 09:37:12 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish Apr 16 09:37:12 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service, Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__ Apr 16 09:37:12 foreman pulpcore-worker-2: self.delete() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete Apr 16 09:37:12 foreman pulpcore-worker-2: self._delete() Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete Apr 16 09:37:12 foreman pulpcore-worker-2: shutil.rmtree(self.path) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree Apr 16 09:37:12 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info()) Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree Apr 16 09:37:12 foreman pulpcore-worker-2: os.rmdir(path) Apr 16 09:37:12 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/d221672c-e44c-4e00-bd07-5d25c96a1598' Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: 45c3f90d-c1bc-4800-8b05-e73fd910f5d6 Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: Job OK (45c3f90d-c1bc-4800-8b05-e73fd910f5d6)
My /var/lib/pulp in on an NFS mount to a Synology DS918+, would that make a difference at all? Here's the mount options and /proc/mounts
[root@foreman ~]# grep pulp /etc/fstab dsberg:/volume7/foreman/pulp /var/lib/pulp nfs _netdev,defaults 0 0 [root@foreman ~]# grep pulp /proc/mounts dsberg:/volume7/foreman/pulp /var/lib/pulp nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.42.88,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.42.88 0 0
NFS is exported from an btrfs volume, integrity checking enabled, compressions enabled, no squash mapping, security=sys, async, allow connections from non-priv ports, allow access to sub folder. NFS v4.1 not enabled.
I've just altered the clean up library to ignore the errors:
def _delete(self):
"""
Helper method for delete
"""
try:
shutil.rmtree(self.path, ignore_errors=True)
except FileNotFoundError:
pass
but that didn't help either:
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job Apr 16 09:56:37 foreman pulpcore-worker-2: rv = job.perform() Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform Apr 16 09:56:37 foreman pulpcore-worker-2: self._result = self._execute() Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute Apr 16 09:56:37 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs) Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration Apr 16 09:56:37 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor Apr 16 09:56:37 foreman pulpcore-worker-2: pulp2dist, repo_version) Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3 Apr 16 09:56:37 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite) Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish Apr 16 09:56:37 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service, Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__ Apr 16 09:56:37 foreman pulpcore-worker-2: self.delete() Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete Apr 16 09:56:37 foreman pulpcore-worker-2: self._delete() Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete Apr 16 09:56:37 foreman pulpcore-worker-2: shutil.rmtree(self.path, ignore_errors=True) Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree Apr 16 09:56:37 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info()) Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree Apr 16 09:56:37 foreman pulpcore-worker-2: os.rmdir(path) Apr 16 09:56:37 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/a9fa2afc-7301-4af9-a657-8e4f70a65b23' Apr 16 09:56:37 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: bfec9189-6375-4be5-8f8d-54c60ab55a32 Apr 16 09:56:37 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: Job OK (bfec9189-6375-4be5-8f8d-54c60ab55a32)
Updated by Jonathon Turel over 3 years ago
- Status changed from New to Rejected
- Target version changed from Katello 3.18.4 to Katello Recycle Bin
Closing this since the Pulp issue linked above is not yet resolved. Please follow along there to see when it gets fixed. Thanks for filing!
Actions