Project

General

Profile

Bug #32284

Upgrade from 3.14 -> 3.18, Pulp migration failure

Added by Sven Solberg about 1 year ago. Updated about 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Repositories
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

Hi,
I have (had) a 3.14 install and I followed the upgrade process notes on https://theforeman.org/plugins/katello/3.18/upgrade/index.html

It all seemed to go well, no errors encountered.

My next step is to migrate to Pulp3. I followed the instructions for correcting the permissions, and ran the migration routine. It failed with a backtrace


[root@foreman ~]# chmod -R g+rwX /var/lib/pulp/content
[root@foreman ~]# find /var/lib/pulp/content -type d -perm -g-s -exec sudo chmod g+s {} \;
[root@foreman ~]# chgrp -R pulp /var/lib/pulp/content
[root@foreman ~]# foreman-maintain content migration-stats
Running Retrieve Pulp 2 to Pulp 3 migration statistics
================================================================================
Retrieve Pulp 2 to Pulp 3 migration statistics: 
============Migration Summary================
Migrated/Total RPMs: 0/54175
Migrated/Total errata: 0/178581
Migrated/Total repositories: 0/634

Estimated migration time based on yum content: 1 hours, 23 minutes

Note: ensure there is sufficient storage space for /var/lib/pulp/published to triple in size before starting the migration process.
Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/'
                                                                      [OK]
--------------------------------------------------------------------------------

[root@foreman ~]# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Starting task.
2021-04-10 16:13:42 +0100: Pre-migrating Pulp 2 ERRATUM content (general info) 5027/5132Migration failed, You will want to investigate: https://foreman.mydom/foreman_tasks/tasks/201e3c25-55d2-4361-a845-699f019cf940
rake aborted!
ForemanTasks::TaskError: Task 201e3c25-55d2-4361-a845-699f019cf940: Katello::Errors::Pulp3Error: PackageReleaseComponent matching query does not exist.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare" 
[root@foreman ~]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
devtmpfs                      5.8G     0  5.8G   0% /dev
tmpfs                         5.8G   40K  5.8G   1% /dev/shm
tmpfs                         5.8G   98M  5.7G   2% /run
tmpfs                         5.8G     0  5.8G   0% /sys/fs/cgroup
/dev/mapper/foreman-root       10G  5.6G  4.5G  56% /
/dev/vda1                    1014M  329M  686M  33% /boot
/dev/mapper/foreman-var        27G   13G   15G  47% /var
nas1:/volume7/foreman/pulp    576G  180G  397G  32% /var/lib/pulp
tmpfs                         1.2G     0  1.2G   0% /run/user/0

here's the full raw output from the task:

{
  "pulp_tasks": [
    {
      "pulp_href": "/pulp/api/v3/tasks/1711b0a4-c05f-4b1e-aaeb-95a94499e802/",
      "pulp_created": "2021-04-10T15:13:14.087+00:00",
      "state": "failed",
      "name": "pulp_2to3_migration.app.tasks.migrate.migrate_from_pulp2",
      "started_at": "2021-04-10T15:13:14.180+00:00",
      "finished_at": "2021-04-10T15:13:43.901+00:00",
      "error": {
        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n    migrate_content(plan, skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 47, in migrate_content\n    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/migrator.py\", line 85, in migrate_content_to_pulp3\n    loop.run_until_complete(dm.create())\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n    await pipeline\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py\", line 105, in run\n    d_content.content.q()\n  File \"/usr/lib/python3.6/site-packages/django/db/models/manager.py\", line 82, in manager_method\n    return getattr(self.get_queryset(), name)(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/django/db/models/query.py\", line 408, in get\n    self.model._meta.object_name\n",
        "description": "PackageReleaseComponent matching query does not exist." 
      },
      "worker": "/pulp/api/v3/workers/a70acf17-2574-4172-99d8-7545f61a4802/",
      "child_tasks": [],
      "task_group": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
      "progress_reports": [
        {
          "message": "Processing Pulp 2 repositories, importers, distributors",
          "code": "processing.repositories",
          "state": "completed",
          "total": 7,
          "done": 7
        },
        {
          "message": "Pre-migrating Pulp 2 DEB content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 DEB content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 DEB_RELEASE content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 1,
          "done": 1
        },
        {
          "message": "Pre-migrating Pulp 2 DEB_RELEASE content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 1,
          "done": 1
        },
        {
          "message": "Pre-migrating Pulp 2 DEB_COMPONENT content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 1,
          "done": 1
        },
        {
          "message": "Pre-migrating Pulp 2 DEB_COMPONENT content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 1,
          "done": 1
        },
        {
          "message": "Pre-migrating Pulp 2 RPM content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 RPM content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 SRPM content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 SRPM content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 DISTRIBUTION content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 DISTRIBUTION content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 MODULEMD_DEFAULTS content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_CATEGORY content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 ERRATUM content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 5751,
          "done": 5697
        },
        {
          "message": "Pre-migrating Pulp 2 ERRATUM content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 5751,
          "done": 5697
        },
        {
          "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 YUM_REPO_METADATA_FILE content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating deb content to Pulp 3 deb_component",
          "code": "migrating.deb.content",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_LANGPACKS content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_GROUP content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 PACKAGE_ENVIRONMENT content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating deb content to Pulp 3 deb_component2",
          "code": "migrating.deb.content",
          "state": "completed",
          "total": 367,
          "done": 367
        },
        {
          "message": "Pre-migrating Pulp 2 MODULEMD content (general info)",
          "code": "premigrating.content.general",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Pre-migrating Pulp 2 MODULEMD content (detail info)",
          "code": "premigrating.content.detail",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Creating repositories in Pulp 3",
          "code": "creating.repositories",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating importers to Pulp 3",
          "code": "migrating.importers",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating content to Pulp 3",
          "code": "migrating.content",
          "state": "failed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating deb content to Pulp 3 deb_release",
          "code": "migrating.deb.content",
          "state": "completed",
          "total": 0,
          "done": 0
        },
        {
          "message": "Migrating deb content to Pulp 3 deb",
          "code": "migrating.deb.content",
          "state": "completed",
          "total": 13,
          "done": 0
        },
        {
          "message": "Migrating deb content to Pulp 3 deb_component3",
          "code": "migrating.deb.content",
          "state": "completed",
          "total": 1,
          "done": 1
        }
      ],
      "created_resources": [
        "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/" 
      ],
      "reserved_resources_record": [
        "pulp_2to3_migration" 
      ]
    }
  ],
  "task_groups": [
    {
      "pulp_href": "/pulp/api/v3/task-groups/e3fa2a74-4623-4a2a-bd90-95e3e1c3daee/",
      "description": "Migration Sub-tasks",
      "all_tasks_dispatched": false,
      "waiting": 0,
      "skipped": 0,
      "running": 0,
      "completed": 0,
      "canceled": 0,
      "failed": 1,
      "group_progress_reports": [
        {
          "message": "Repo version creation",
          "code": "create.repo_version",
          "total": 0,
          "done": 0
        },
        {
          "message": "Distribution creation",
          "code": "create.distribution",
          "total": 0,
          "done": 0
        }
      ]
    }
  ],
  "poll_attempts": {
    "total": 18,
    "failed": 1
  }
}

History

#1 Updated by Sven Solberg about 1 year ago

  • Description updated (diff)

#2 Updated by Chris Roberts about 1 year ago

  • Triaged changed from No to Yes
  • Target version set to Katello 3.18.4
  • Category changed from Upgrades to Repositories

#3 Updated by Justin Sherrill about 1 year ago

Can you confirm what version of python3-pulp-2to3-migration is installed?

rpm -q python3-pulp-2to3-migration

Since this is an error from pulp, i went ahead and filed a pulp bug for this: https://pulp.plan.io/issues/8562

#4 Updated by Sven Solberg about 1 year ago

Justin Sherrill wrote:

Can you confirm what version of python3-pulp-2to3-migration is installed?
Since this is an error from pulp, i went ahead and filed a pulp bug for this: https://pulp.plan.io/issues/8562

Hi Justin, thanks for that.

Err.. no! Its not installed and not in my repositories?!

[root@foreman ~]# rpm -q python3-pulp2to3-migration
package python3-pulp2to3-migration is not installed

[root@foreman ~]# yum install python3-pulp2to3-migration
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * base: mirror.strencom.net
 * centos-sclo-rh: mirrors.ukfast.co.uk
 * epel: mirrors.ukfast.co.uk
 * extras: mirror.strencom.net
 * updates: mirror.strencom.net
No package python3-pulp2to3-migration available.
Error: Nothing to do
[root@foreman ~]# yum search pulp2to3
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile
 * base: mirror.strencom.net
 * centos-sclo-rh: mirrors.ukfast.co.uk
 * epel: mirrors.ukfast.co.uk
 * extras: mirror.strencom.net
 * updates: mirror.strencom.net
Warning: No matches found for: pulp2to3
No matches found

Here's my katello rpm repo list:

[root@foreman yum.repos.d]# cat katello.repo 
# Place this file in your /etc/yum.repos.d/ directory

[katello]
name=Katello 3.18
baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.18/katello/el7/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1
module_hotfixes=1

[pulp]
name=Pulp Community Release
baseurl=https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.21/$releasever/$basearch
gpgkey=https://repos.fedorapeople.org/repos/pulp/pulp/GPG-RPM-KEY-pulp-2
enabled=1
gpgcheck=1
module_hotfixes=1

# Candlepin RPMs as supported by Katello - This is a coordinated
# copy of Candlepin's packages in order to ensure compatibility

[katello-candlepin]
name=Candlepin: an open source entitlement management system.
baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.18/candlepin/el7/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1
module_hotfixes=1

[pulpcore]
name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
baseurl=https://yum.theforeman.org/pulpcore/3.7/el7/$basearch/
gpgkey=https://yum.theforeman.org/pulpcore/3.7/GPG-RPM-KEY-pulpcore
enabled=1
gpgcheck=1
module_hotfixes=1

# source repositories

[katello-source]
name=Katello 3.18 Source
baseurl=https://fedorapeople.org/groups/katello/releases/source/srpm/3.18/katello/el7/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=0
gpgcheck=1

[pulp-source]
name=Pulp Community Release Source
baseurl=https://repos.fedorapeople.org/repos/pulp/pulp/stable/2.21/$releasever/src/
gpgkey=https://repos.fedorapeople.org/repos/pulp/pulp/GPG-RPM-KEY-pulp-2
enabled=0
gpgcheck=1

[katello-candlepin-source]
name=Katello Candlepin source
baseurl=https://fedorapeople.org/groups/katello/releases/source/srpm/3.18/candlepin/el7/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=0
gpgcheck=1

[pulpcore-source]
name=pulpcore source
baseurl=https://yum.theforeman.org/pulpcore/3.7/el7/source/
gpgkey=https://yum.theforeman.org/pulpcore/3.7/GPG-RPM-KEY-pulpcore
enabled=0
gpgcheck=1

#5 Updated by Sven Solberg about 1 year ago

Actually - I found that I have the package, there's a slight typo in the name: python3-pulp-2to3-migration


Installed Packages
Name        : python3-pulp-2to3-migration
Arch        : noarch
Version     : 0.10.0
Release     : 1.el7
Size        : 1.1 M
Repo        : installed
From repo   : pulpcore
Summary     : Pulp 2 to Pulp 3 migration tool
URL         : http://www.pulpproject.org
Licence     : GPLv2+
Description : Pulp 2 to Pulp 3 migration tool

i'll attempt to update it to 0.11.0 and report back.

#6 Updated by Sven Solberg about 1 year ago

Installed 0.11.0 without issue, updated all of the remaining packages in the system.

Migration was progressing successfully without issue, then the server ran out of memory and an OoM kicked in and stopped everything.

I shut down the VM, increased from 12GB to 16GB memory. Restarted the migration process and:

OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/cf25f398-bfda-4751-9309-a5883f019f5b'

I shut down all the services, cleared out the temporary folders, restarted all foreman services, ran a health check and started again.

More time passed and more of the same folder not empty errors. I've checked the folder after this failure and the folder are empty.

Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: c1e4ecd5-234a-45da-a1bc-f0dc325b76d5
Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: 967bc087-ed3f-4632-8e1e-67c03aea864d
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (c1e4ecd5-234a-45da-a1bc-f0dc325b76d5)
Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: Job OK (967bc087-ed3f-4632-8e1e-67c03aea864d)
Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: 2f502df4-b258-4dfb-8024-a63bb57712ac
Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (d8262d2b-e97c-42c1-914c-37860e2632d1)
Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: e3edba80-c690-43da-a516-b933df03a298
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: 107e3757-7a68-4792-8229-f3eb1ee81f6b
Apr 16 09:37:04 foreman pulpcore-resource-manager: pulp: rq.worker:INFO: resource-manager: Job OK (e3edba80-c690-43da-a516-b933df03a298)
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (107e3757-7a68-4792-8229-f3eb1ee81f6b)
Apr 16 09:37:04 foreman pulpcore-worker-3: pulp: pulp_rpm.app.tasks.publishing:INFO: Publishing: repository=cce2d175-f054-4b8c-8519-654a99274232, version=1
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: 932ec04a-233d-47e2-80e4-78658b850bc5
Apr 16 09:37:04 foreman pulpcore-worker-1: pulp: rq.worker:INFO: 14345@foreman.mydom: Job OK (932ec04a-233d-47e2-80e4-78658b850bc5)
Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 16 09:37:05 foreman pulpcore-worker-3: rv = job.perform()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 16 09:37:05 foreman pulpcore-worker-3: self._result = self._execute()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 16 09:37:05 foreman pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration
Apr 16 09:37:05 foreman pulpcore-worker-3: migrated_repo.pulp3_repository_version
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor
Apr 16 09:37:05 foreman pulpcore-worker-3: pulp2dist, repo_version)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
Apr 16 09:37:05 foreman pulpcore-worker-3: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish
Apr 16 09:37:05 foreman pulpcore-worker-3: metadata_signing_service=metadata_signing_service,
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Apr 16 09:37:05 foreman pulpcore-worker-3: self.delete()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Apr 16 09:37:05 foreman pulpcore-worker-3: self._delete()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Apr 16 09:37:05 foreman pulpcore-worker-3: shutil.rmtree(self.path)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Apr 16 09:37:05 foreman pulpcore-worker-3: onerror(os.rmdir, path, sys.exc_info())
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Apr 16 09:37:05 foreman pulpcore-worker-3: os.rmdir(path)
Apr 16 09:37:05 foreman pulpcore-worker-3: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14343@foreman.mydom/2f502df4-b258-4dfb-8024-a63bb57712ac'
Apr 16 09:37:05 foreman pulpcore-worker-3: Traceback (most recent call last):
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 16 09:37:05 foreman pulpcore-worker-3: rv = job.perform()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 16 09:37:05 foreman pulpcore-worker-3: self._result = self._execute()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 16 09:37:05 foreman pulpcore-worker-3: return self.func(*self.args, **self.kwargs)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration
Apr 16 09:37:05 foreman pulpcore-worker-3: migrated_repo.pulp3_repository_version
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor
Apr 16 09:37:05 foreman pulpcore-worker-3: pulp2dist, repo_version)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
Apr 16 09:37:05 foreman pulpcore-worker-3: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish
Apr 16 09:37:05 foreman pulpcore-worker-3: metadata_signing_service=metadata_signing_service,
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Apr 16 09:37:05 foreman pulpcore-worker-3: self.delete()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Apr 16 09:37:05 foreman pulpcore-worker-3: self._delete()
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Apr 16 09:37:05 foreman pulpcore-worker-3: shutil.rmtree(self.path)
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Apr 16 09:37:05 foreman pulpcore-worker-3: onerror(os.rmdir, path, sys.exc_info())
Apr 16 09:37:05 foreman pulpcore-worker-3: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Apr 16 09:37:05 foreman pulpcore-worker-3: os.rmdir(path)
Apr 16 09:37:05 foreman pulpcore-worker-3: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14343@foreman.mydom/2f502df4-b258-4dfb-8024-a63bb57712ac'
Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: d8349d28-584f-433b-ac58-7386ab122c43
Apr 16 09:37:05 foreman pulpcore-worker-3: pulp: rq.worker:INFO: 14343@foreman.mydom: Job OK (d8349d28-584f-433b-ac58-7386ab122c43)
Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 16 09:37:12 foreman pulpcore-worker-2: rv = job.perform()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 16 09:37:12 foreman pulpcore-worker-2: self._result = self._execute()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 16 09:37:12 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration
Apr 16 09:37:12 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor
Apr 16 09:37:12 foreman pulpcore-worker-2: pulp2dist, repo_version)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
Apr 16 09:37:12 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish
Apr 16 09:37:12 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service,
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Apr 16 09:37:12 foreman pulpcore-worker-2: self.delete()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Apr 16 09:37:12 foreman pulpcore-worker-2: self._delete()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Apr 16 09:37:12 foreman pulpcore-worker-2: shutil.rmtree(self.path)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Apr 16 09:37:12 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info())
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Apr 16 09:37:12 foreman pulpcore-worker-2: os.rmdir(path)
Apr 16 09:37:12 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/d221672c-e44c-4e00-bd07-5d25c96a1598'
Apr 16 09:37:12 foreman pulpcore-worker-2: Traceback (most recent call last):
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 16 09:37:12 foreman pulpcore-worker-2: rv = job.perform()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 16 09:37:12 foreman pulpcore-worker-2: self._result = self._execute()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 16 09:37:12 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration
Apr 16 09:37:12 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor
Apr 16 09:37:12 foreman pulpcore-worker-2: pulp2dist, repo_version)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
Apr 16 09:37:12 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish
Apr 16 09:37:12 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service,
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Apr 16 09:37:12 foreman pulpcore-worker-2: self.delete()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Apr 16 09:37:12 foreman pulpcore-worker-2: self._delete()
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Apr 16 09:37:12 foreman pulpcore-worker-2: shutil.rmtree(self.path)
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Apr 16 09:37:12 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info())
Apr 16 09:37:12 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Apr 16 09:37:12 foreman pulpcore-worker-2: os.rmdir(path)
Apr 16 09:37:12 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/d221672c-e44c-4e00-bd07-5d25c96a1598'
Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: 45c3f90d-c1bc-4800-8b05-e73fd910f5d6
Apr 16 09:37:12 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: Job OK (45c3f90d-c1bc-4800-8b05-e73fd910f5d6)

My /var/lib/pulp in on an NFS mount to a Synology DS918+, would that make a difference at all? Here's the mount options and /proc/mounts

[root@foreman ~]# grep pulp /etc/fstab 
dsberg:/volume7/foreman/pulp /var/lib/pulp    nfs    _netdev,defaults 0 0
[root@foreman ~]# grep pulp /proc/mounts 
dsberg:/volume7/foreman/pulp /var/lib/pulp nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.42.88,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.42.88 0 0

NFS is exported from an btrfs volume, integrity checking enabled, compressions enabled, no squash mapping, security=sys, async, allow connections from non-priv ports, allow access to sub folder. NFS v4.1 not enabled.

I've just altered the clean up library to ignore the errors:

    def _delete(self):
        """ 
    Helper method for delete
        """ 
        try:
            shutil.rmtree(self.path, ignore_errors=True)
        except FileNotFoundError:
            pass

but that didn't help either:

Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 16 09:56:37 foreman pulpcore-worker-2: rv = job.perform()
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 16 09:56:37 foreman pulpcore-worker-2: self._result = self._execute()
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 16 09:56:37 foreman pulpcore-worker-2: return self.func(*self.args, **self.kwargs)
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 224, in complex_repo_migration
Apr 16 09:56:37 foreman pulpcore-worker-2: migrated_repo.pulp3_repository_version
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 371, in migrate_repo_distributor
Apr 16 09:56:37 foreman pulpcore-worker-2: pulp2dist, repo_version)
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
Apr 16 09:56:37 foreman pulpcore-worker-2: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 334, in publish
Apr 16 09:56:37 foreman pulpcore-worker-2: metadata_signing_service=metadata_signing_service,
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 178, in __exit__
Apr 16 09:56:37 foreman pulpcore-worker-2: self.delete()
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 80, in delete
Apr 16 09:56:37 foreman pulpcore-worker-2: self._delete()
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/services/storage.py", line 90, in _delete
Apr 16 09:56:37 foreman pulpcore-worker-2: shutil.rmtree(self.path, ignore_errors=True)
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 490, in rmtree
Apr 16 09:56:37 foreman pulpcore-worker-2: onerror(os.rmdir, path, sys.exc_info())
Apr 16 09:56:37 foreman pulpcore-worker-2: File "/usr/lib64/python3.6/shutil.py", line 488, in rmtree
Apr 16 09:56:37 foreman pulpcore-worker-2: os.rmdir(path)
Apr 16 09:56:37 foreman pulpcore-worker-2: OSError: [Errno 39] Directory not empty: '/var/lib/pulp/tmp/14346@foreman.mydom/a9fa2afc-7301-4af9-a657-8e4f70a65b23'
Apr 16 09:56:37 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: bfec9189-6375-4be5-8f8d-54c60ab55a32
Apr 16 09:56:37 foreman pulpcore-worker-2: pulp: rq.worker:INFO: 14346@foreman.mydom: Job OK (bfec9189-6375-4be5-8f8d-54c60ab55a32)

#7 Updated by Jonathon Turel about 1 year ago

  • Target version changed from Katello 3.18.4 to Katello Recycle Bin
  • Status changed from New to Rejected

Closing this since the Pulp issue linked above is not yet resolved. Please follow along there to see when it gets fixed. Thanks for filing!

Also available in: Atom PDF