Plugins » Katello

Pulp is running out of memory.

[Tue Apr 16 17:06:31 2019] Out of memory: Kill process 9408 (celery) score 306 or sacrifice child
[Tue Apr 16 17:06:31 2019] Killed process 9408 (celery) total-vm:7501044kB, anon-rss:6543420kB, file-rss:188kB, shmem-rss:8kB

https://community.theforeman.org/t/pulp-deb-with-celery-cannot-allocate-memory-and-out-of-memory/11789/9

2019-04-17T08:52:13 [I|kat|] GET: https://testhost/pulp/api/v2/tasks/09050f09-4b3a-4277-abd4-552f1bfaded4/: {"content_type"=>"application/json", "accept"=>"application/json"}
Response: 200: {"exception": null, "task_type": "pulp.server.managers.repo.publish.publish", "_href": "/pulp/api/v2/tasks/09050f09-4b3a-4277-abd4-552f1bfaded4/", "task_id": "09050f09-
4b3a-4277-abd4-552f1bfaded4", "tags": ["pulp:repository:14-Server_CentOS_7-v4_0-dd519f45-2dbc-4bce-9866-44876474b108", "pulp:action:publish"], "finish_time": "2019-04-16T15:06:39Z
", "_ns": "task_status", "start_time": "2019-04-16T12:14:21Z", "traceback": "Traceback (most recent call last):\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line
367, in trace_task\n R = retval = fun(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in call\n return super(Task, self).
call(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in call\n return super(PulpTask, self).__call__(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in protected_call\n return self.run(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1110, in publish\n result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1207, in check_publish\n result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 1259, in _do_publish\n publish_report = publish_repo(transfer_repo, conduit, call_config)\n File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 737, in wrap_f\n return f(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py\", line 80, in publish_repo\n return self._publisher.process_lifecycle()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 572, in process_lifecycle\n super(PluginStep, self).process_lifecycle()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 163, in process_lifecycle\n step.process()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 256, in process\n self._process_block()\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n self.process_main()\n File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py\", line 334, in process_main\n with_symlinks=True)\n File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 478, in create\n component=component, architecture=architecture)\n File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 449, in add_packages\n pkg = debpkg.DebPkg.from_file(filename, Size=sz)\n File \"/usr/lib/python2.7/site-packages/debpkgr/debpkg.py\", line 416, in from_file\n control = debpkg.control.debcontrol().copy()\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 222, in debcontrol\n return Deb822(self.get_content(CONTROL_FILE))\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 171, in get_content\n f = self.get_file(fname, encoding=encoding, errors=errors)\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 145, in get_file\n fobj = self.tgz().extractfile('./' + fname)\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 92, in tgz\n raise DebError(\"%s\" % e)\nDebError: [Errno 12] Cannot allocate memory\n", "spawned_tasks": [], "progress_report": {"14-ZID_Server_CentOS_7-v4_0-dd519f45-2dbc-4bce-9866-44876474b108": [{"num_success": 0, "description": "Publishing modules", "step_type": "publish_modules", "sub_steps": [{"num_success": 1, "description": "", "step_type": "publish_deb_releases", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "9893857c-7465-4ed5-954f-571a35a82d0d", "num_processed": 1}, {"num_success": 1, "description": "", "step_type": "publish_deb_components", "items_total": 1, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "0f0e15b8-e9b9-4673-ad3e-122f08e07ec5", "num_processed": 1}, {"num_success": 53596, "description": "", "step_type": "publish_deb", "items_total": 53596, "state": "FINISHED", "error_details": [], "details": "", "num_failures": 0, "step_id": "5066ad0f-da79-4dae-939f-a54bb345fadb", "num_processed": 53596}, {"num_success": 0, "description": "", "step_type": "publish_repodata", "items_total": 1, "state": "FAILED", "error_details": [{"traceback": " File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 256, in process\n self._process_block()\n\n File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n self.process_main()\n\n File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py\", line 334, in process_main\n with_symlinks=True)\n\n File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 478, in create\n component=component, architecture=architecture)\n\n File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 449, in add_packages\n pkg = debpkg.DebPkg.from_file(filename, Size=sz)\n\n File \"/usr/lib/python2.7/site-packages/debpkgr/debpkg.py\", line 416, in from_file\n control = debpkg.control.debcontrol().copy()\n\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 222, in debcontrol\n return Deb822(self.get_content(CONTROL_FILE))\n\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 171, in get_content\n f = self.get_file(fname, encoding=encoding, errors=errors)\n\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 145, in get_file\n fobj = self.tgz().extractfile('./' + fname)\n\n File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 92, in tgz\n raise DebError(\"%s\" % e)\n", "error": "[Errno 12] Cannot allocate memory"}], "details": "", "num_failures": 1, "step_id": "0a3f7dbb-96c2-49a0-a242-7e35aef7b14f", "num_processed": 1}], "items_total": 1, "state": "FAILED", "error_details": [{"traceback": null, "error": null}], "details": "", "num_failures": 1, "step_id": "8bfb0b30-a19f-433f-b786-daded4b7b998", "num_processed": 1}, {"num_success": 0, "description": "Publishing files to web", "step_type": "publish_directory", "items_total": 1, "state": "NOT_STARTED", "error_details": [], "details": "", "num_failures": 0, "step_id": "569afc79-c76e-4d05-bc9f-9e97d0181d68", "num_processed": 0}, {"num_success": 0, "description": "Writing Listings File", "step_type": "generate_listing_files", "items_total": 1, "state": "NOT_STARTED", "error_details": [], "details": "", "num_failures": 0, "step_id": "2fbf5375-386e-4003-b109-006cc568a301", "num_processed": 0}, {"num_success": 0, "description": "Writing Listings File", "step_type": "generate_listing_files", "items_total": 1, "state": "NOT_STARTED", "error_details": [], "details": "", "num_failures": 0, "step_id": "b8c841ac-3a64-4d52-b571-5669435ac1c3", "num_processed": 0}]}, "queue": "reserved_resource_worker-0@testhost.dq2", "state": "error", "worker_name": "reserved_resource_worker-0@testhost", "result": null, "error": {"code": "PLP0000", "data": {}, "description": "[Errno 12] Cannot allocate memory", "sub_errors": []}, "_id": {"$oid": "5cb5c334357678b3055bb4d6"}, "id": "5cb5c334357678b3055bb4d6"}

Regards,
Florian