Plugins » Katello

Hello Katello support team,

My company runs a katello system on CentOS7.

In the end of march 2021 we did update this system. We did install all pending OS Updates. Then we did update:

Foreman: to 2.3.3-1
Katello: to 3.18.2-1

Since then the /var/log/messages logfile gets flooded with lots of pulp messages. The logfile grows a few hundred megabytes per day.

here is a part of this logfile. I replace the actual hostname with dummy values.

May 3 15:29:55 hostname pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker 'resource-manager'.
May 3 15:29:55 hostname pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '1153@hostname.domain.tld'.
May 3 15:29:55 hostname pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '1141@hostname.domain.tld'.
May 3 15:29:55 hostname pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '1137@hostname.domain.tld'.
May 3 15:29:55 hostname pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '1133@hostname.domain.tld'.
May 3 15:29:55 hostname pulpcore-worker-2: pulp: rq.worker:INFO: Worker rq:worker:1137@hostname.domain.tld: started, version 1.5.2
May 3 15:29:55 hostname pulpcore-worker-2: pulp: rq.worker:INFO: * Listening on 1137@hostname.domain.tld...
May 3 15:29:55 hostname pulpcore-worker-3: pulp: rq.worker:INFO: Worker rq:worker:1141@hostname.domain.tld: started, version 1.5.2
May 3 15:29:55 hostname pulpcore-worker-3: pulp: rq.worker:INFO: Listening on 1141@hostname.domain.tld...
May 3 15:29:55 hostname pulpcore-worker-4: pulp: rq.worker:INFO: Worker rq:worker:1153@hostname.domain.tld: started, version 1.5.2
May 3 15:29:55 hostname pulpcore-worker-4: pulp: rq.worker:INFO: Listening on 1153@hostname.domain.tld...
May 3 15:29:55 hostname pulpcore-worker-1: pulp: rq.worker:INFO: Worker rq:worker:1133@hostname.domain.tld: started, version 1.5.2
May 3 15:29:55 hostname pulpcore-worker-1: pulp: rq.worker:INFO: * Listening on 1133@hostname.domain.tld...
May 3 15:29:55 hostname pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker '1141@hostname.domain.tld' discovered
May 3 15:29:55 hostname pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker '1137@hostname.domain.tld' discovered
May 3 15:29:55 hostname pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker '1153@hostname.domain.tld' discovered
May 3 15:29:55 hostname pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:INFO: New worker '1133@hostname.domain.tld' discovered
May 3 15:29:55 hostname pulpcore-resource-manager: Traceback (most recent call last):
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/bin/rq", line 11, in <module>
May 3 15:29:55 hostname pulpcore-resource-manager: load_entry_point('rq==1.5.2', 'console_scripts', 'rq')()
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in call
May 3 15:29:55 hostname pulpcore-resource-manager: return self.main(*args, **kwargs)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
May 3 15:29:55 hostname pulpcore-resource-manager: rv = self.invoke(ctx)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
May 3 15:29:55 hostname pulpcore-resource-manager: return _process_result(sub_ctx.command.invoke(sub_ctx))
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
May 3 15:29:55 hostname pulpcore-resource-manager: return ctx.invoke(self.callback, **ctx.params)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
May 3 15:29:55 hostname pulpcore-resource-manager: return callback(*args, **kwargs)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/rq/cli/cli.py", line 78, in wrapper
May 3 15:29:55 hostname pulpcore-resource-manager: return ctx.invoke(func, cli_config, *args[1:], **kwargs)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
May 3 15:29:55 hostname pulpcore-resource-manager: return callback(*args, **kwargs)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/rq/cli/cli.py", line 268, in worker
May 3 15:29:55 hostname pulpcore-resource-manager: max_jobs=max_jobs, with_scheduler=with_scheduler)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 508, in work
May 3 15:29:55 hostname pulpcore-resource-manager: self.register_birth()
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/pulpcore/tasking/worker.py", line 171, in register_birth
May 3 15:29:55 hostname pulpcore-resource-manager: return super().register_birth(*args, **kwargs)
May 3 15:29:55 hostname pulpcore-resource-manager: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 273, in register_birth
May 3 15:29:55 hostname pulpcore-resource-manager: raise ValueError(msg.format(self.name))
May 3 15:29:55 hostname pulpcore-resource-manager: ValueError: There exists an active worker named 'resource-manager' already
May 3 15:29:55 hostname pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:55 hostname pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp
will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:55 hostname pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:55 hostname pulpcore-worker-3: pulp: rq.worker:INFO: Cleaning registries for queue: 1141@hostname.domain.tld
May 3 15:29:55 hostname pulpcore-worker-2: pulp: rq.worker:INFO: Cleaning registries for queue: 1137@hostname.domain.tld
May 3 15:29:55 hostname pulpcore-worker-4: pulp: rq.worker:INFO: Cleaning registries for queue: 1153@hostname.domain.tld
May 3 15:29:55 hostname pulpcore-worker-4: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:56 hostname pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:56 hostname pulpcore-worker-1: pulp: rq.worker:INFO: Cleaning registries for queue: 1133@hostname.domain.tld
May 3 15:29:56 hostname pulpcore-worker-2: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:56 hostname pulpcore-worker-3: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:56 hostname pulpcore-worker-1: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
May 3 15:29:56 hostname systemd: pulpcore-resource-manager.service: main process exited, code=exited, status=1/FAILURE
May 3 15:29:56 hostname systemd: Unit pulpcore-resource-manager.service entered failed state.
May 3 15:29:56 hostname systemd: pulpcore-resource-manager.service failed.
May 3 15:29:59 hostname systemd: pulpcore-resource-manager.service holdoff time over, scheduling restart.

Is this a known issue? What can I do about it?

sincerely yours

icingauserm