Bug #19938: race condition when starting pulp_resource_manager and pulp_celerybeat after installation - Katello - Foreman

Actions

Copy link

Bug #19938

closed

race condition when starting pulp_resource_manager and pulp_celerybeat after installation

Added by Evgeni Golov over 7 years ago. Updated over 6 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Evgeni Golov

Category:

Installer

Target version:

Katello 3.4.5

Difficulty:

Triaged:

Bugzilla link:

Pull request:

https://github.com/Katello/puppet-pulp/pull/239

Fixed in Releases:

Found in Releases:

Red Hat JIRA:

Description

while working on cleaning up forklift (https://github.com/theforeman/forklift/pull/465), I stumbled over the fact that pulp_resource_manager and pulp_celerybeat would not properly start up after installation

resource_manager:

pulp[18766]: celery.worker:ERROR: (18766-09440) Unrecoverable error: InternalError('Traceback (most recent call last):\n  File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch\n    self.connect()\n  File "/usr/lib/python2.7/site
pulp[18766]: celery.worker:ERROR: (18766-09440) Traceback (most recent call last):
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     self.blueprint.start(self)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     step.start(parent)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 373, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     return self.obj.start()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     blueprint.start(self)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     step.start(parent)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 478, in start
pulp[18766]: celery.worker:ERROR: (18766-09440)     c.connection = c.connect()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/celery/worker/consumer.py", line 375, in connect
pulp[18766]: celery.worker:ERROR: (18766-09440)     callback=maybe_shutdown,
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
pulp[18766]: celery.worker:ERROR: (18766-09440)     interval_start, interval_step, interval_max, callback)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
pulp[18766]: celery.worker:ERROR: (18766-09440)     return fun(*args, **kwargs)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
pulp[18766]: celery.worker:ERROR: (18766-09440)     return self.connection
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
pulp[18766]: celery.worker:ERROR: (18766-09440)     self._connection = self._establish_connection()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
pulp[18766]: celery.worker:ERROR: (18766-09440)     conn = self.transport.establish_connection()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
pulp[18766]: celery.worker:ERROR: (18766-09440)     conn = self.Connection(**opts)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
pulp[18766]: celery.worker:ERROR: (18766-09440)     self._qpid_conn = establish(**self.connection_options)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
pulp[18766]: celery.worker:ERROR: (18766-09440)     conn.open(timeout=timeout)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "<string>", line 6, in open
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
pulp[18766]: celery.worker:ERROR: (18766-09440)     self.attach(timeout=timeout)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "<string>", line 6, in attach
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
pulp[18766]: celery.worker:ERROR: (18766-09440)     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
pulp[18766]: celery.worker:ERROR: (18766-09440)     self.check_error()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
pulp[18766]: celery.worker:ERROR: (18766-09440)     raise e
pulp[18766]: celery.worker:ERROR: (18766-09440) InternalError: Traceback (most recent call last):
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
pulp[18766]: celery.worker:ERROR: (18766-09440)     self.connect()
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
pulp[18766]: celery.worker:ERROR: (18766-09440)     self._transport = trans(self.connection, host, port)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
pulp[18766]: celery.worker:ERROR: (18766-09440)     cert_reqs=validate)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket
pulp[18766]: celery.worker:ERROR: (18766-09440)     ciphers=ciphers)
pulp[18766]: celery.worker:ERROR: (18766-09440)   File "/usr/lib64/python2.7/ssl.py", line 549, in __init__
pulp[18766]: celery.worker:ERROR: (18766-09440)     self._context.load_cert_chain(certfile, keyfile)
pulp[18766]: celery.worker:ERROR: (18766-09440) IOError: [Errno 2] No such file or directory
pulp[18766]: celery.worker:ERROR: (18766-09440)
pulp[18766]: pulp.server.async.tasks:INFO: Cleaning up shutdown worker 'resource_manager@pipeline-katello-3-4-centos7.example.com'.

celerybeat:

pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448) Traceback (most recent call last):
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)     self.connect()
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)     self._transport = trans(self.connection, host, port)
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)   File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)     cert_reqs=validate)
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)   File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)     ciphers=ciphers)
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)   File "/usr/lib64/python2.7/ssl.py", line 549, in __init__
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)     self._context.load_cert_chain(certfile, keyfile)
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448) IOError: [Errno 2] No such file or directory
pulp[18483]: pulp.server.async.scheduler:ERROR: (18483-08448)

pulp[18483]: celery.beat:CRITICAL: (18483-65632) Traceback (most recent call last):
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in start_scheduler
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     beat.start()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     interval = self.scheduler.tick()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 292, in tick
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     ret = self.call_tick(self, CELERYBEAT_NAME)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line 233, in call_tick
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     ret = super(Scheduler, self).tick()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     next_time_to_run = self.maybe_due(entry, self.publisher)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in __get__
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     value = obj.__dict__[self.__name__] = self.__get(obj)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     return self.Publisher(self._ensure_connected())
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in _ensure_connected
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     _error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in ensure_connection
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     interval_start, interval_step, interval_max, callback)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     return fun(*args, **kwargs)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     return self.connection
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self._connection = self._establish_connection()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     conn = self.transport.establish_connection()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in establish_connection
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     conn = self.Connection(**opts)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in __init__
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self._qpid_conn = establish(**self.connection_options)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112, in establish
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     conn.open(timeout=timeout)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "<string>", line 6, in open
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323, in open
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self.attach(timeout=timeout)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "<string>", line 6, in attach
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 341, in attach
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     if not self._ewait(lambda: self._transport_connected and not self._unlinked(), timeout=timeout):
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 274, in _ewait
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self.check_error()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 267, in check_error
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     raise e
pulp[18483]: celery.beat:CRITICAL: (18483-65632) InternalError: Traceback (most recent call last):
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 545, in dispatch
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self.connect()
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/driver.py", line 572, in connect
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self._transport = trans(self.connection, host, port)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib/python2.7/site-packages/qpid/messaging/transports.py", line 120, in __init__
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     cert_reqs=validate)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib64/python2.7/ssl.py", line 936, in wrap_socket
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     ciphers=ciphers)
pulp[18483]: celery.beat:CRITICAL: (18483-65632)   File "/usr/lib64/python2.7/ssl.py", line 549, in __init__
pulp[18483]: celery.beat:CRITICAL: (18483-65632)     self._context.load_cert_chain(certfile, keyfile)
pulp[18483]: celery.beat:CRITICAL: (18483-65632) IOError: [Errno 2] No such file or directory
pulp[18483]: celery.beat:CRITICAL: (18483-65632)

forklift/bats was just hiding this from us, as it was rerunning the installer, and thus restarting the services during bats.

Related issues 1 (0 open — 1 closed)