Project

General

Profile

Bug #22671

Sync plans not running automatically

Added by Philipp Mueller over 2 years ago. Updated about 1 year ago.

Status:
Duplicate
Priority:
High
Assignee:
-
Category:
-
Target version:
Difficulty:
Triaged:
Yes
Bugzilla link:

Description

Hi,

on our Katello 3.5.1 system, sync plans are not running as scheduled. When i click "sync now" its working.
Deleting or adding new sync plans also doenst work.

Kind regards,
Philipp


Related issues

Is duplicate of Katello - Bug #24083: Add an upgrade rake task for sync plansClosed

History

#1 Updated by Philipp Mueller over 2 years ago

[root@XXX ~]# su - postgres c "psql -d foreman -c 'select * from katello_sync_plans;'"
id | name | description | sync_date | interval | organization_id | created_at | updated_at | enabled
---
--------------------------+---------------------+----------+-----------------+----------------------------+----------------------------+---------
5 | Daily | | 2017-11-08 11:21:00 | daily | 1 | 2017-11-08 08:22:38.246492 | 2017-12-29 15:34:42.042774 | t
2 | daily-gedis | | 2017-06-08 00:00:00 | daily | 29 | 2017-06-08 08:22:18.763727 | 2017-06-08 08:22:18.763727 | t
3 | daily | | 2017-06-22 02:35:00 | daily | 30 | 2017-06-22 13:17:01.277477 | 2017-06-22 13:17:01.277477 | t
6 | test | | 2017-11-15 11:11:00 | hourly | 1 | 2017-11-16 09:12:07.282427 | 2017-11-16 09:12:07.282427 | t

[root@xxx~]# pulp-admin u admin -p $pulpAdminPassword rpm repo sync schedules list --repo-id 1-Internet-CentOS7-v5_0-rs-REMI-REMI7
---------------------------------------------------------------------

Schedules
--------------------------------------------------------------------

There are no schedules defined for this operation.

#2 Updated by Andrew Kofink about 2 years ago

  • Legacy Backlogs Release (now unused) set to 339
  • Target version set to 160
  • Assignee set to Andrew Kofink

#3 Updated by Philipp Mueller about 2 years ago

I have found a workaround for this problem.

When i checked the repo schedules with pulp-admin, it appeared, that they are running at the correct times according to the sync-plan, which was created in katello.
The mechanism, that pulp uses, to report the sync_complete status to katello via the rest-api was not working, due to a ssl-certificate, that could not be verified.
We are using custom ssl-certificates and have imported the root-ca to the trusted root CAs.

In the pulp log, after every successfully scheduled sync, pulp fails with this error:
Mar 12 10:54:47 hostname pulp: requests.packages.urllib3.connectionpool:INFO: [2dec033d] Starting new HTTPS connection (1): hostname.domain.told
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) HTTP Notification Failed
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) Traceback (most recent call last):
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 57, in _send_post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) headers={'Content-Type': 'application/json'}, timeout=15)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) return request('post', url, data=data, json=json, *kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) response = session.request(method=method, url=url, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) resp = self.send(prep, **send_kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) r = adapter.send(request, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) raise SSLError(e, request=request)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) SSLError: [SSL: *CERTIFICATE_VERIFY_FAILED
] certificate verify failed (_ssl.c:579

The function for the post is found in: /usr/lib/python2.7/site-packages/pulp/server/event/http.py. After adding the parameter verify=false, the post works again and the Sync-Plans in the Katello frontend show their respective status.

def _send_post(notifier_config, json_body):

response = post(url, data=json_body, auth=auth,
headers={'Content-Type': 'application/json'}, timeout=15,*verify=False)*

http://projects.theforeman.org/issues/22671

#4 Updated by Andrew Kofink about 2 years ago

  • Legacy Backlogs Release (now unused) changed from 339 to 351

#5 Updated by Martin Angermeier almost 2 years ago

With Katello 3.6.0 this problem seems to occur again.

Kind regards,
Martin Angermeier

#6 Updated by Andrew Kofink almost 2 years ago

Thanks for the info. I'm also able to reproduce this on nightly Katello.

#7 Updated by Andrew Kofink almost 2 years ago

  • Triaged set to No
  • Target version changed from Katello 3.5.3 to Katello 3.9.0
  • Priority changed from Normal to High
  • Assignee deleted (Andrew Kofink)
  • Subject changed from Katello 3.5.1 - sync plans not running automatically to Sync plans not running automatically
  • Found in Releases Katello 3.5.0, Katello 3.5.1, Katello 3.5.2, Katello 3.5.3, Katello 3.6.0, Katello 3.6.1, Katello 3.7.0 added

This doesn't seem to be resolved in 3.6 or 3.7. I'm unsetting triaged so we'll review it this week in triage for inclusion in 3.9.

#8 Updated by Andrew Kofink almost 2 years ago

  • Is duplicate of Bug #24083: Add an upgrade rake task for sync plans added

#9 Updated by Andrew Kofink almost 2 years ago

  • Triaged changed from No to Yes
  • Target version changed from Katello 3.9.0 to Katello Recycle Bin
  • Status changed from New to Duplicate

#10 Updated by Ludwig Bayerlein about 1 year ago

  • Found in Releases Katello 3.7.1, Katello 3.8.1 added

This is a Bug in python-certifi. On our side, there was installed python-certifi with pip. We verified this with

# pip list|grep certifi
certifi        2018.10.15

Then I uninstalled it with

# pip uninstall certifi

And did a reinstall of python-requests

# yum reinstall -y python-requests

After this few steps, everything works great. The user @jsherrill explained that as a bug on the pip installed version of certifi, because it uses the wrong ca path.

Thanks for your support!

Also available in: Atom PDF