Project

General

Profile

Actions

Bug #22671

closed

Sync plans not running automatically

Added by Philipp Mueller over 6 years ago. Updated over 5 years ago.

Status:
Duplicate
Priority:
High
Assignee:
-
Category:
-
Target version:
Difficulty:
Triaged:
Yes

Description

Hi,

on our Katello 3.5.1 system, sync plans are not running as scheduled. When i click "sync now" its working.
Deleting or adding new sync plans also doenst work.

Kind regards,
Philipp


Related issues 1 (0 open1 closed)

Is duplicate of Katello - Bug #24083: Add an upgrade rake task for sync plansClosedPartha AjiActions
Actions #1

Updated by Philipp Mueller over 6 years ago

[root@XXX ~]# su - postgres c "psql -d foreman -c 'select * from katello_sync_plans;'"
id | name | description | sync_date | interval | organization_id | created_at | updated_at | enabled
---
--------------------------+---------------------+----------+-----------------+----------------------------+----------------------------+---------
5 | Daily | | 2017-11-08 11:21:00 | daily | 1 | 2017-11-08 08:22:38.246492 | 2017-12-29 15:34:42.042774 | t
2 | daily-gedis | | 2017-06-08 00:00:00 | daily | 29 | 2017-06-08 08:22:18.763727 | 2017-06-08 08:22:18.763727 | t
3 | daily | | 2017-06-22 02:35:00 | daily | 30 | 2017-06-22 13:17:01.277477 | 2017-06-22 13:17:01.277477 | t
6 | test | | 2017-11-15 11:11:00 | hourly | 1 | 2017-11-16 09:12:07.282427 | 2017-11-16 09:12:07.282427 | t

[root@xxx~]# pulp-admin u admin -p $pulpAdminPassword rpm repo sync schedules list --repo-id 1-Internet-CentOS7-v5_0-rs-REMI-REMI7
---------------------------------------------------------------------

Schedules
--------------------------------------------------------------------

There are no schedules defined for this operation.

Actions #2

Updated by Andrew Kofink over 6 years ago

  • Assignee set to Andrew Kofink
  • Target version set to 160
  • Translation missing: en.field_release set to 339
Actions #3

Updated by Philipp Mueller over 6 years ago

I have found a workaround for this problem.

When i checked the repo schedules with pulp-admin, it appeared, that they are running at the correct times according to the sync-plan, which was created in katello.
The mechanism, that pulp uses, to report the sync_complete status to katello via the rest-api was not working, due to a ssl-certificate, that could not be verified.
We are using custom ssl-certificates and have imported the root-ca to the trusted root CAs.

In the pulp log, after every successfully scheduled sync, pulp fails with this error:
Mar 12 10:54:47 hostname pulp: requests.packages.urllib3.connectionpool:INFO: [2dec033d] Starting new HTTPS connection (1): hostname.domain.told
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) HTTP Notification Failed
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) Traceback (most recent call last):
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 57, in _send_post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) headers={'Content-Type': 'application/json'}, timeout=15)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) return request('post', url, data=data, json=json, *kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) response = session.request(method=method, url=url, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) resp = self.send(prep, **send_kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) r = adapter.send(request, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) raise SSLError(e, request=request)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) SSLError: [SSL: *CERTIFICATE_VERIFY_FAILED
] certificate verify failed (_ssl.c:579

The function for the post is found in: /usr/lib/python2.7/site-packages/pulp/server/event/http.py. After adding the parameter verify=false, the post works again and the Sync-Plans in the Katello frontend show their respective status.

def _send_post(notifier_config, json_body):

response = post(url, data=json_body, auth=auth,
headers={'Content-Type': 'application/json'}, timeout=15,*verify=False)*

http://projects.theforeman.org/issues/22671

Actions #4

Updated by Andrew Kofink over 6 years ago

  • Translation missing: en.field_release changed from 339 to 351
Actions #5

Updated by Martin Angermeier over 6 years ago

With Katello 3.6.0 this problem seems to occur again.

Kind regards,
Martin Angermeier

Actions #6

Updated by Andrew Kofink over 6 years ago

Thanks for the info. I'm also able to reproduce this on nightly Katello.

Actions #7

Updated by Andrew Kofink over 6 years ago

  • Subject changed from Katello 3.5.1 - sync plans not running automatically to Sync plans not running automatically
  • Assignee deleted (Andrew Kofink)
  • Priority changed from Normal to High
  • Target version changed from Katello 3.5.3 to Katello 3.9.0
  • Triaged set to No
  • Found in Releases Katello 3.5.0, Katello 3.5.1, Katello 3.5.2, Katello 3.5.3, Katello 3.6.0, Katello 3.6.1, Katello 3.7.0 added

This doesn't seem to be resolved in 3.6 or 3.7. I'm unsetting triaged so we'll review it this week in triage for inclusion in 3.9.

Actions #8

Updated by Andrew Kofink over 6 years ago

  • Is duplicate of Bug #24083: Add an upgrade rake task for sync plans added
Actions #9

Updated by Andrew Kofink over 6 years ago

  • Status changed from New to Duplicate
  • Target version changed from Katello 3.9.0 to Katello Recycle Bin
  • Triaged changed from No to Yes
Actions #10

Updated by Ludwig Bayerlein over 5 years ago

  • Found in Releases Katello 3.7.1, Katello 3.8.1 added

This is a Bug in python-certifi. On our side, there was installed python-certifi with pip. We verified this with

# pip list|grep certifi
certifi        2018.10.15

Then I uninstalled it with

# pip uninstall certifi

And did a reinstall of python-requests

# yum reinstall -y python-requests

After this few steps, everything works great. The user @jsherrill explained that as a bug on the pip installed version of certifi, because it uses the wrong ca path.

Thanks for your support!

Actions

Also available in: Atom PDF