Project

General

Profile

Bug #11324

Pulp: endless stream of Resetting dropped connection in Logfiles when syncing repo

Added by Sebastian @ over 4 years ago. Updated 3 months ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
Repositories
Target version:
-
Difficulty:
Triaged:
Yes
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Pulp throws an endless stream of localhost pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: when a repo is synced currently.
Usual rate is 5-10 log events per second.

I believe this behavior slow the overall sync process + it kinda kills the origin server.
I had a test case on a local server where the load went up to 20 instantly with a huge mass of httpd processes spawning.

Changing the pulp worker count does not fix this issue entirely, even with 1 worker it still hammers the target repo server.

History

#1 Updated by Eric Helms about 4 years ago

  • Legacy Backlogs Release (now unused) set to 77
  • Triaged changed from No to Yes

#2 Updated by Stephen Benjamin about 4 years ago

  • Legacy Backlogs Release (now unused) changed from 77 to 78

#3 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) changed from 78 to 113

#4 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) changed from 113 to 86

#5 Updated by Greg Koch over 3 years ago

We are also seeing this issue. Our capsule is connected remotely via a VPN over the internet, which accounts for some latency and delay, but I'm not sure it accounts for all the reset connections. The sync seems to continue fine, but due to the speed of the VPN connection, the sync timesout after 12 hours.

Apr  7 07:29:53 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:31:10 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:31:25 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:31:25 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:31:29 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:33:14 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:33:17 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:33:41 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:33:57 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:34:09 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:34:27 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:35:37 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:35:56 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:36:45 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:36:47 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:37:22 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:37:59 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:38:12 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com
Apr  7 07:38:42 capsule pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: katello.example.com

#6 Updated by Justin Sherrill over 3 years ago

Greg,

The 12 hour timeout can be adjusted under administer > settings > Katello > pulp_sync_node_action_finish_timeout

I think it is unrelated to these dropped connections.

-Justin

#7 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) changed from 86 to 144

#8 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) changed from 144 to 168

#9 Updated by Eric Helms over 3 years ago

  • Legacy Backlogs Release (now unused) changed from 168 to 114

#10 Updated by John Mitsch 3 months ago

  • Target version deleted (Katello Backlog)
  • Status changed from New to Rejected

Thanks for reporting this issue. This issue was created over 4 years ago and hasn't seen an update in 1 year. We are closing this in an effort to keep a realistic backlog. Please open up a new issue that includes a link to this issue if you feel this still needs to be addressed. We can then triage the new issue and reassess.

Also available in: Atom PDF