Plugins » Katello

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1160670
Description of problem:
timeout trying to process 218 hypervisors and 5607 guests between candlepin and katello

Version-Release number of selected component (if applicable):
satellite 6.0

How reproducible:
all the time in default config

Steps to Reproduce:
1. create an environemnt with 5000 guests shared between a large number of hypervisors (218 confirmed here)
2. use virt-who to populate the satellite content

Actual results:
(production log)

...
Processing by Katello::Api::Rhsm::CandlepinProxiesController#hypervisors_update as JSON
Parameters: {"34383733-3831-4742-3839-353344565741"=>["421722cb-3c07-ace7-b026-db187d622ab5", "422aa4f6-8504-d83d-31ca-e835a56ab5a5", "422a51f2-bf2e-b1f3-7e07-613697d8b47b", "422a374e-f03a-b0f2-3027-f669a3b5befb", "4213ace9-8156-4939-0d30-af6439ffa381", "4217d1de-1ec4-54d5-f1dd-40564f07048b", "421754eb-86c4-f825-17c1-689145f166df", "42170edb-63f8-f611-4fb6-a090a262990d", "42136d13-92af-0fba-ee85-fc54dfbbbb01", "4217ae63-d2ef-4e5e-55a4-ce0
...
...
RestClient::RequestTimeout: Katello::Resources::Candlepin::Consumer: Request Timeout (POST /candlepin/hypervisors?owner=itnow&env=Library)
Body:
...
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:56:in `rescue in transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rbovirt-0.0.29/lib/restclient_ext/request.rb:11:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/http_resource.rb:111:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/lib/katello/resources/candlepin.rb:150:in `register_hypervisors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/models/katello/glue/candlepin/consumer.rb:525:in `register_hypervisors'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-1.5.0/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:106:in `hypervisors_update'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:560:in `block (3 levels) in run_759513737194825094__process_action__3704380723127754382__callbacks'

(candlepin log)
2014-10-29 11:37:23,405 [req=359a2d3e-d854-4b6d-943f-d5ab81c9b307, org=itnow] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime
Error ClientAbortException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Broken pi
pe at java.net.SocketOutputStream.socketWrite0:-2
org.jboss.resteasy.spi.WriterException: ClientAbortException: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: ja
va.net.SocketException: Broken pipe

Expected results:
the registration passes

Additional info:
it looks like we need to redesign how such environments can be treated, possibly by trying to split it in multiple calls. I am not certain if this is something to do with virt-who or should be done by katello at this point. a workaround is to change the rest_client_timeout in /etc/foreman/plugins/katello.yaml and restart the appliance. This may also be a problem on the candlepin side depending on what we should expect out of it. I'm attaching the katello-debug from the affected appliance.