Project

General

Profile

Bug #14601

virt-who hypervisor check in failing

Added by Tyler Milam about 6 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Difficulty:
Triaged:
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:
Red Hat JIRA:

Description

virt-who sends vmware hypervisor information to Katello, but the task fails with "Runtime Error null at org.candlepin.resource.HypervisorResource.hypervisorCheckIn:137"

#Trackeback from /var/log/rhsm/rhsm.log:
2016-04-12 21:29:23,049 [ERROR] @virtwho.py:123 - Error in communication with subscription manager: Traceback (most recent call last): File "/usr/share/virt-who/virtwho.py", line 107, in send self._sendGuestAssociation(report) File "/usr/share/virt-who/virtwho.py", line 134, in _sendGuestAssociation result = manager.hypervisorCheckIn(report.config, report.association, report.config.type) File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 152, in hypervisorCheckIn return self.connection.hypervisorCheckIn(config.owner, config.env, serialized_mapping) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 903, in hypervisorCheckIn res = self.conn.request_post(url, host_guest_mapping) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 647, in request_post return self._request("POST", method, params) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 564, in _request self.validateResponse(result, request_type, handler) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 611, in validateResponse raise RestlibException(response['status'], error_msg, response.get('headers')) RestlibException: Task 273eec3c-ae92-4f8c-b8d2-3e69c6007aa3: Katello::Errors::CandlepinError: Runtime Error null at org.candlepin.resource.HypervisorResource.hypervisorCheckIn:137
Traceback from foreman task:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:inrescue in propagate_candlepin_errors'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:19:in propagate_candlepin_errors' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:inrun'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/remote_action.rb:16:inblock in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/models/katello/concerns/user_extensions.rb:20:incp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/remote_action.rb:39:inas_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/remote_action.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:30:in with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action/progress.rb:16:inrun'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/keep_locale.rb:11:inblock in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/keep_locale.rb:22:in with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.1.0/app/lib/actions/middleware/keep_locale.rb:11:inrun'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:26:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:17:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware.rb:30:inrun'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/middleware/world.rb:30:inexecute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:505:in block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:incatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:504:in block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:incatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:419:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:499:inexecute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/action.rb:260:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:inblock (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract.rb:155:inwith_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:inopen_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:15:inblock in on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in block in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:intap'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:inmatch_value'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in block in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:ineach'
/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:inmatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/executors/parallel/worker.rb:12:in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:inon_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.10/lib/dynflow/actor.rb:26:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:inprocess_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:inpass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:inprocess_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in block in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:inblock (2 levels) in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/synchronization/mri_lockable_object.rb:38:in block in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:inblock in schedule_execution'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/serialized_execution.rb:18:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/serialized_execution.rb:18:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/serialized_execution.rb:96:in work' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/serialized_execution.rb:77:inblock in call_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:inrun_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in block (3 levels) in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:inloop'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in block (2 levels) in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:incatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.1/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in block in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:incall'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in block in create_with_logging_context'

/etc/sysconfig/virt-who:
`
VIRTWHO_BACKGROUND=1

VIRTWHO_DEBUG=0

VIRTWHO_SATELLITE6=0

VIRTWHO_ESX=0

VIRTWHO_ESX_OWNER=6972987

VIRTWHO_ESX_ENV=Library

VIRTWHO_ESX_SERVER=domain.org

VIRTWHO_ESX_USERNAME=DOMAIN\VCentersvcaccount

VIRTWHO_ESX_PASSWORD=secretpassword
`

#/etc/virt-who.d/vsphere.conf:
`
[vsphere]

type=esx ; insert one of libvirt/esx/hyperv/rhevm/vdsm/fake

server=vsphere.domain.org ; insert hostname or ip address of the server to connect to

username=DOMAIN\VCentersvcaccount ; username for server authentication

password=ultrasecretpassword ; password for server authentication

owner=admin ; owner for use with SAM, Customer Portal, or Satellite 6

env=Library ; environment for use with SAM, Customer Portal, or Satellite 6

hypervisor_id=hostname ; how will be the hypervisor identified, one of: uuid, hostname, hwuuid
`

History

#1 Updated by Dominic Cleal about 6 years ago

  • Project changed from Foreman to Katello
  • Category deleted (Reporting)

#2 Updated by Eric Helms about 6 years ago

  • Category set to 78
  • Assignee set to Thomas McKay
  • Legacy Backlogs Release (now unused) set to 144

#3 Updated by Eric Helms about 6 years ago

  • Legacy Backlogs Release (now unused) changed from 144 to 168

#4 Updated by Eric Helms about 6 years ago

  • Legacy Backlogs Release (now unused) deleted (168)

#5 Updated by Klaas D about 6 years ago

I think I'm experiencing the same bahavoir with current stable release (katello 3.0)

virt-who -d output:

2016-06-30 15:57:44,053 ERROR: Communication with server failed:
Traceback (most recent call last):
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 167, in hypervisorCheckIn
    result = self.connection.hypervisorCheckIn(config.owner, config.env, serialized_mapping)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 903, in hypervisorCheckIn
    res = self.conn.request_post(url, host_guest_mapping)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 647, in request_post
    return self._request("POST", method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 564, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 611, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Task f7babe9d-4c01-4002-a83b-cdb417f73f19: Katello::Errors::CandlepinError: Runtime Error null
2016-06-30 15:57:44,053 ERROR: Unable to send data: Communication with subscription manager failed with code 500: Task f7babe9d-4c01-4002-a83b-cdb417f73f19: Katello::Errors::CandlepinError: Runtime Error null
2016-06-30 15:57:44,054 ERROR: Sending data failed 5 times, report skipped
2016-06-30 15:57:44,055 DEBUG: Association for config vcenter.tld.com gathered
2016-06-30 15:57:45,057 DEBUG: virt-who shut down started

candlepin Log:

2016-06-30 15:57:34,176 [thread=http-bio-8443-exec-7] [req=f39e484b-c8b8-4595-95bc-9e16ae862400, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2016-06-30 15:57:34,178 [thread=http-bio-8443-exec-7] [req=f39e484b-c8b8-4595-95bc-9e16ae862400, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=2
2016-06-30 15:57:34,206 [thread=http-bio-8443-exec-4] [req=cc3fcd55-9227-40e0-bb49-d7851db3f149, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=GET, uri=/candlepin/status
2016-06-30 15:57:34,207 [thread=http-bio-8443-exec-4] [req=cc3fcd55-9227-40e0-bb49-d7851db3f149, org=] INFO  org.candlepin.common.filter.LoggingFilter - Response: status=200, content-type="application/json", time=2
2016-06-30 15:57:34,523 [thread=http-bio-8443-exec-2] [req=4e432e53-1de2-4535-abc0-c0d29e6863b1, org=] INFO  org.candlepin.common.filter.LoggingFilter - Request: verb=POST, uri=/candlepin/hypervisors?owner=admin&env=Library
2016-06-30 15:57:34,550 [thread=http-bio-8443-exec-2] [req=4e432e53-1de2-4535-abc0-c0d29e6863b1, org=ORG] INFO  org.candlepin.resource.HypervisorResource - Hypervisor check-in by principal: {"type":"trusteduser","name":"foreman_admin"}
2016-06-30 15:57:34,561 [thread=http-bio-8443-exec-2] [req=4e432e53-1de2-4535-abc0-c0d29e6863b1, org=ORG] ERROR org.candlepin.common.exceptions.mappers.CandlepinExceptionMapper - Runtime Error null
java.lang.NullPointerException: null

production.log

2016-06-30 16:05:28 [app] [I] Completed 500 Internal Server Error in 1546ms (Views: 0.3ms | ActiveRecord: 9.0ms)
2016-06-30 16:05:28 [app] [I] Started GET "/rhsm/status/" for 10.165.1.21 at 2016-06-30 16:05:28 +0200
2016-06-30 16:05:28 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2016-06-30 16:05:28 [app] [I] Completed 200 OK in 142ms (Views: 0.4ms | ActiveRecord: 0.0ms)
2016-06-30 16:05:28 [app] [I] Started GET "/rhsm/status" for 10.165.1.21 at 2016-06-30 16:05:28 +0200
2016-06-30 16:05:28 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#server_status as JSON
2016-06-30 16:05:28 [app] [I] Completed 200 OK in 139ms (Views: 0.4ms | ActiveRecord: 0.0ms)
2016-06-30 16:05:28 [app] [I] Started POST "/rhsm/hypervisors?owner=admin&env=Library" for 10.165.1.21 at 2016-06-30 16:05:28 +0200
2016-06-30 16:05:28 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#hypervisors_update as JSON
2016-06-30 16:05:28 [app] [I]   Parameters: {"esx001.tld.com"=>[{"guestId"=>"423a5b5c-1df0-84bf-3dad-d0250322db72", "state"=>1, "attributes"=>{"active"=>1, "virtWhoTyp
e"=>"esx", "hypervisorType"=>"vmware"}}, ...
2016-06-30 16:08:38 [foreman-tasks/action] [E] Runtime Error null (Katello::Errors::CandlepinError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:21:in `rescue in propagate_candlepin_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:19:in `propagate_candlepin_errors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/propagate_candlepin_errors.rb:9:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/models/katello/concerns/user_extensions.rb:20:in `cp_config'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:30:in `with_progress_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action/progress.rb:16:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in `block in run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:26:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:17:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware.rb:30:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/stack.rb:22:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/middleware/world.rb:30:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:505:in `block (2 levels) in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:504:in `block in execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:419:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:499:in `execute_run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/action.rb:260:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'
  | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/executors/parallel/worker.rb:12:in `on_message'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.11/lib/dynflow/actor.rb:26:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in `process_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in `block in on_envelope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in `block in schedule_execution'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in `work'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in `run_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in `block (3 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `loop'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in `block (2 levels) in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `block in create_worker'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
 2016-06-30 16:08:38 [katello/cp_proxy] [E] ForemanTasks::TaskError: Task b25d1915-d09e-4e5b-abdb-900663603a4e: Katello::Errors::CandlepinError: Runtime Error null
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:40:in `block in sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks.rb:39:in `sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.7.18/lib/foreman_tasks/triggers.rb:22:in `sync_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:116:in `block in hypervisors_update'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:72:in `as'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:115:in `hypervisors_update'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/base.rb:189:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/rendering.rb:10:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:113:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:113:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/controllers/katello/api/rhsm/candlepin_proxies_controller.rb:27:in `repackage_message'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
  | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:152:in `disable_json_root'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in `around'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:447:in `public_send'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:447:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in `set_timezone'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:424:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:229:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:166:in `block in halting'
  | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:299:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in `around'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:447:in `public_send'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:447:in `block in make_lambda'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:298:in `block in halting'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:149:in `block in halting_and_conditional'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:86:in `run_callbacks'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/callbacks.rb:19:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/rescue.rb:29:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/notifications.rb:159:in `block in instrument'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/notifications.rb:159:in `instrument'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.1/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/abstract_controller/base.rb:136:in `process'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionview-4.1.5/lib/action_view/rendering.rb:30:in `process'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal.rb:196:in `dispatch'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_controller/metal.rb:232:in `block in action'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/routing/route_set.rb:82:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/routing/route_set.rb:50:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:71:in `block in call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:59:in `each'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:59:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/routing/route_set.rb:678:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/engine.rb:514:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in `public_send'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in `method_missing'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:71:in `block in call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:59:in `each'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/journey/router.rb:59:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/routing/route_set.rb:678:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/extractor/recorder.rb:132:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
  | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/etag.rb:23:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/conditionalget.rb:35:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/head.rb:11:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/flash.rb:254:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:in `context'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/cookies.rb:560:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/query_cache.rb:36:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/callbacks.rb:82:in `run_callbacks'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/rack/logger.rb:22:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/request_id.rb:21:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/runtime.rb:17:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/cache/strategy/local_cache_middleware.rb:26:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/action_dispatch/middleware/static.rb:64:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/engine.rb:514:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/application.rb:144:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in `public_send'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in `method_missing'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/builder.rb:138:in `call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in `block in call'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `each'
 | /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2016-06-30 16:08:38 [app] [I] Completed 500 Internal Server Error in 1543ms (Views: 0.3ms | ActiveRecord: 9.1ms)

virt-who-0.14-9.el7_2.1.noarch

candlepin-0.9.54.4-1.el7.noarch
candlepin-selinux-0.9.54.4-1.el7.noarch
foreman-installer-katello-3.0.2-1.el7.noarch
katello-3.0.0-9.el7.noarch
katello-agent-2.5.0-3.el7.noarch
katello-ca-consumer-deployc001.srv.muenchen.de-1.0-1.noarch
katello-certs-tools-2.4.0-1.el7.noarch
katello-client-bootstrap-1.1.0-1.el7.noarch
katello-common-3.0.0-9.el7.noarch
katello-debug-3.0.0-9.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.0.2-1.el7.noarch
katello-repos-3.0.0-3.el7.noarch
katello-selinux-3.0.1-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-service-3.0.0-9.el7.noarch
pulp-katello-1.0-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.0.25-1.el7.noarch
tfm-rubygem-katello-3.0.1-1.el7.noarch

#6 Updated by Dylan Baars almost 6 years ago

I am seeing the same thing (katello 3.0), same errors as Klaas in my logs. Happy to supply any logs/details that may help you out just let me know what you need

Dylan

#7 Updated by Klaas D almost 6 years ago

Thomas McKay suspected missing backports of fixes and told me to try latest nightly. To make it short, latest nightly works and content hosts get added correctly (although names have changed but I suspect thats expected behaviour (hostname to virt-who-hostname-1).
Not sure how we should continue now - is this backportable to stable or do we have to wait for the next release to fix this issue?

#8 Updated by Eric Helms almost 6 years ago

  • Status changed from New to Resolved
  • Legacy Backlogs Release (now unused) set to 168

Klaas reports that candlepin-0.9.54.6 fixes this issue, so I am moving this into 3.0.2 to tag in this newer version of Candlepin to be released and resolve this issue.

#9 Updated by Klaas D almost 6 years ago

I manually upgraded candlepin and candlepin-selinux and it works again in katello 3.0

  1. rpm -qa|grep candlepin
    candlepin-0.9.54.6-1.el7.noarch
    candlepin-selinux-0.9.54.6-1.el7.noarch

I used the two rpms from the katello nightly repository.

Also available in: Atom PDF