Apologies if a bit long winded.

Previously we had one VMware service account per Datacentre that Foreman used to manage Virtual Machines. To separate the selectable resources and only show the organisations/teams what is applicable to them (data stores, port groups and VM folders, etc), we created a VMware service account per VMware cluster, assigning specific permissions to appropriate VMware cluster, specific port groups, folders and datastores. This means when one chooses the Compute Resource, you now choose by cluster name rather than datacentre name.

Everything works as expected (only applicable data stores, folders, port groups are displayed in "New Host" -> "Virtual Machine", only Virtual Machines in that cluster are shown in "Compute Resource -> $clustername -> "Virtual Machines"), except provisioning a new Virtual Machine. When creating a new VM the first step fails, and following is displayed:

Unable to save
Failed to create a compute $Compute_Resource (VMWare) instance $guest_fqdn: failed to create vm: undefined method `uuid' for nil:NilClass

Looking at production.log:

Started POST "/hosts" for 10.0.70.34 at Sun Jan 26 22:12:49 +0000 2014
Processing by HostsController#create as */*
  Parameters: {"host"=>{"organization_id"=>"1", "operatingsystem_id"=>"1", "is_owned_by"=>"", "mac"=>"", "disk"=>"", "name"=>"stephendev", "compute_resource_id"=>"2", "type"=>"Host::Managed", "medium_id"=>"3", "root_pass"=>"[FILTERED]", "overwrite"=>"false", "domain_id"=>"25", "provision_method"=>"build", "environment_id"=>"2", "puppetclass_ids"=>[""], "subnet_id"=>"18", "ip"=>"10.0.83.22", "build"=>"1", "ptable_id"=>"9", "comment"=>"", "progress_report_id"=>"[FILTERED]", "managed"=>"true", "hostgroup_id"=>"4", "puppet_ca_proxy_id"=>"1", "interfaces_attributes"=>{"new_interfaces"=>{"mac"=>"", "type"=>"Nic::Managed", "name"=>"", "_destroy"=>"false", "domain_id"=>"", "provider"=>"IPMI", "ip"=>""}}, "architecture_id"=>"1", "puppet_proxy_id"=>"1", "compute_attributes"=>{"path"=>"/Datacenters/HOD/vm/Development/Ops", "cpus"=>"1", "cluster"=>"HOD-DEV-SYSOPS", "volumes_attributes"=>{"new_volumes"=>{"size_gb"=>"8", "name"=>"Hard disk", "thin"=>"true", "datastore"=>"vmware-hod-fs50-sp2", "_delete"=>""}, "0"=>{"size_gb"=>"8", "name"=>"Hard disk", "datastore"=>"vmware-hod-fs50-sp2", "_delete"=>""}}, "memory_mb"=>"1024", "interfaces_attributes"=>{"new_interfaces"=>{"type"=>"VirtualE1000", "network"=>"dvportgroup-1061", "_delete"=>""}, "0"=>{"type"=>"VirtualE1000", "network"=>"dvportgroup-1061", "_delete"=>""}}}, "enabled"=>"1"}, "authenticity_token"=>"4YJwnJdGNFCnpSDuCrPs4eU2U2L8ts5t2+JhvW5rpAI=", "capabilities"=>"build", "utf8"=>"✓"}
Imported report for srvctgdc01.ctg.local in 0.15 seconds
Completed 201 Created in 155.7ms (Views: 2.2ms | ActiveRecord: 0.0ms)
Adding Compute instance for stephendev.rd.eu.domain.tld
Failed to create a compute HOD-DEV (VMWare) instance stephendev.rd.eu.domain.tld: failed to create vm: undefined method `uuid' for nil:NilClass
 /usr/share/foreman/vendor/ruby/1.8/gems/fog-1.19.0/lib/fog/vsphere/requests/compute/create_vm.rb:27:in `create_vm'
/usr/share/foreman/vendor/ruby/1.8/gems/fog-1.19.0/lib/fog/vsphere/models/compute/server.rb:234:in `save'
/usr/share/foreman/app/models/compute_resources/foreman/model/vmware.rb:110:in `create_vm'
/usr/share/foreman/app/models/concerns/orchestration/compute.rb:59:in `setCompute'
/usr/share/foreman/app/models/concerns/orchestration.rb:148:in `send'
/usr/share/foreman/app/models/concerns/orchestration.rb:148:in `execute'
/usr/share/foreman/app/models/concerns/orchestration.rb:88:in `process'
/usr/share/foreman/app/models/concerns/orchestration.rb:80:in `each'
/usr/share/foreman/app/models/concerns/orchestration.rb:80:in `process'
/usr/share/foreman/app/models/concerns/orchestration.rb:18:in `on_save'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:638:in `_run__524528178__save__4__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/callbacks.rb:264:in `create_or_update'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/persistence.rb:84:in `save'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/validations.rb:50:in `save'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/attribute_methods/dirty.rb:22:in `save'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:259:in `save_without_type'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:313:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:208:in `transaction'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:259:in `save_without_type'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:270:in `rollback_active_record_state!'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/transactions.rb:258:in `save_without_type'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in `save'
/usr/share/foreman/app/controllers/hosts_controller.rb:89:in `create'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/abstract_controller/base.rb:167:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/rendering.rb:10:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/abstract_controller/callbacks.rb:18:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:527:in `_run__1998537858__process_action__218180046__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:215:in `_conditional_callback_around_5392'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:310:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:310:in `_callback_around_807'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_5392'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:526:in `_run__1998537858__process_action__218180046__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:215:in `_conditional_callback_around_5391'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_5391'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:415:in `_run__1998537858__process_action__218180046__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:215:in `_conditional_callback_around_5390'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:326:in `around'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:310:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_5390'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:414:in `_run__1998537858__process_action__218180046__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/abstract_controller/callbacks.rb:17:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/rescue.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/notifications.rb:123:in `instrument'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/notifications.rb:123:in `instrument'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/abstract_controller/base.rb:121:in `process'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/abstract_controller/rendering.rb:45:in `process'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal.rb:203:in `dispatch'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_controller/metal.rb:246:in `action'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/routing/route_set.rb:73:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/routing/route_set.rb:36:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/journey-1.0.4/lib/journey/router.rb:68:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/usr/share/foreman/vendor/ruby/1.8/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/routing/route_set.rb:608:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/apipie-rails-0.0.24/lib/apipie/static_dispatcher.rb:65:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/etag.rb:23:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/conditionalget.rb:35:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/head.rb:14:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/flash.rb:242:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/cookies.rb:341:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/query_cache.rb:64:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/activerecord-3.2.16/lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/callbacks.rb:28:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `_run__1609753012__call__4__callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:405:in `__run_callback'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/callbacks.rb:81:in `run_callbacks'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/rack/logger.rb:32:in `call_app'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/rack/logger.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/tagged_logging.rb:22:in `tagged'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/rack/logger.rb:16:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/activesupport-3.2.16/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/actionpack-3.2.16/lib/action_dispatch/middleware/static.rb:63:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/engine.rb:484:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/application.rb:231:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/railtie/configurable.rb:30:in `send'
/usr/share/foreman/vendor/ruby/1.8/gems/railties-3.2.16/lib/rails/railtie/configurable.rb:30:in `method_missing'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/urlmap.rb:64:in `call'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `each'
/usr/share/foreman/vendor/ruby/1.8/gems/rack-1.4.5/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/1.8/phusion_passenger/rack/request_handler.rb:92:in `process_request'
/usr/lib/ruby/1.8/phusion_passenger/abstract_request_handler.rb:207:in `main_loop'
/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:118:in `run'
/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:65:in `spawn_application'
/usr/lib/ruby/1.8/phusion_passenger/utils.rb:184:in `safe_fork'
/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:58:in `spawn_application'
/usr/lib/ruby/1.8/phusion_passenger/rack/application_spawner.rb:41:in `spawn_application'
/usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb:159:in `spawn_application'
/usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb:287:in `handle_spawn_application'
/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:352:in `__send__'
/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:352:in `main_loop'
/usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb:196:in `start_synchronously'
/usr/lib/phusion_passenger/passenger-spawn-server:61
Rolling back due to a problem: Set up compute instance stephendev.rd.eu.domain.tld     2     failed     stephendev.rd.eu.domain.tldsetCompute
Failed to save: Failed to create a compute HOD-DEV (VMWare) instance stephendev.rd.eu.domain.tld: failed to create vm: undefined method `uuid' for nil:NilClass

After a fair bit of testing, all VMware functions seem to work except creating a VM. Logging in as the Foreman user with the vSphere client and creating a Virtual Machine by hand works as expected so isn't a VMware permission.

To replicate, set the following permissions using the vSphere client:

Under Hosts & Clusters:

• Add foreman user as "Administrator" to the top of the hierarchy, deselect propagate
• Add foreman user as "Administrator" to a datacentre, deselect propagate
• Add foreman user as "Administrator" to a cluster under datacentre, propagate

Under VMs & Templates:

• Add foreman user as "Administrator" to a folder under same datacentre, propagate

Under Datastores * Datastore Clusters:

• Add foreman user as "Administrator" to a datastore under same datacentre, propagate

Under Networking:

• Add foreman user as "Administrator" to a standard port group or VDS port group under same datacentre, propagate

The fix at the moment is to add foreman user as "Administrator" to the top of the hierarchy and propagate permissions down, this results in all available.