Dominic Cleal wrote:

Tentatively setting to 1.11.0 as it may be a regression, though I can't tell where. The change in #5510 only appears to be in the network provisioning code path, I can't see where it could have changed image provisioning.

Was the template you're building from created from a Foreman-created VM or something?

In addition (may be related), once the machine is provisioned via PXE, it still tries to boot from network first.

This is by design, see #5510. It allows the host to be rebuilt from Foreman.

Yes, correct. The image was built by Foreman as well.

Any hints on how to see what's being sent via API ? I tried to change boot order of the template and lower HW version but it didn't help.

There's not an easy way, only http://projects.theforeman.org/projects/foreman/wiki/Troubleshooting#Compute-resource-debugging which may not work properly on a production installation.

I don't think it's anything Foreman is doing during clone, as the code change appears to only be used during network provisioning. My guess is that it's stored inside the template somehow.

I was able to get around the same problem by deleting these two options from the vmx:

bios.hddOrder
bios.bootOrder

The default boot order seems to be defined by the default bios settings. This may not work for people who have non-default or special a special boot order. There is a VMWare article that describes these options:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2011654

I had also this issue with 1.11.0. As workaround I commented out line 23 in file /opt/theforeman/tfm/root/usr/share/gems/gems/fog-vsphere-0.6.1/lib/fog/vsphere/requests/compute/create_vm.rb, where boot options are reconfigured:

20           vm_cfg[:cpuHotAddEnabled] = attributes[:cpuHotAddEnabled] if attributes.key?(:cpuHotAddEnabled)
21           vm_cfg[:memoryHotAddEnabled] = attributes[:memoryHotAddEnabled] if attributes.key?(:memoryHotAddEnabled)
22           vm_cfg[:firmware] = attributes[:firmware] if attributes.key?(:firmware)
23           #vm_cfg[:bootOptions] = boot_options(attributes) if attributes.key?(:boot_order) || attributes.key?(:boot_retry) # WORKAROUND
24           resource_pool = if attributes[:resource_pool] && attributes[:resource_pool] != 'Resources'
25                             get_raw_resource_pool(attributes[:resource_pool], attributes[:cluster], attributes[:datacenter])
26                           else
27                             get_raw_cluster(attributes[:cluster], attributes[:datacenter]).resourcePool
28                           end

Maybe we could add a checkbox to set whether we want to hardcode bootoptions. While by default it's useful, when we want to build a host that we plan to covert to a template we could disable it. Chances are we won't rebuild it so the boot order does not matter.

This seems like a VMware bug to me. I can't even delete disks from a host that was built via foreman because the parameter configSpec.bootOptions.bootOrder

According to my research this error happens because the boot order setting referenced both storage volumes and network interfaces. During a clone operation via Foreman, the interface of the vm is replaced. The interface from the template is deleted and a new interface is added. The boot order reference is not updated and therefor invalid.

I did not find a way to actually modify the boot order of a vm during cloning, however I found a workaround for this: Instead of deleting the interface from the template and adding a new one, it's possible to update/edit the interface. Therefore the boot order reference doesn't get broken.
I have developed a patch for fog-vsphere that changes the behaviour accordingly. I'll open a PR once i have cleaned it up.

This PR should fix this issue: https://github.com/fog/fog-vsphere/pull/98