Bug #11106

Foreman 1.8.2 with VMware errors with: Could not find virtual machine network interface matching x.x.x.x

Added by Seth Doty over 2 years ago. Updated 11 days ago.

Status:Closed
Priority:Normal
Assigned To:Ivan Necas
Category:Compute resources - VMware
Target version:Team Ivan Iteration 7
Difficulty: Bugzilla link:1394290
Found in release:1.8.2 Pull request:https://github.com/theforeman/foreman/pull/2714
Story points-
Velocity based estimate-
Release1.14.1Release relationshipAuto

Description

I have been nagging the IRC channel with this one and the consensus is that its a bug. When building a VMware guest I receive: Could not find virtual machine network interface matching x.x.x.x.

Logs come back with : 2015-07-13 12:40:20 [W] Orchestration::Compute: Could not match network interface #<Nic::Managed id: nil, mac: nil, ip: "10.21.3.245", type: "Nic::Managed", name: "test.domain.local", host_id: nil, subnet_id: 1, domain_id: 1, attrs: {}, created_at: nil, updated_at: nil, provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: "", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: false, compute_attributes: {"type"=>"VirtualVmxnet3", "network"=>"pg-vlan-3"}>
2015-07-13 12:40:20 [W] Could not find virtual machine network interface matching 10.21.3.245
2015-07-13 12:40:20 [W] Rolling back due to a problem: [Query instance details for test.domain.local 4 failed [#<Host::Managed id: nil, name: "test.domain.local", last_compile: nil, last_freshcheck: nil, last_report: nil, updated_at: nil, source_file_id: nil, created_at: nil, root_pass: "", serial: nil, puppet_status: 0, architecture_id: 1, operatingsystem_id: 1, environment_id: 2, ptable_id: 14, medium_id: 8, build: true, comment: "", disk: "", installed_at: nil, model_id: nil, hostgroup_id: 1, owner_id: 3, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: "5032e796-3769-ee7b-3ed4-d2b036c20cb0", compute_resource_id: 1, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: 1, location_id: 5, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: 4, provision_method: "build", content_source_id: 1, grub_pass: "", content_view_id: 1, lifecycle_environment_id: 1, discovery_rule_id: nil>, :setComputeDetails]]
2015-07-13 12:40:20 [I] Removing Compute instance for test.domain.local.

This is a fresh install of foreman 1.8.2 with katello.

production.log (261 KB) Seth Doty, 07/14/2015 09:19 AM

production.log (528 KB) Sandro Roth, 06/16/2016 09:18 AM

Associated revisions

Revision d4af7cae
Added by Ivan Necas 11 months ago

Fixes #5483,#11106 - pass the virtualswitch value to fog

We can't pass the network id directly to the fog, but it accepts the
virtualswitch parameter, that we can use to point to the right value.

Also, it fixes another issue, where we used the vsphere network id to
match against the interface network value, in case of portgroup, but a
`key` value should be used instead, as the key and id might differ in
some situations.

Revision 443412c1
Added by Ivan Necas 10 months ago

Fixes #5483,#11106 - pass the virtualswitch value to fog

We can't pass the network id directly to the fog, but it accepts the
virtualswitch parameter, that we can use to point to the right value.

Also, it fixes another issue, where we used the vsphere network id to
match against the interface network value, in case of portgroup, but a
`key` value should be used instead, as the key and id might differ in
some situations.
(cherry picked from commit d4af7cae07f78bd047290e68af37b55122e0ed4b)

History

#1 Updated by Seth Doty over 2 years ago

I also just uploaded a debug file for reference to the debug-incomming directory: foreman-debug-GB7sy.tar.xz

#2 Updated by Dominic Cleal over 2 years ago

  • Category changed from Orchestration to Compute resources - VMware
  • Status changed from New to Need more information

Could you please enable debugging and try again? Some extra information showing which NICs were created on the VM will be printed to /var/log/foreman/production.log which you can attach to this ticket.

http://projects.theforeman.org/projects/foreman/wiki/Troubleshooting#How-do-I-enable-debugging

#3 Updated by Seth Doty over 2 years ago

Here you go. I ran a build twice, first setting the interface name in foreman and then I ran it again leaving it empty, only setting an IP address on the primary interface. It doesn't change much as far as errors, but I'm hoping it provides a bit more complete of a test.

#4 Updated by Dominic Cleal over 2 years ago

  • Status changed from Need more information to New

Extract from log:

2015-07-14 08:09:58 [D] Orchestration::Compute: Trying to match network interfaces from fog   <Fog::Compute::Vsphere::Interfaces
    server_id="50328ecb-196d-a21d-b661-88b38e822f0d" 
    [
                  <Fog::Compute::Vsphere::Interface
        mac="00:50:56:b2:7a:23",
        network="dvportgroup-87629",
        name="Network adapter 1",
        status="untried",
        summary="DVSwitch: 12 a9 12 50 6e 04 89 81-1c ec 7d 3a 09 1b 6b 0e",
        type=VirtualVmxnet3,
        key=4000,
        virtualswitch=nil,
        server_id="50328ecb-196d-a21d-b661-88b38e822f0d" 
      >, 
                  <Fog::Compute::Vsphere::Interface
        mac="00:50:56:b2:20:11",
        network="dvportgroup-87636",
        name="Network adapter 2",
        status="untried",
        summary="DVSwitch: 12 a9 12 50 6e 04 89 81-1c ec 7d 3a 09 1b 6b 0e",
        type=VirtualVmxnet3,
        key=4001,
        virtualswitch=nil,
        server_id="50328ecb-196d-a21d-b661-88b38e822f0d" 
      >    
    ]
  >
2015-07-14 08:09:58 [W] Orchestration::Compute: Could not match network interface #<Nic::Managed id: nil, mac: nil, ip: "10.21.3.245", type: "Nic::Managed", name: "corp-testforeman.hammocks.local", host_id: nil, subnet_id: 1, domain_id: 1, attrs: {}, created_at: nil, updated_at: nil, provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: "", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: false, compute_attributes: {"type"=>"VirtualVmxnet3", "network"=>"pg-vlan-3"}>
2015-07-14 08:09:58 [W] Could not find virtual machine network interface matching 10.21.3.245

Looks like it's struggling to match the requested network "pg-vlan-3" against either of the networks on the two created NICs, "dvportgroup-87629" and "dvportgroup-87636".

#5 Updated by Seth Doty over 2 years ago

To provide a bit of history on this, we had a semi-working install of foreman 1.7 running. We hit a lot of katello side bugs, but the build process worked ok most of the time. Something changed with the upgrade to 1.8 though, aside from the obvious, since this occurs with all builds I have tested now in different data centers with different networks/port groups, although it is the same vcenter instance.

I rebuilt this foreman/katello server since the old box was a bit of a Frankenstein install, so nothing is in this box yet if we need to poke around a bit more. This box is only a 3 or 4 days old at this point so its all VERY recent data.

As far as the interface match struggles, I have a co-worker who wrote some powershell scripts for the windows side and had similar issues querying interfaces due to the order he queried the vmware switching component. He couldn't directly query for the port group, but had to query for the distributed switch first and work his way down through port groups or he would never get an interface match either. I'm not sure if this is pertinent information at this point, but it is the process we had to go through before for the windows side of the world.

#6 Updated by Seth Doty about 2 years ago

As an FYI, I just upgraded to the latest release and this bug still persists.

#7 Updated by Przemysław Ciesielski about 2 years ago

The same problem with Foreman 1.9.2 and vCenter 6.0.

#8 Updated by Sandro Roth over 1 year ago

Just ran into this after vCenter upgrade from 5.5 to 6.0.
Tested with Foreman 1.11.2 and 1.12 RC1.

Attached log from 1.12 RC1 with debug enabled.

#9 Updated by salman butt over 1 year ago

I'm also hitting this bug with foreman 1.11.2, 1.12-RC1.

Provisioning exists with:

"2016-07-27 19:17:06 [app] [E] Unprocessable entity Host::Managed (id: new):

 |   Could not find virtual machine network interface matching eth0
 | 
2016-07-27 19:17:06 [app] [I]   Rendered api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout (1.1ms)
2016-07-27 19:17:06 [app] [D] Body: {
 |   "error": {"id":null,"errors":{"base":["Could not find virtual machine network interface matching eth0"]},"full_messages":["Could not find virtual machine network interface matching eth0"]}
 | }
 | 
"

This started after a recent upgrade to vcenter 6.0.

#10 Updated by salman butt over 1 year ago

Actually, looks like this works fine in 6.0 vcenter build number 3018524, but not with 6.0 vcenter build number 3634793. I'm not sure where in-between this might have broken. Up until last week we were using the 3018524 build, and provisioning was working fine.

#11 Updated by salman butt over 1 year ago

We have found a potential fix for this in our environment (YMMV). It looks like something internal to VCenter changed in our all of our VDS, during the upgrade. Due to this, provisioning against the existing VDS keeps failing. A coworker performed a separate test on a newly created, temporary VDS, and provisioning worked fine against the VLANs defined there.

We're going to do some work internally to create a brand new VDS layout here, move VMs around, and hopefully that will fix this problem.

#12 Updated by Bryan Kearney about 1 year ago

  • Bugzilla link set to 1394290

#13 Updated by The Foreman Bot 12 months ago

  • Status changed from New to Ready For Testing
  • Assigned To set to Ivan Necas
  • Pull request https://github.com/theforeman/foreman/pull/2714 added

#14 Updated by Ivan Necas 12 months ago

  • Assigned To deleted (Ivan Necas)

I think we've found the reason for this issue, the problem was we were matching `interface.network` value against network `id` in case of portgroup, while the `network.key` should be the thing to match against

#15 Updated by Ivan Necas 12 months ago

  • Assigned To set to Ivan Necas

#16 Updated by Ivan Necas 12 months ago

  • Target version set to Team Ivan Iteration 7

#17 Updated by The Foreman Bot 12 months ago

  • Pull request https://github.com/theforeman/foreman/pull/4074 added

#18 Updated by Ivan Necas 11 months ago

  • Pull request deleted (https://github.com/theforeman/foreman/pull/4074)

#19 Updated by Ivan Necas 11 months ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#20 Updated by Dominic Cleal 11 months ago

  • Release set to 1.14.1

#21 Updated by Oliver Neumann 26 days ago

Hi There,

I ran into the same Problem. Creating a VM in vCenter leads to the error message of this ticket. vCenter-Version is 6.5 an theforman is on Version 1.15.6.

The production.log give the following message:

017-10-24T15:24:44 d7225981 [app] [I] Could not find Vsphere network for {"network"=>"dvportgroup-xxxx", "type"=>"VirtualVmxnet3"}
2017-10-24T15:24:44 d7225981 [app] [I] Started GET "/tasks/ea1944c7-05ea-490d-a512-6cbdbbb1cdfc" for xxx.xxx.xxx.xxx at 2017-10-24 15:24:44 +0200
2017-10-24T15:24:44 d7225981 [app] [I] Processing by TasksController#show as */*

#22 Updated by Marek Hulán 22 days ago

Could you double check that the given network is in the cluster you've selected in Foreman? We don't limit resources according to cluster in Foreman UI which could result in such errors.

#23 Updated by Oliver Neumann 11 days ago

Double checked. The Network is available in the selected Cluster.

Also available in: Atom PDF