Project

General

Profile

Actions

Bug #11434

closed

Foreman 1.7.x -> 1.8.2 (Puppet collects/adds Lots of bogus interface entries)

Added by Will Foster over 8 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Network
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

We've noticed an issue with some existing hosts after upgrading from Foreman 1.7.2 -> 1.8.2
where there are sometimes dozens of new interfaces populated under interfaces, this causes updates to the host like changing hostgroup, toggling build to reprovision etc to fail.

Via hammer changing the hostgroup would render:

Started PUT "/api/hosts/92" for 127.0.0.1 at 2015-08-19 13:11:35 +0000
2015-08-19 13:11:35 [I] Processing by Api::V2::HostsController#update as JSON
2015-08-19 13:11:35 [I]   Parameters: {"host"=>{"name"=>"host29-rack06.scale.openstack.engineering.XXXXXX.com", "hostgroup_id"=>"183", "interfaces_attributes"=>{}, "host_parameters_attributes"=>{}, "compute_attributes"=>{"volumes_attributes"=>{}}}, "apiv"=>"v2", "id"=>"92"}
2015-08-19 13:11:35 [I] Authorized user admin(Admin User)
2015-08-19 13:11:36 [E] Unprocessable entity Host::Managed (id: 92):
  Interfaces some interfaces are invalid

Attempting to remove the non-primary interfaces listed, toggling to unmanaged and performing this or removing attributes from the primary interface (and deleting the generated non-primary interfaces) would yield:

NoMethodError
undefined method `changed?' for nil:NilClass
app/models/host/managed.rb:910:in `trigger_nic_orchestration'
app/models/concerns/orchestration.rb:48:in `valid?'
app/models/concerns/foreman/sti.rb:29:in `save_with_type'
app/controllers/hosts_controller.rb:100:in `block in update'
app/models/taxonomy.rb:54:in `block in no_taxonomy_scope'
app/models/taxonomy.rb:61:in `block (2 levels) in as_taxonomy'
app/models/concerns/foreman/thread_session.rb:147:in `as_location'
app/models/taxonomy.rb:60:in `block in as_taxonomy'
app/models/concerns/foreman/thread_session.rb:112:in `as_org'
app/models/taxonomy.rb:59:in `as_taxonomy'
app/models/taxonomy.rb:53:in `no_taxonomy_scope'
app/controllers/hosts_controller.rb:99:in `update'
app/controllers/concerns/application_shared.rb:13:in `set_timezone'
app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
lib/middleware/catch_json_parse_errors.rb:9:in `call'

We have found a workaround with manually removing the interfaces via hammer CLI
and skipping the primary/provision interface: this lets us further edit and manage the host.

problemhost=yourhostFQDN
skip_id=$(hammer host info --name $problemhost | grep -B 3 "nterface (primary, provision" | grep Id: | awk '{ print $NF }')
hammer host info --name $problemhost > /tmp/$problemhost
for interface in $(grep Id /tmp/$problemhost  | grep ')' | grep -v $skip_id | awk '{ print $NF }') ; do \
hammer host interface delete --host $problemhost --id $interface \
done

Files

foreman_issue_11434_1.png View foreman_issue_11434_1.png 98.2 KB Will Foster, 08/20/2015 03:07 AM
foreman_issue_11434_1.yaml foreman_issue_11434_1.yaml 7.94 KB YAML output of facter Will Foster, 08/20/2015 08:08 AM

Related issues 1 (0 open1 closed)

Related to Foreman - Feature #11972: Implement regex for option ignore_puppet_facts_for_provisioningClosedMarek Hulán09/26/2015Actions
Actions #1

Updated by Dominic Cleal over 8 years ago

  • Category set to Network
Actions #2

Updated by Marek Hulán over 8 years ago

Could you please attach the output of facter for this host? A screenshot of interfaces from webui would also help.

Actions #3

Updated by Will Foster over 8 years ago

Marek Hulán wrote:

Could you please attach the output of facter for this host? A screenshot of interfaces from webui would also help.

Hi Marek,

Here is facter output from that host and screenshot from the UI. Note this is another host that has this problem (I believe they all seem to in this set of hostgroups from 1.7.3 --> 1.8.2). The one filed in this report previously we used the above shell workaround to fix.

This is the state where they cannot be edited in the UI unless we manually remove the other non-primary, non-provisioning interfaces via hammer CLI above.

[root@host27-rack06 ~]# facter
Error: NetworkManager is not running.
architecture => x86_64
augeasversion => 1.1.0
bios_release_date => 03/07/2013
bios_vendor => Dell Inc.
bios_version => 1.6.0
blockdevice_sda_model => PERC H710P
blockdevice_sda_size => 999653638144
blockdevice_sda_vendor => DELL
blockdevice_sdb_model => PERC H710P
blockdevice_sdb_size => 999653638144
blockdevice_sdb_vendor => DELL
blockdevice_sdc_model => PERC H710P
blockdevice_sdc_size => 999653638144
blockdevice_sdc_vendor => DELL
blockdevice_sdd_model => PERC H710P
blockdevice_sdd_size => 999653638144
blockdevice_sdd_vendor => DELL
blockdevices => sda,sdb,sdc,sdd
boardmanufacturer => Dell Inc.
boardproductname => 0KCKR5
boardserialnumber => ..CN747512CJ0256.
domain => scale.openstack.engineering.XXXXX.com
facterversion => 2.4.1
filesystems => ext2,ext3,ext4,xfs
fqdn => host27-rack06.scale.openstack.engineering.XXXXXX.com
gid => root
hardwareisa => x86_64
hardwaremodel => x86_64
hostname => host27-rack06
id => root
interfaces => bond0,bond0_9,bond0_30,br1,em1,em2,em3,em4,lo,virbr0,virbr0_nic,vnet0,vnet1
ipaddress => 10.1.64.51
ipaddress_bond0 => 10.1.64.51
ipaddress_bond0_30 => 10.1.247.100
ipaddress_lo => 127.0.0.1
ipaddress_virbr0 => 192.168.122.1
is_virtual => false
kernel => Linux
kernelmajversion => 3.10
kernelrelease => 3.10.0-229.1.2.el7.x86_64
kernelversion => 3.10.0
macaddress => b8:ca:3a:63:89:38
macaddress_bond0 => b8:ca:3a:63:89:38
macaddress_bond0_30 => b8:ca:3a:63:89:38
macaddress_bond0_9 => b8:ca:3a:63:89:38
macaddress_br1 => 96:ec:0f:24:fa:d2
macaddress_em1 => B8:CA:3A:63:89:38
macaddress_em2 => B8:CA:3A:63:89:3A
macaddress_em3 => b8:ca:3a:63:89:3c
macaddress_em4 => b8:ca:3a:63:89:3d
macaddress_virbr0 => 52:54:00:bd:ec:b3
macaddress_virbr0_nic => 52:54:00:bd:ec:b3
macaddress_vnet0 => fe:54:00:80:56:4d
macaddress_vnet1 => fe:54:00:f1:c9:87
manufacturer => Dell Inc.
memoryfree => 53.62 GB
memoryfree_mb => 54905.31
memorysize => 62.72 GB
memorysize_mb => 64220.45
mtu_bond0 => 9000
mtu_bond0_30 => 9000
mtu_bond0_9 => 9000
mtu_br1 => 1500
mtu_em1 => 9000
mtu_em2 => 9000
mtu_em3 => 1500
mtu_em4 => 1500
mtu_lo => 65536
mtu_virbr0 => 1500
mtu_virbr0_nic => 1500
mtu_vnet0 => 1500
mtu_vnet1 => 1500
netmask => 255.255.192.0
netmask_bond0 => 255.255.192.0
netmask_bond0_30 => 255.255.255.224
netmask_lo => 255.0.0.0
netmask_virbr0 => 255.255.255.0
network_bond0 => 10.1.64.0
network_bond0_30 => 10.1.247.96
network_lo => 127.0.0.0
network_virbr0 => 192.168.122.0
operatingsystem => RedHat
operatingsystemmajrelease => 7
operatingsystemrelease => 7.1
os => {"name"=>"RedHat", "family"=>"RedHat", "release"=>{"major"=>"7", "minor"=>"1", "full"=>"7.1"}}
osfamily => RedHat
partitions => {"sda1"=>{"size"=>"134217728", "label"=>"1", "filesystem"=>"linux_raid_member"}, "sda2"=>{"size"=>"1048576", "label"=>"0", "filesystem"=>"linux_raid_member"}, "sda3"=>{"size"=>"1817180160", "label"=>"2", "filesystem"=>"linux_raid_member"}, "sdb1"=>{"size"=>"134217728", "label"=>"1", "filesystem"=>"linux_raid_member"}, "sdb2"=>{"size"=>"1048576", "label"=>"0", "filesystem"=>"linux_raid_member"}, "sdb3"=>{"size"=>"1817180160", "label"=>"2", "filesystem"=>"linux_raid_member"}, "sdc1"=>{"size"=>"1952446464", "label"=>"3", "filesystem"=>"linux_raid_member"}, "sdd1"=>{"size"=>"1952446464", "label"=>"3", "filesystem"=>"linux_raid_member"}}
path => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/sbin
physicalprocessorcount => 2
processor0 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor1 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor10 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor11 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor12 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor13 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor14 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor15 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor16 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor17 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor18 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor19 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor2 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor20 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor21 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor22 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor23 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor3 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor4 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor5 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor6 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor7 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor8 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processor9 => Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
processorcount => 24
processors => {"models"=>["Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz", "Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz"], "count"=>24, "physicalcount"=>2}
productname => PowerEdge R620
ps => ps -ef
puppetversion => 3.7.4
rubyplatform => x86_64-linux
rubysitedir => /usr/local/share/ruby/site_ruby/
rubyversion => 2.0.0
selinux => true
selinux_config_mode => permissive
selinux_config_policy => unknown
selinux_current_mode => permissive
selinux_enforced => false
selinux_policyversion => 28
serialnumber => 3HNNQW1
swapfree => 63.97 GB
swapfree_mb => 65503.87
swapsize => 63.97 GB
swapsize_mb => 65503.87
system_uptime => {"seconds"=>3942327, "hours"=>1095, "days"=>45, "uptime"=>"45 days"}
timezone => UTC
type => Rack Mount Chassis
uniqueid => 010a3340
uptime => 45 days
uptime_days => 45
uptime_hours => 1095
uptime_seconds => 3942327
uuid => 4C4C4544-0048-4E10-804E-B3C04F515731
virtual => physical
vlans => 30,9
Actions #4

Updated by Will Foster over 8 years ago

Will Foster wrote:

Marek Hulán wrote:

Could you please attach the output of facter for this host? A screenshot of interfaces from webui would also help.

Hi Marek,

Also attaching the YAML output of facter

Here is facter output from that host and screenshot from the UI. Note this is another host that has this problem (I believe they all seem to in this set of hostgroups from 1.7.3 --> 1.8.2). The one filed in this report previously we used the above shell workaround to fix.

This is the state where they cannot be edited in the UI unless we manually remove the other non-primary, non-provisioning interfaces via hammer CLI above.

[...]

Actions #5

Updated by Marek Hulán almost 8 years ago

  • Related to Feature #11972: Implement regex for option ignore_puppet_facts_for_provisioning added
Actions #6

Updated by Marek Hulán almost 8 years ago

  • Status changed from New to Assigned
  • Assignee set to Marek Hulán

There are two problems in this case. One is invalid eth0 vs bond0, second is that vdsmdummy interfaces being created. The first one I believe is because after you provision a host on eth0 you start a bond which becomes new primary interface but this was not updated in Foreman DB, that's why they have same IP which I believe is the cause for eth0 being invalid. Second issue is that vdsmdummy are temporary interfaces created by oVirt and they should not be imported in the first place. Since Foreman 1.10 we have setting for selectively ignoring interfaces based on their identifier so to fix this we can simply add vdsmdummy to the list.

Actions #7

Updated by Marek Hulán almost 8 years ago

  • Status changed from Assigned to Closed
  • translation missing: en.field_release set to 136
  • Pull request https://github.com/theforeman/foreman/pull/3429 added
Actions

Also available in: Atom PDF