Project

General

Profile

Bug #12486

Duplicate IP addresses after upgrading from 1.7 -> 1.8 -> 1.9

Added by Imri Zvik over 5 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Network
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

After upgrading to 1.9 (from 1.7, passing through 1.8), we suddenly got duplicated IP addresses in managed hosts.
For example, from 1.7:

mysql> select * from nics where ip='10.0.30.219';
+-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+
| id    | mac               | ip           | type      | name | host_id | subnet_id | domain_id | attrs                                                                                                           | created_at          | updated_at          | provider | username | password | virtual | link | identifier | tag | attached_to | managed | mode       | attached_devices | bond_options |
+-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+
| 24293 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Bond | NULL |    9633 |      NULL |      NULL | --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess
network: 10.0.30.0
mtu: '1500'
netmask: 255.255.254.0
 | 2015-07-27 12:30:27 | 2015-11-11 07:32:55 | NULL     | NULL     | NULL     |       1 |    1 | bond0      |     |             |       0 | balance-rr |                  |              |
+-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+
1 row in set (0.01 sec)

After the upgrade:

mysql> select * from nics where ip='10.0.30.219';

+-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+
| id    | mac               | ip           | type         | name                   | host_id | subnet_id | domain_id | attrs                                                                                                           | created_at          | updated_at          | provider | username | password | virtual | link | identifier | tag | attached_to | managed | mode       | attached_devices | bond_options | primary | provision | compute_attributes |
+-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+
| 24293 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Bond    | NULL                   |    9633 |      NULL |      NULL | --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess
network: 10.0.30.0
mtu: '1500'
netmask: 255.255.254.0
 | 2015-07-27 12:30:27 | 2015-11-03 08:02:46 | NULL     | NULL     | NULL     |       1 |    1 | bond0      |     |             |       0 | balance-rr |                  |              |       0 |         0 | NULL               |
| 31234 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Managed | data120.example.domain |    9633 |        11 |        11 | NULL                                                                                                            | 2015-11-03 11:54:37 | 2015-11-03 11:54:37 | NULL     | NULL     | NULL     |       0 |    1 | eth0       |     |             |       1 | balance-rr |                  |              |       1 |         1 | NULL               |
+-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+
2 rows in set (0.02 sec)

mysql> 

This of course fails the uniq validation ("Ip has been, thus preventing the removal of the duplicate NIC.

This seems a bit like the issue mentioned here:
https://groups.google.com/forum/#!topic/foreman-users/7i3xCfgb8Xk

ignore_puppet_facts_for_provisioning is set to "true".


Related issues

Related to Foreman - Bug #11034: network/interfaces validation checks the removed interfaces Closed2015-07-07

History

#1 Updated by Imri Zvik over 5 years ago

In addition to finding why this happened in the migration process, maybe it is also a good idea to exclude deleted interfaces from the uniqueness check?
e.g.


      if Nic::Interface.where("ip = BINARY ? AND id NOT IN (?)",self.ip,host.interfaces.find_all {|s| s._destroy == true or s.id == self.id }.map { |s| s.id}).size > 0
        errors.add(:ip," is already taken")
      end

#2 Updated by Dominic Cleal over 5 years ago

  • Category set to Network

It would be useful if you could narrow this down a bit by identifying whether it's a particular DB migration generating this, or if it's coming in from fact imports etc.

Note that #12391 prevents the ignore_puppet_facts_for_provisioning setting from working if you have Discovery installed.

#3 Updated by Imri Zvik over 5 years ago

Dominic Cleal wrote:

It would be useful if you could narrow this down a bit by identifying whether it's a particular DB migration generating this, or if it's coming in from fact imports etc.

We are working on re-doing the upgrade, and will try to pinpoint it.

Note that #12391 prevents the ignore_puppet_facts_for_provisioning setting from working if you have Discovery installed.

Interesting, and we do use the discovery plugin. I think the main problem is that we got duplicated IPs, more than the fact that the interfaces got imported.

#4 Updated by Evgeny Inberg over 5 years ago

While trying to narrow down the exact operation, my conclusion that this is happening during "MoveHostNicsToInterfaces".
I was able to reproduce this issue again, and due to an error regarding removal of a foreign key "hosts_subnet_id_fk", the migrate process stopped and I was able to detect that the new interface was infect created during one of the previous steps.

#5 Updated by Evgeny Inberg over 5 years ago

Evgeny Inberg wrote:

While trying to narrow down the exact operation, my conclusion that this is happening during "MoveHostNicsToInterfaces".
I was able to reproduce this issue again, and due to an error regarding removal of a foreign key "hosts_subnet_id_fk", the migrate process stopped and I was able to detect that the new interface was infect created during one of the previous steps.

To be more precise, I believe that this is happening during "Migrating Host interfaces to standalone Interfaces".

#6 Updated by Evgeny Inberg over 5 years ago

Looks like this this is the code that does this:

Host::Managed.all.each do |host|
      nic = FakeNic.new
      nic.host_id = host.id
      nic.name = host.name
      nic.mac = host.attributes.with_indifferent_access[:mac]
      nic.ip = host.attributes.with_indifferent_access[:ip]
      nic.subnet_id = host.attributes.with_indifferent_access[:subnet_id]
      nic.domain_id = host.attributes.with_indifferent_access[:domain_id]
      nic.virtual = false
      nic.identifier = host.primary_interface || "eth0" 
      nic.managed = host.attributes.with_indifferent_access[:managed]
      nic.primary = true
      nic.provision = true
      nic.type = 'Nic::Managed'
      nic.save!

      say "  Migrated #{nic.name}-#{nic.identifier} to nics" 
    end

#7 Updated by Imri Zvik over 5 years ago

It seems like the workaround which allows you to at least edit and remove conflicting NICs as been added in
https://github.com/theforeman/foreman/commit/35241dd65a6b4f426374a03e83d6f796f59a9d35

This should most definitely be backported to 1.9

#8 Updated by Anonymous over 5 years ago

This occurs for me after upgrading to 1.9.3 on a host where there are bond interfaces. Say the ip is initially 10.1.2.3 and during the upgrade migration it gets set to eth0, and then puppet runs and it wants to set bond0 to 10.1.2.3, but it can't because it's already on eth0.

Is there any good way around this currently? Will the above patch work with 1.9?

#9 Updated by The Foreman Bot over 5 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Imri Zvik
  • Pull request https://github.com/theforeman/foreman/pull/2931 added

#10 Updated by Imri Zvik over 5 years ago

Alyssa H wrote:

This occurs for me after upgrading to 1.9.3 on a host where there are bond interfaces. Say the ip is initially 10.1.2.3 and during the upgrade migration it gets set to eth0, and then puppet runs and it wants to set bond0 to 10.1.2.3, but it can't because it's already on eth0.

Is there any good way around this currently? Will the above patch work with 1.9?

Yes, it should work. I cherry picked this fix to the 1.9 branch, and opened a PR. According to Dominic, there is no point release for 1.9 planned, but maybe they will change their minds :)

It should be noted that this doesn't "fix" the problem, but rather allows you to go workaround it (by allowing you to manually remove the offending duplicate IP).

As for a complete fix, we can maybe think about not importing the IP if it already exists in the NIC table.

#11 Updated by Dominic Cleal over 5 years ago

  • Related to Bug #11034: network/interfaces validation checks the removed interfaces added

#12 Updated by Dominic Cleal over 5 years ago

  • Status changed from Ready For Testing to New
  • Assignee deleted (Imri Zvik)
  • Pull request deleted (https://github.com/theforeman/foreman/pull/2931)

The PR to backport #11034 is closed, but I'll leave this ticket open as it's applicable to the original bug in the DB migration or similar.

#13 Updated by Anonymous about 4 years ago

  • Status changed from New to Duplicate

this won't get fixed, sorry...

#14 Updated by Anonymous about 4 years ago

  • Status changed from Duplicate to Resolved

Also available in: Atom PDF