Bug #12486
closed
Duplicate IP addresses after upgrading from 1.7 -> 1.8 -> 1.9
Description
After upgrading to 1.9 (from 1.7, passing through 1.8), we suddenly got duplicated IP addresses in managed hosts.
For example, from 1.7:
mysql> select * from nics where ip='10.0.30.219'; +-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+ | id | mac | ip | type | name | host_id | subnet_id | domain_id | attrs | created_at | updated_at | provider | username | password | virtual | link | identifier | tag | attached_to | managed | mode | attached_devices | bond_options | +-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+ | 24293 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Bond | NULL | 9633 | NULL | NULL | --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess network: 10.0.30.0 mtu: '1500' netmask: 255.255.254.0 | 2015-07-27 12:30:27 | 2015-11-11 07:32:55 | NULL | NULL | NULL | 1 | 1 | bond0 | | | 0 | balance-rr | | | +-------+-------------------+--------------+-----------+------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+ 1 row in set (0.01 sec)
After the upgrade:
mysql> select * from nics where ip='10.0.30.219'; +-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+ | id | mac | ip | type | name | host_id | subnet_id | domain_id | attrs | created_at | updated_at | provider | username | password | virtual | link | identifier | tag | attached_to | managed | mode | attached_devices | bond_options | primary | provision | compute_attributes | +-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+ | 24293 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Bond | NULL | 9633 | NULL | NULL | --- !ruby/hash:ActiveSupport::HashWithIndifferentAccess network: 10.0.30.0 mtu: '1500' netmask: 255.255.254.0 | 2015-07-27 12:30:27 | 2015-11-03 08:02:46 | NULL | NULL | NULL | 1 | 1 | bond0 | | | 0 | balance-rr | | | 0 | 0 | NULL | | 31234 | 44:a8:42:14:79:71 | 10.0.30.219 | Nic::Managed | data120.example.domain | 9633 | 11 | 11 | NULL | 2015-11-03 11:54:37 | 2015-11-03 11:54:37 | NULL | NULL | NULL | 0 | 1 | eth0 | | | 1 | balance-rr | | | 1 | 1 | NULL | +-------+-------------------+--------------+--------------+------------------------+---------+-----------+-----------+-----------------------------------------------------------------------------------------------------------------+---------------------+---------------------+----------+----------+----------+---------+------+------------+-----+-------------+---------+------------+------------------+--------------+---------+-----------+--------------------+ 2 rows in set (0.02 sec) mysql>
This of course fails the uniq validation ("Ip has been, thus preventing the removal of the duplicate NIC.
This seems a bit like the issue mentioned here:
https://groups.google.com/forum/#!topic/foreman-users/7i3xCfgb8Xk
ignore_puppet_facts_for_provisioning is set to "true".
Updated by Imri Zvik over 8 years ago
In addition to finding why this happened in the migration process, maybe it is also a good idea to exclude deleted interfaces from the uniqueness check?
e.g.
if Nic::Interface.where("ip = BINARY ? AND id NOT IN (?)",self.ip,host.interfaces.find_all {|s| s._destroy == true or s.id == self.id }.map { |s| s.id}).size > 0
errors.add(:ip," is already taken")
end
Updated by Dominic Cleal over 8 years ago
- Category set to Network
It would be useful if you could narrow this down a bit by identifying whether it's a particular DB migration generating this, or if it's coming in from fact imports etc.
Note that #12391 prevents the ignore_puppet_facts_for_provisioning setting from working if you have Discovery installed.
Updated by Imri Zvik over 8 years ago
Dominic Cleal wrote:
It would be useful if you could narrow this down a bit by identifying whether it's a particular DB migration generating this, or if it's coming in from fact imports etc.
We are working on re-doing the upgrade, and will try to pinpoint it.
Note that #12391 prevents the ignore_puppet_facts_for_provisioning setting from working if you have Discovery installed.
Interesting, and we do use the discovery plugin. I think the main problem is that we got duplicated IPs, more than the fact that the interfaces got imported.
Updated by Evgeny Inberg over 8 years ago
While trying to narrow down the exact operation, my conclusion that this is happening during "MoveHostNicsToInterfaces".
I was able to reproduce this issue again, and due to an error regarding removal of a foreign key "hosts_subnet_id_fk", the migrate process stopped and I was able to detect that the new interface was infect created during one of the previous steps.
Updated by Evgeny Inberg over 8 years ago
Evgeny Inberg wrote:
While trying to narrow down the exact operation, my conclusion that this is happening during "MoveHostNicsToInterfaces".
I was able to reproduce this issue again, and due to an error regarding removal of a foreign key "hosts_subnet_id_fk", the migrate process stopped and I was able to detect that the new interface was infect created during one of the previous steps.
To be more precise, I believe that this is happening during "Migrating Host interfaces to standalone Interfaces".
Updated by Evgeny Inberg over 8 years ago
Looks like this this is the code that does this:
Host::Managed.all.each do |host|
nic = FakeNic.new
nic.host_id = host.id
nic.name = host.name
nic.mac = host.attributes.with_indifferent_access[:mac]
nic.ip = host.attributes.with_indifferent_access[:ip]
nic.subnet_id = host.attributes.with_indifferent_access[:subnet_id]
nic.domain_id = host.attributes.with_indifferent_access[:domain_id]
nic.virtual = false
nic.identifier = host.primary_interface || "eth0"
nic.managed = host.attributes.with_indifferent_access[:managed]
nic.primary = true
nic.provision = true
nic.type = 'Nic::Managed'
nic.save!
say " Migrated #{nic.name}-#{nic.identifier} to nics"
end
Updated by Imri Zvik over 8 years ago
It seems like the workaround which allows you to at least edit and remove conflicting NICs as been added in
https://github.com/theforeman/foreman/commit/35241dd65a6b4f426374a03e83d6f796f59a9d35
This should most definitely be backported to 1.9
Updated by Anonymous over 8 years ago
This occurs for me after upgrading to 1.9.3 on a host where there are bond interfaces. Say the ip is initially 10.1.2.3 and during the upgrade migration it gets set to eth0, and then puppet runs and it wants to set bond0 to 10.1.2.3, but it can't because it's already on eth0.
Is there any good way around this currently? Will the above patch work with 1.9?
Updated by The Foreman Bot over 8 years ago
- Status changed from New to Ready For Testing
- Assignee set to Imri Zvik
- Pull request https://github.com/theforeman/foreman/pull/2931 added
Updated by Imri Zvik over 8 years ago
Alyssa H wrote:
This occurs for me after upgrading to 1.9.3 on a host where there are bond interfaces. Say the ip is initially 10.1.2.3 and during the upgrade migration it gets set to eth0, and then puppet runs and it wants to set bond0 to 10.1.2.3, but it can't because it's already on eth0.
Is there any good way around this currently? Will the above patch work with 1.9?
Yes, it should work. I cherry picked this fix to the 1.9 branch, and opened a PR. According to Dominic, there is no point release for 1.9 planned, but maybe they will change their minds :)
It should be noted that this doesn't "fix" the problem, but rather allows you to go workaround it (by allowing you to manually remove the offending duplicate IP).
As for a complete fix, we can maybe think about not importing the IP if it already exists in the NIC table.
Updated by Dominic Cleal over 8 years ago
- Related to Bug #11034: network/interfaces validation checks the removed interfaces added
Updated by Dominic Cleal over 8 years ago
- Status changed from Ready For Testing to New
- Assignee deleted (
Imri Zvik) - Pull request deleted (
https://github.com/theforeman/foreman/pull/2931)
The PR to backport #11034 is closed, but I'll leave this ticket open as it's applicable to the original bug in the DB migration or similar.
Updated by Anonymous almost 7 years ago
- Status changed from New to Duplicate
this won't get fixed, sorry...
Updated by Anonymous almost 7 years ago
- Status changed from Duplicate to Resolved