Since I'm not sure what the correct solution is (or even if a solution would be accepted into mainline), I've implemented this very ugly solution for the population with the x540 NICs. Perhaps it will be useful to someone else with the same problem that stumbles over this ticket.

class facter::macaddress {

  file { "/etc/profile.d/facter_macaddress.sh":
    owner   => 'root',
    group   => 'root',
    mode    => 0644,
    ensure  => present,
    replace => true,
    content => "export FACTER_macaddress=\"$::macaddress_eth2\"\n",
  }
}

I've just stumbeld upon this issue as well, in a slightly different situation, actually BEFORE even being able to start deployment:

All our VM hosts already are in foreman (puppet fact upload), now I have just added all compute resources that provide these VMs.

Within our company, eth0 is used for provisioning in 99% of cases - with VMs that'd be 100%.

Foreman though takes the "main identifying" MACaddress from puppets "macaddress" fact, which in turn seems to take the first lexically sorted match from the list of all other macaddress_$name facts - and in our case, most VMs have these macaddress facts:

macaddress_dummy0 = "something"      #virtual NOARP interface
macaddress_eth0 = "something_else"   #actual provisioning-NIC
macaddress = "something "            #from macaddress_dummy0, because "d" comes before "e" 
[other]

So foreman sets the hosts "main identifying" MAC to that of dummy0 (via "macaddress" fact) - an
d now there is a mismatch between what the compute resource exports (a host with the "main identifying" MAC address with the value from macaddress_eth0), and that "same" hosts discovered MAC (via normal facts), which leads to about half of our VMs not being able to be associated to their compute resources, becuase MAC addresses do not match.

Setting ignore_puppet_facts_for_provisioning does not help here - and we actually do not want that, we want to have puppet update foremans data, including IPs and MACs, so we f.e. can replace a broken eth0 NIC, reboot the machine and foreman automatically knows of the changed MAC behind that interface, and provisioning just keeps working.

The solution I would think best is for foreman to accept an optional, custom fact "foreman_provisioning_mac" whcih gets used in stead of the default "macaddress" fact, if it exists, so we do not have to so something nasty like Joshua Hoblitt suggested.

Then each host could just export the correct MAC to provision to, and we could have nice things like "Yes, all of YOU provision through eth0, but I provision through eth4, always" decided by the host.

To clarify: If I just where to go ahead and hack that default macaddress fact so that it always skips over dummy0 NICs or only ever exports macaddress_eth0, that'd be wrong too, since we do have machines with bonded NICs as the primary interface, or machines where, due to reasons, eth1 is used to provision in stead of eth0.

hth,
Simon