I'm pretty certain using multiple matchers isn't possible either way - what's happening when you use hostgroup= first is that it's able to verify this against known host groups. When using osfamily= first, it's matching on a fact and so can't verify possible values.

The Foreman Manual for release 1.4 under the section "Example 2 - Change for a group of hosts (via custom fact) with validation and ordering" shows that matching on multiple values is possible. The manual states this is for smart variables, but it also works for smart class parameters.

We have multiple instances running in our environment where we are successfully matching on multiple values and applying the correct overrides. For example, we currently have sitecfg_purpose= ,hostgroup= as match criteria for a smart class parameter (sitecfg_purpose being a custom fact). The server meeting the credentials will successfully override the value. When we had the order of the two reversed to hostgroup= , sitecfg_purpose= Foreman would throw the error "hostgroup=(hostgroup),sitecfg_purpose=(purpose) does not match an existing hostgroup".