When running the full test suite with rake test, the following test usually fails:

  1) Failure:
AuthSourceLdapTest::refresh ldap#test_0003_update_usergroups refreshes on all external user groups, in LDAP and in Foreman auth source [/home/dcleal/code/foreman/foreman/app/models/external_usergroup.rb:40]:
unexpected invocation: #<AuthSourceLdap:0xd10c810>.valid_group?('external_usergroup2')
unsatisfied expectations:
- expected exactly once, not yet invoked: #<AnyInstance:LdapFluff>.group_list('test')
- expected exactly once, not yet invoked: #<AuthSourceLdap:0xd10c810>.valid_group?('external_usergroup1')
satisfied expectations:
- allowed any number of times, not yet invoked: #<AnyInstance:LdapFluff>.find_user(any_parameters)
- allowed any number of times, not yet invoked: #<AnyInstance:LdapFluff>.valid_user?(any_parameters)

The test has a hardcoded ExternalUsergroup name in it, but because it's created via FactoryGirl, the name will increment and change.

Easy to reproduce by running test/controllers/usergroups_controller_test.rb test/models/auth_sources/auth_source_ldap_test.rb in sequence.

Likely not an issue with jenkins:* rake tasks as they run in a particular strict order. Only some controller tests create external user groups, this only happens if they run before the model tests.