Project

General

Profile

Bug #14155

Tests are order dependent, should be runnable with random order

Added by Dominic Cleal about 6 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Tests
Target version:
Difficulty:
Triaged:
Bugzilla link:

Description

When enabling activesupport/minitest's random test ordering on Rails 4.2, some failures are seen:

Some when executing Api::V2 OS controller tests in this order:

10:10:13 Api::V2::OperatingsystemsControllerTest#test_0005_should not create os without version = 0.11 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0014_should show os if id is description = 0.14 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0013_should show os if id is fullname = 0.10 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0009_should update associated architectures by name with UNWRAPPED node = 0.16 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0004_should create os with os parameters = 0.01 s = E
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0011_should add association of architectures by name with WRAPPED node = 0.17 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0006_should update os = 0.14 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0001_should get index = 0.06 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0010_should add association of architectures by ids with WRAPPED node = 0.51 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0003_should create os = 0.06 s = F
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0008_should update associated architectures by ids with UNWRAPPED node = 0.16 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0002_should show os = 0.13 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0007_should destroy os = 0.08 s = .
10:10:15 Api::V2::OperatingsystemsControllerTest#test_0012_should remove association of architectures with WRAPPED node = 0.16 s = .

This functional test has a class-level os variable which is being reassigned with OS resources.

And also seed tests in this order:

SeedsTest::populating an initial admin user#test_0002_with environment overrides = 6.05 s = .
SeedsTest::populating an initial admin user#test_0001_with defaults = 5.61 s = F

This unit test uses the with_env helper, which doesn't unset previously unassigned environment variables on completion.

consoleText1.txt consoleText1.txt 789 KB index test failures Dominic Cleal, 03/11/2016 09:45 AM

Related issues

Related to Foreman - Feature #13244: Upgrade Ruby on Rails to 4.2Closed2016-01-15

Associated revisions

Revision 945dd6d7 (diff)
Added by Dominic Cleal about 6 years ago

fixes #14155 - fix leaks between tests with random ordering

- A test for the Rabl use_controller_name_as_json_root extension was
leaking as the old value was only reset in `setup` methods. When the
test was last in the test case, the configuration wasn't reset.
- The with_env helper failed to unset environment variables that weren't
set before, it now stores a nil value and resets them.
- OS functional tests used a class-level `os` variable to hold default
parameters, but was reassigned in some tests.

History

#1 Updated by Dominic Cleal about 6 years ago

#2 Updated by Dominic Cleal about 6 years ago

Sporadic failures on some API index tests remain:

  • Api::V2::OsDefaultTemplatesControllerTest
  • Api::V2::OverrideValuesControllerTest
  • Api::V2::ParametersControllerTest
  • Api::V2::PuppetclassesControllerTest
  • Api::V2::ReportsControllerTest
  • Api::V2::RolesControllerTest
  • Api::V2::SmartClassParametersControllerTest
  • Api::V2::SmartProxiesControllerTest
  • Api::V2::SmartVariablesControllerTest
  • Api::V2::SubnetsControllerTest
  • Api::V2::TasksControllerTest
  • Api::V2::TemplateCombinationsControllerTest

#3 Updated by Dominic Cleal about 6 years ago

Index tests from OsDefaultTemplatesControllerTest onwards fail when the last LocationsControllerTest is:

Api::V2::LocationsControllerTest#test_0014_root name on index is configured to be controller name = 0.05 s = .

This test reconfigures Rabl.configuration.use_controller_name_as_json_root but the value is reset to false in the setup method rather than a teardown, so if there's no test following it, then it never gets reset. Further test cases fail as they get index JSON root nodes named after their controllers, e.g.

{"total"=>6, "subtotal"=>6, "page"=>1, "per_page"=>20, "search"=>nil, "sort"=>{"by"=>nil, "order"=>nil}, "os_default_templates"=>[{"id"=>51848956,  ... 

#4 Updated by The Foreman Bot about 6 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3321 added

#5 Updated by Dominic Cleal about 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100

#6 Updated by Dominic Cleal about 6 years ago

  • Legacy Backlogs Release (now unused) set to 136

#7 Updated by The Foreman Bot about 6 years ago

  • Pull request https://github.com/theforeman/foreman/pull/3324 added

Also available in: Atom PDF