Project

General

Profile

Actions

Bug #14155

closed

Tests are order dependent, should be runnable with random order

Added by Dominic Cleal about 8 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Tests
Target version:
Fixed in Releases:
Found in Releases:

Description

When enabling activesupport/minitest's random test ordering on Rails 4.2, some failures are seen:

Some when executing Api::V2 OS controller tests in this order:

10:10:13 Api::V2::OperatingsystemsControllerTest#test_0005_should not create os without version = 0.11 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0014_should show os if id is description = 0.14 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0013_should show os if id is fullname = 0.10 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0009_should update associated architectures by name with UNWRAPPED node = 0.16 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0004_should create os with os parameters = 0.01 s = E
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0011_should add association of architectures by name with WRAPPED node = 0.17 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0006_should update os = 0.14 s = .
10:10:13 Api::V2::OperatingsystemsControllerTest#test_0001_should get index = 0.06 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0010_should add association of architectures by ids with WRAPPED node = 0.51 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0003_should create os = 0.06 s = F
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0008_should update associated architectures by ids with UNWRAPPED node = 0.16 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0002_should show os = 0.13 s = .
10:10:14 Api::V2::OperatingsystemsControllerTest#test_0007_should destroy os = 0.08 s = .
10:10:15 Api::V2::OperatingsystemsControllerTest#test_0012_should remove association of architectures with WRAPPED node = 0.16 s = .

This functional test has a class-level os variable which is being reassigned with OS resources.

And also seed tests in this order:

SeedsTest::populating an initial admin user#test_0002_with environment overrides = 6.05 s = .
SeedsTest::populating an initial admin user#test_0001_with defaults = 5.61 s = F

This unit test uses the with_env helper, which doesn't unset previously unassigned environment variables on completion.


Files

consoleText1.txt consoleText1.txt 789 KB index test failures Dominic Cleal, 03/11/2016 09:45 AM

Related issues 1 (0 open1 closed)

Related to Foreman - Feature #13244: Upgrade Ruby on Rails to 4.2ClosedDominic Cleal01/15/2016Actions
Actions #1

Updated by Dominic Cleal about 8 years ago

Actions #2

Updated by Dominic Cleal about 8 years ago

Sporadic failures on some API index tests remain:

  • Api::V2::OsDefaultTemplatesControllerTest
  • Api::V2::OverrideValuesControllerTest
  • Api::V2::ParametersControllerTest
  • Api::V2::PuppetclassesControllerTest
  • Api::V2::ReportsControllerTest
  • Api::V2::RolesControllerTest
  • Api::V2::SmartClassParametersControllerTest
  • Api::V2::SmartProxiesControllerTest
  • Api::V2::SmartVariablesControllerTest
  • Api::V2::SubnetsControllerTest
  • Api::V2::TasksControllerTest
  • Api::V2::TemplateCombinationsControllerTest
Actions #3

Updated by Dominic Cleal about 8 years ago

Index tests from OsDefaultTemplatesControllerTest onwards fail when the last LocationsControllerTest is:

Api::V2::LocationsControllerTest#test_0014_root name on index is configured to be controller name = 0.05 s = .

This test reconfigures Rabl.configuration.use_controller_name_as_json_root but the value is reset to false in the setup method rather than a teardown, so if there's no test following it, then it never gets reset. Further test cases fail as they get index JSON root nodes named after their controllers, e.g.

{"total"=>6, "subtotal"=>6, "page"=>1, "per_page"=>20, "search"=>nil, "sort"=>{"by"=>nil, "order"=>nil}, "os_default_templates"=>[{"id"=>51848956,  ... 
Actions #4

Updated by The Foreman Bot about 8 years ago

  • Status changed from Assigned to Ready For Testing
  • Pull request https://github.com/theforeman/foreman/pull/3321 added
Actions #5

Updated by Dominic Cleal about 8 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #6

Updated by Dominic Cleal about 8 years ago

  • translation missing: en.field_release set to 136
Actions #7

Updated by The Foreman Bot about 8 years ago

  • Pull request https://github.com/theforeman/foreman/pull/3324 added
Actions

Also available in: Atom PDF