When running rake test with a syntax error in an app controller (e.g. caused by #7495), the syntax error is logged by the rake process during startup but minitest execution continues anyway:

+ bundle exec rake pkg:generate_source jenkins:unit TESTOPTS=-v
rm -rf jenkins/reports/unit/
The Apipie cache is turned off. Enable it and run apipie:cache rake task to speed up API calls.
WARNING: `rake webpack:compile` failed to run. This is only important if running integration tests. (cause: Can't find our webpack executable at /var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/node_modules/.bin/webpack - have you run `npm install`?)
The Apipie cache is turned off. Enable it and run apipie:cache rake task to speed up API calls.
rake aborted!
SyntaxError: /var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/app/controllers/api/v2/autosign_controller.rb:44: syntax error, unexpected ':', expecting =>
        render :status => 500, :json => { "error": exception.message }
                                                  ^
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/app/controllers/api/v2/autosign_controller.rb:44: syntax error, unexpected '}', expecting keyword_end
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:360:in `require_or_load'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:494:in `load_missing_constant'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:184:in `const_missing'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:276:in `const_get'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:276:in `block in constantize'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `inject'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/inflector/methods.rb:259:in `constantize'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/test/unit/shared/access_permissions_test_base.rb:21:in `block in check_routes'
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/test/unit/shared/access_permissions_test_base.rb:15:in `each'
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/test/unit/shared/access_permissions_test_base.rb:15:in `check_routes'
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/test/unit/foreman/access_permissions_test.rb:48:in `<class:AccessPermissionsTest>'
/var/lib/workspace/workspace/test_develop/database/mysql/ruby/2.0.0/slave/fast/test/unit/foreman/access_permissions_test.rb:12:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/railties-4.2.7.1/lib/rails/test_unit/sub_test_task.rb:114:in `block (3 levels) in define'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/railties-4.2.7.1/lib/rails/test_unit/sub_test_task.rb:114:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/railties-4.2.7.1/lib/rails/test_unit/sub_test_task.rb:114:in `block (2 levels) in define'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/railties-4.2.7.1/lib/rails/test_unit/sub_test_task.rb:113:in `each'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/railties-4.2.7.1/lib/rails/test_unit/sub_test_task.rb:113:in `block in define'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.0.0-p481@test_develop-0/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => jenkins:unit => test:units
(See full trace by running task with --trace)
Run options: -v --ci-reporter --seed 13083

# Running:

ApplicationHelperTest#test_generate_link_for = 2.29 s = .
ApplicationHelperTest::documentation#test_connection_compute_resources_url = 0.00 s = .

(ditto with plain rake test)

This is caused by AccessPermissionsTest which constantizes and causes loading of controllers etc outside of the test process. Normally syntax errors would be picked up during test execution for that controller:

  1) Error:
Api::V2::AutosignControllerTest#test_0003_should not create autosign entry:
SyntaxError: /home/dcleal/code/foreman/foreman/app/controllers/api/v2/autosign_controller.rb:44: syntax error, unexpected ':', expecting =>
        render :status => 500, :json => { "error": exception.message }
                                                  ^
/home/dcleal/code/foreman/foreman/app/controllers/api/v2/autosign_controller.rb:44: syntax error, unexpected '}', expecting keyword_end

(if running the controller test specifically)

Moving the constantization inside the test should make it act like other tests.