Project

General

Profile

Bug #13293

Intermittent host#edit parameter integration test failures after timeout

Added by Dominic Cleal over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Category:
Tests
Target version:
-
Difficulty:
Triaged:
No
Bugzilla link:
Pull request:
Fixed in Releases:
Found in Releases:

Description

Further to #13111, some host#edit parameter integration tests are failing intermittently but after the Capybara timeout (#13111 fixed instances of the timeout not being used). These can be seen from the verbose test logs when it takes over 30 seconds.

1) Assertion failure when removing an overridden global parameter and checking the override button is added back.

HostIntegrationTest::edit page.test_0004_correctly override global params
test/integration/host_test.rb:461

Fails on the third line:

refute page.has_selector?('#inherited_parameters .btn[data-tag=override]')
page.find('#global_parameters_table a[data-original-title="Remove Parameter"]').click
assert page.has_selector?('#inherited_parameters .btn[data-tag=override]')

2) Assertion failure when overriding a global parameter and checking the override button is removed.

Failed assertion, no message given. (Minitest::Assertion)
/var/lib/workspace/workspace/test_develop_pr_core/database/postgresql/label/fast/ruby/1.9.3/test/integration/host_test.rb:453

Third line:

assert page.has_selector?('#inherited_parameters .btn[data-tag=override]')
page.find('#inherited_parameters .btn[data-tag=override]').click
assert page.has_no_selector?('#inherited_parameters .btn[data-tag=override]')

3) Assertion failure when overriding a smart class parameter and checking the text area is enabled.

HostIntegrationTest::edit page.test_0001_class parameters and overrides are displayed correctly for booleans (from HostIntegrationTest__edit page) 
Capybara::ElementNotFound: Unable to find css "textarea:enabled" 
    test/integration/host_test.rb:392:in `block (2 levels) in <class:HostIntegrationTest>' (Capybara::ElementNotFound)

Fourth line:

assert_equal class_params.find("textarea").value, "true" 
assert class_params.find("textarea:disabled")
class_params.find("a[data-tag='override']").click
assert class_params.find("textarea:enabled")
screenshot_2016-01-18-15-15-15.704.html screenshot_2016-01-18-15-15-15.704.html 81.4 KB HTML from capybara-screenshot, for issue (1) Dominic Cleal, 01/21/2016 03:08 AM
screenshot_2016-01-18-15-15-15.704.png View screenshot_2016-01-18-15-15-15.704.png 42.2 KB screenshot from capybara-screenshot, for issue (1) Dominic Cleal, 01/21/2016 03:08 AM
Screenshot 2016 01 18 15 15 15.704

Related issues

Related to Foreman - Bug #13111: Intermittent host#edit parameter integration test failuresClosed2016-01-11
Related to Foreman - Bug #14034: Intermediate failures on Jenkins for puppetclass_lookup_key integration testResolved2016-03-03
Related to Foreman - Bug #14067: Host#edit integration tests should not wait for tasks or #show graphsClosed2016-03-07
Related to Foreman - Bug #14340: Icons loading causes intermittent issues in integration tests (especially under heavier load of the test machine)Closed2016-03-24

History

#1 Updated by Dominic Cleal over 6 years ago

  • Related to Bug #13111: Intermittent host#edit parameter integration test failures added

#2 Updated by Daniel Lobato Garcia over 6 years ago

Mind to post the screenshots saved in jenkins? I gave #3 a try before but I can't reproduce it locally at all with chromedriver-helper (nor poltergeist). It always removed the override right away, I don't know why would it take 30s+ on Jenkins. Are there any memory stats for jobs in Jenkins?

#3 Updated by Daniel Lobato Garcia over 6 years ago

Number 3 is synchronous Javascript (override_class_param), so I don't think that one failure caused by concurrency. Aside from that, Caoybara docs say that .find waits for Javascript :/

#4 Updated by Dominic Cleal over 6 years ago

I think the attached screenshot/HTML are for issue (1), but I've realised that I didn't keep the associated log output.

Not sure that we have any memory stats, but it's likely likely that some swapping is happening when running multiple integration tests. It shouldn't be lasting for 30 seconds though, and I'd expect to see more of an effect on adjacent tests.

#5 Updated by Daniel Lobato Garcia over 6 years ago

Notice new intermittent errors are showing up:

HostIntegrationTest::create new host page.test_0001_tabs are present
HostIntegrationTest::edit page.test_0007_choosing a hostgroup does not override other host attributes

http://ci.theforeman.org/job/test_develop_pr_core/8119/

#6 Updated by The Foreman Bot over 6 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Daniel Lobato Garcia
  • Pull request https://github.com/theforeman/foreman/pull/3199 added

#7 Updated by Dominic Cleal over 6 years ago

  • Related to Bug #14034: Intermediate failures on Jenkins for puppetclass_lookup_key integration test added

#8 Updated by Daniel Lobato Garcia over 6 years ago

Found a reproducer - slow down the server thread by half the Capybara.default_max_time by changing ApplicationContrller to include

after_filter -> { sleep(15) }

If you set the default_max_time to 10, and slow down the requests by 5 seconds it's also easy to reproduce.

#9 Updated by Daniel Lobato Garcia over 6 years ago

Some interesting findings:

When I slow down requests on ApplicationController with after_filter -> { sleep 5 } , Capybara.default_max_wait_time = 10, I get interesting results for 'correctly override global params':

click_button('Submit')                                                                                                                 
assert page.has_link?("Edit")

Fails , unexpectedly, as has_link? should work for 10 seconds, but it doesn't - the request returns in 5.1 seconds.

click_button('Submit')                                                                                                                 
wait_for_ajax
assert page.has_link?("Edit")

Fails, with Timeout error, unexpectedly. wait_for_ajax should work for 10 seconds, in that time the request has returned already.

click_button('Submit')                                                                                                                 
sleep 6
assert page.has_link?("Edit")

Succeeds - the request is definitely back by the time has_link? runs.

In short, if has_link has to wait for the server thread to return, it won't wait for Capybara.default_max_time, I don't think it waits at all. If has_link does not have to wait because the request is back (as in the example with sleep), then it succeeds.

#10 Updated by Daniel Lobato Garcia over 6 years ago

I'm going to break this one up in a few others, as not all tests fail for the exact same reason.

#11 Updated by Dominic Cleal over 6 years ago

  • Status changed from Ready For Testing to Assigned
  • Pull request deleted (https://github.com/theforeman/foreman/pull/3199)

#12 Updated by Daniel Lobato Garcia over 6 years ago

  • Related to Bug #14067: Host#edit integration tests should not wait for tasks or #show graphs added

#14 Updated by Dominic Cleal over 6 years ago

Daniel Lobato Garcia wrote:

More spotty test results:

http://ci.theforeman.org/job/test_develop_pr_core/8392/database=postgresql,label=fast,ruby=1.9.3/testReport/junit/%28root%29/HostIntegrationTest__clone%20page/test_0001_clones_lookup_values/

clones lookup values - "unable to find field login_login"

This should be fixed by a combination of #14097, #14067 and #14291.

#15 Updated by Ivan Necas over 6 years ago

  • Related to Bug #14340: Icons loading causes intermittent issues in integration tests (especially under heavier load of the test machine) added

#16 Updated by Dominic Cleal over 6 years ago

  • Status changed from Assigned to Resolved

Marking this as resolved now as #14340 and earlier tickets should significantly improve this. We can open new, more specific tickets to address any further failures.

Also available in: Atom PDF