Bug #13293
closedIntermittent host#edit parameter integration test failures after timeout
Description
Further to #13111, some host#edit parameter integration tests are failing intermittently but after the Capybara timeout (#13111 fixed instances of the timeout not being used). These can be seen from the verbose test logs when it takes over 30 seconds.
1) Assertion failure when removing an overridden global parameter and checking the override button is added back.
HostIntegrationTest::edit page.test_0004_correctly override global params test/integration/host_test.rb:461
Fails on the third line:
refute page.has_selector?('#inherited_parameters .btn[data-tag=override]') page.find('#global_parameters_table a[data-original-title="Remove Parameter"]').click assert page.has_selector?('#inherited_parameters .btn[data-tag=override]')
2) Assertion failure when overriding a global parameter and checking the override button is removed.
Failed assertion, no message given. (Minitest::Assertion) /var/lib/workspace/workspace/test_develop_pr_core/database/postgresql/label/fast/ruby/1.9.3/test/integration/host_test.rb:453
Third line:
assert page.has_selector?('#inherited_parameters .btn[data-tag=override]') page.find('#inherited_parameters .btn[data-tag=override]').click assert page.has_no_selector?('#inherited_parameters .btn[data-tag=override]')
3) Assertion failure when overriding a smart class parameter and checking the text area is enabled.
HostIntegrationTest::edit page.test_0001_class parameters and overrides are displayed correctly for booleans (from HostIntegrationTest__edit page) Capybara::ElementNotFound: Unable to find css "textarea:enabled" test/integration/host_test.rb:392:in `block (2 levels) in <class:HostIntegrationTest>' (Capybara::ElementNotFound)
Fourth line:
assert_equal class_params.find("textarea").value, "true" assert class_params.find("textarea:disabled") class_params.find("a[data-tag='override']").click assert class_params.find("textarea:enabled")
Files
Updated by Dominic Cleal almost 9 years ago
- Related to Bug #13111: Intermittent host#edit parameter integration test failures added
Updated by Daniel Lobato Garcia almost 9 years ago
Mind to post the screenshots saved in jenkins? I gave #3 a try before but I can't reproduce it locally at all with chromedriver-helper (nor poltergeist). It always removed the override right away, I don't know why would it take 30s+ on Jenkins. Are there any memory stats for jobs in Jenkins?
Updated by Daniel Lobato Garcia almost 9 years ago
Number 3 is synchronous Javascript (override_class_param), so I don't think that one failure caused by concurrency. Aside from that, Caoybara docs say that .find waits for Javascript :/
Updated by Dominic Cleal almost 9 years ago
- File screenshot_2016-01-18-15-15-15.704.html screenshot_2016-01-18-15-15-15.704.html added
- File screenshot_2016-01-18-15-15-15.704.png screenshot_2016-01-18-15-15-15.704.png added
I think the attached screenshot/HTML are for issue (1), but I've realised that I didn't keep the associated log output.
Not sure that we have any memory stats, but it's likely likely that some swapping is happening when running multiple integration tests. It shouldn't be lasting for 30 seconds though, and I'd expect to see more of an effect on adjacent tests.
Updated by Daniel Lobato Garcia almost 9 years ago
Notice new intermittent errors are showing up:
HostIntegrationTest::create new host page.test_0001_tabs are present
HostIntegrationTest::edit page.test_0007_choosing a hostgroup does not override other host attributes
Updated by The Foreman Bot almost 9 years ago
- Status changed from New to Ready For Testing
- Assignee set to Daniel Lobato Garcia
- Pull request https://github.com/theforeman/foreman/pull/3199 added
Updated by Dominic Cleal almost 9 years ago
- Related to Bug #14034: Intermediate failures on Jenkins for puppetclass_lookup_key integration test added
Updated by Daniel Lobato Garcia almost 9 years ago
Found a reproducer - slow down the server thread by half the Capybara.default_max_time by changing ApplicationContrller to include
after_filter -> { sleep(15) }
If you set the default_max_time to 10, and slow down the requests by 5 seconds it's also easy to reproduce.
Updated by Daniel Lobato Garcia almost 9 years ago
Some interesting findings:
When I slow down requests on ApplicationController with after_filter -> { sleep 5 } , Capybara.default_max_wait_time = 10, I get interesting results for 'correctly override global params':
click_button('Submit') assert page.has_link?("Edit")
Fails , unexpectedly, as has_link? should work for 10 seconds, but it doesn't - the request returns in 5.1 seconds.
click_button('Submit') wait_for_ajax assert page.has_link?("Edit")
Fails, with Timeout error, unexpectedly. wait_for_ajax should work for 10 seconds, in that time the request has returned already.
click_button('Submit') sleep 6 assert page.has_link?("Edit")
Succeeds - the request is definitely back by the time has_link? runs.
In short, if has_link has to wait for the server thread to return, it won't wait for Capybara.default_max_time, I don't think it waits at all. If has_link does not have to wait because the request is back (as in the example with sleep), then it succeeds.
Updated by Daniel Lobato Garcia almost 9 years ago
I'm going to break this one up in a few others, as not all tests fail for the exact same reason.
Updated by Dominic Cleal almost 9 years ago
- Status changed from Ready For Testing to Assigned
- Pull request deleted (
https://github.com/theforeman/foreman/pull/3199)
Updated by Daniel Lobato Garcia almost 9 years ago
- Related to Bug #14067: Host#edit integration tests should not wait for tasks or #show graphs added
Updated by Daniel Lobato Garcia almost 9 years ago
More spotty test results:
clones lookup values - "unable to find field login_login"
show page - "looked for h1 tag, not found"
class parameters and overrides are displayed correctly for booleans - Host::Base#name= delegated to primary_interface.name=, but primary_interface is nil
class parameters and overrides are displayed correctly for strings - "Capybara::ElementNotFound: Unable to find css "textarea:enabled"..."
Updated by Dominic Cleal over 8 years ago
Updated by Ivan Necas over 8 years ago
- Related to Bug #14340: Icons loading causes intermittent issues in integration tests (especially under heavier load of the test machine) added
Updated by Dominic Cleal over 8 years ago
- Status changed from Assigned to Resolved
Marking this as resolved now as #14340 and earlier tickets should significantly improve this. We can open new, more specific tickets to address any further failures.