Ava skips next tests in file if a previous test timeouts

[emilis-ideait]

Minimal reproducible example

In our test suite we have multiple test(...) tests per file. We are using Selenium to drive a browser in these tests.

Sometimes a test runs longer than the specified timeout.

I noticed that in such case all the remaining tests in the file will be skipped and a "X tests were pending in example-file.js" message will be printed to output.

I expected only the slow test to timeout and the remaining tests to run.

ava.config.js:

export default {
    concurrency:            1,
    files: [
        'test-a.test.js',
        'test-b.test.js',
    ],
    serial:                 true,
    timeout:                '2s',
    verbose:                true,
};

Same happens with both Ava 4.2.0 and 5.1.1.

[novemberborn]

AVA's timeout detection works by tracking how long it's been since there was last an "event" coming out of a test worker. If that takes too long, the worker is stopped.

Your setup executes tests serially, so subsequent tests are not run.

I think if you'd use t.timeout() within the test, then that specific test would time out but the rest would continue running.

[emilis-ideait]

Thank you for your time!

The current "pending" behavior is unexpected, confusing and undocumented (there's nothing about it in the docs).

The worst part is that it hides all test output for the timed out test and makes it harder to figure out the cause of a timeout.

At first glance it seems like the remaining tests timed out too. The probable cause of which would be performance issues of a CI server -- not a problem with a single test.

The workaround (t.timeout()) is also non-obvious and does not communicate its purpose well for the people reading the test code.

If I understand correctly, non-serial test() tests would run in separate workers? Why can't serial tests run in separate workers?

[novemberborn]

The current "pending" behavior is unexpected, confusing and undocumented (there's nothing about it in the docs).

It does not explicitly say that pending tests that remain in currently executing test files, or queued test files that haven't been loaded yet are not executed, correct. I'm not convinced that's a useful addition to the docs though (https://github.com/avajs/ava/blob/main/docs/07-test-timeouts.md).

At first glance it seems like the remaining tests timed out too. The probable cause of which would be performance issues of a CI server -- not a problem with a single test.

It should say something about pending tests that remain. The latest AVA 5 should be a little clearer with that output but always happy to improve it further.

The workaround (t.timeout()) is also non-obvious and does not communicate its purpose well for the people reading the test code.

Yes, it's better when you have a long global timeout, and then a specific smaller timeout in an individual test.

If I understand correctly, non-serial test() tests would run in separate workers? Why can't serial tests run in separate workers?

All tests from a single file run in the same worker.