Parallel Testing Enhancements

[smark-1]

Code of Conduct

• I agree to follow Django's Code of Conduct

Feature Description

Introduce a new command-line flag that, when enabled, automatically re-executes any tests that failed due to being non-pickleable in parallel mode on the main thread. This lets developers inspect the actual failures without rerunning the entire test suite or manually re-selecting individual tests.

Problem

Running tests in parallel is often drastically faster but it comes with the downside that sometimes when the tests fail you cannot see the error message. Which means that if there is any error you either have to run all the tests again (or for a specific app) in a single thread just to see the error message(s). This can lead to often just running all the tests once on a single thread.

Request or proposal

proposal

Additional Details

I envision it to work something like this. I am not sure about the exact order of when the failed tests would rerun or if the database should be destroyed and recreated before running the failed tests.

python manage.py test --parallel --rerunfailed
Found 300 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
.........................................................................F........................................

FAIL: test_was_published_recently_with_future_poll (polls.tests.PollMethodTests)

----------------------------------------------------------------------
Ran 300 test in 8.403s

FAILED (failures=1)


Running failed tests

F

======================================================================
FAIL: test_was_published_recently_with_future_poll (polls.tests.PollMethodTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/dev/mysite/polls/tests.py", line 16, in test_was_published_recently_with_future_poll
    self.assertIs(future_poll.was_published_recently(), False)
AssertionError: True is not False

----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (failures=1)

Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

Implementation Suggestions

No response

[github-actions]

Thank you smark-1 for sharing your idea! We have a lot of them so please be patient. You can see the current queue here.

Community instructions

For commenters, please use the emoji reactions on the issue to express support, and/or concern easily. Please use the comments to ask questions or contribute knowledge about the idea. It is unhelpful to post comments of "I'd love this" or "What's the state of this?"

Reaction Guide

• 👍 This is something I would use
• 👎 This is something that would cause problems for me or Django
• 😕 I’m indifferent to this
• 🎉 This is an easy win

[carltongibson]

Q: does installing tblib not resolve the traceback issue? (IIRC the stack trace says to do this... 🤔)

[smark-1]

In the docs there is a warning saying that "Django may be unable to display the exception traceback..." Which I don't think it is so uncommon to happen even with tblib installed.

If tblib is installed and was able to parse the error then that test would not be rerun.

Also the documentation is not so clear if only tblib doesn't work on windows or if running tests in parallel as a whole doesn't work on windows.

If this is not correct then shouldn't the tests be run in parallel by default and a flag should be added to run in a single thread (--parallel 1) since there is rarely a downside to this?