Running Parallel Tests in Suite

[eyphka]

When running multiple tests in a suite which have been set to suite.t().Parallel(), tests will pass even if they should fail.

[dwlnetnl]

That's true, but as it's currently structured it isn't possible to run suite tests in parallel. This is because before and after each suite test SetupTest and TearDownTest is run.

[ernesto-jimenez]

@eyphka do you want to make a PR to address it? :)

[ernesto-jimenez]

Related: #52

[eyphka]

Sure

On Sunday, November 1, 2015, Ernesto Jiménez notifications@github.com
wrote:

Related: #52 #52

—
Reply to this email directly or view it on GitHub
#187 (comment).

[oryband]

Reviving this issue. Presumably it should be possible to run suite tests in parallel.

The problem with parallelism here is that the suite members used in tests have a shared single instance for all. Tests running in parallel could potentially access these members at the same time, causing them to "collide" with each other via these members.

However, my suggestion is that testify will add a special Parallel() flag to the suite. This new flag will instantiate new instance members for every new Parallel() test, calling Setup/TearDownTest() as well.

This will of course require extra work from the test developer, to make sure Setup/TearDownTest() are suitable for being called with/without parallel (As some tests will be run in parallel, and some don't).

Thoughts?

[dansimau]

I just whipped up a slightly different approach to running tests in parallel:
https://gist.github.com/dansimau/128826e692d7834eb594bb7fd41d2926

Similar to #369, it introduces a RunParallel, except that it uses reflection to create a new instance of the suite for each test run.

Because a new suite is initialised before every test, it means tests can't share data on the suite struct. This is a change in behaviour, though I imagine you could change the patch to make a copy of the suite struct instead of initialising a new one (that way tests could still share data and setup/teardown code using SetupSuite).

I actually prefer the share-nothing approach though; it feels like a bad practice to modify data on the suite struct while tests are running, especially now in parallel. I can't see why tests should ever share data (outside what is set up in SetupTest); but maybe I'm missing a use case.

[varunrau]

is it still the case that suite tests can't be run in parallel?

[alessio]

Hi! Any updates on this? Are you planning to do work on this feature request?

[ysmood]

One solution:

package main_test

import (
	"testing"
	"time"

	"github.com/stretchr/testify/assert"
	"github.com/ysmood/got"
)

func Test(t *testing.T) {
	got.Each(t, beforeEach)
}

func beforeEach(t *testing.T) Suite {
	t.Parallel()
	return Suite{assert.New(t)}
}

type Suite struct { // struct that holds subtests
	*assert.Assertions
}

func (s Suite) A() { // test case A
	time.Sleep(time.Second)
	s.Equal(1, 1)
}

func (s Suite) B() { // test case B
	time.Sleep(time.Second)
	s.Equal(2, 1)
}

As you can see, test A is not affected:

$ go test       
--- FAIL: Test (0.00s)
    --- FAIL: Test/B (1.00s)
        main_test.go:31: 
            	Error Trace:	main_test.go:31
            	            				value.go:475
            	            				value.go:336
            	            				each.go:42
            	            				value.go:564
            	            				asm_amd64.s:20
            	Error:      	Not equal: 
            	            	expected: 2
            	            	actual  : 1
            	Test:       	Test/B
FAIL
exit status 1
FAIL	goplay/default	1.254s

[tisonkun]

Reviving this issue. Presumably it should be possible to run suite tests in parallel.

The problem with parallelism here is that the suite members used in tests have a shared single instance for all. Tests running in parallel could potentially access these members at the same time, causing them to "collide" with each other via these members.

However, my suggestion is that testify will add a special Parallel() flag to the suite. This new flag will instantiate new instance members for every new Parallel() test, calling Setup/TearDownTest() as well.

This will of course require extra work from the test developer, to make sure Setup/TearDownTest() are suitable for being called with/without parallel (As some tests will be run in parallel, and some don't).

Thoughts?

@oryband can we implement it now?

[tisonkun]

func (s *mySuiteA) TestA2() { // test case A
	t := s.T()
	t.Parallel()
	assert.Equal(t, 3, 1)
}

cache the *testing.T pointer can solve the problem.

This issue is caused by the following code snippet

test := testing.InternalTest{
  Name: method.Name,
  F: func(t *testing.T) {
    // ...
    defer func() {
      // ...
      suite.SetT(parentT)
      // ...
    }()
    // ...
    method.Func.Call([]reflect.Value{reflect.ValueOf(suite)})
  }
}

... and when run test in parallel, defer function executed from TestA even if the body of TestB not be executed, T set to the parent T so the child test failed but reported as passed. Because the fail happened on the parent T instead of the child T.