fix(runner): show stacktrace on test timeout error

[hi-ogawa]

Description

• Related fix(runner): show stacktrace on hook timeout error #7502

#7502 added error stack for hooks, but I forgot to use it for test function. This still avoids unconditionally accessing stackTraceError.stack since that's where custom prepareStackTrace kicks in and it might be costly. It's only accessed either when includeTaskLocation enabled or actually timeout error occurs.

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• Ideally, include a test that fails without this PR but passes with it.
• Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• Run the tests with pnpm test:ci.

Documentation

• If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	ece842f
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/67f0d96a8cc344000893ff11
😎 Deploy Preview	https://deploy-preview-7799--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[sheremet-va]

#7502 added error stack for hooks, but I forgot to use it for test function. This still avoids unconditionally accessing stackTraceError.stack since that's where custom prepareStackTrace kicks in and it might be costly. It's only accessed either when includeTaskLocation enabled or actually timeout error occurs.

I just want to add that this is true only for Node (and probably Deno?), browsers evaluate stack in place (stack traces are much faster in browsers tho). This is why I wanted to reuse the error instead of overwriting the stack property

[hi-ogawa]

That's true, but I was just thinking not wasting prepareStackTrace on Node is worth enough. I think I should just bench this once and for all. If the perf is not significant, we can get errors as many as we want 😃

[hi-ogawa]

I created a benchmark to compare test collection time with and without includeTaskLocation https://github.com/hi-ogawa/reproductions/tree/main/vitest-error-stack-perf

Number of tests	Time (ms) - Default	Time (ms) - With Task Location
0010	5.89	11.27
0020	7.46	11.57
0030	5.40	13.69
0040	5.19	15.05
0050	6.86	16.97
0100	7.06	25.87
0200	9.14	43.55
0300	10.14	56.97
0400	12.60	70.06
0500	14.28	76.59
1000	27.10	122.63
2000	41.47	224.46
3000	64.94	335.66
4000	91.33	451.15
5000	157.71	639.02

It looks like the extra cost Error.stack is about 0.1ms, so I guess this is negligible territoy?

[sheremet-va]

I am not sure this works like that. Since we don't access .stack here, does stackTraceLimit matter at all? Shouldn't it wrap .stack?

[hi-ogawa]

Good point, but it turns out this is actually fine based on this repro

// repro.js
console.log("[default Error.stackTraceLimit]", Error.stackTraceLimit)

function testThrow(depth) {
  if (depth === 0) {
    const original = Error.stackTraceLimit;
    Error.stackTraceLimit = 1;
    const error1 = new Error('1')
    Error.stackTraceLimit = 2;
    const error2 = new Error('2')
    Error.stackTraceLimit = 3;
    const error3 = new Error('3')
    Error.stackTraceLimit = original;
    return {
      stack1: error1.stack,
      stack2: error2.stack,
      stack3: error3.stack,
    }
  }
  return testThrow(depth - 1)
}

console.log(testThrow(10))

$ node repro.js 
[default Error.stackTraceLimit] 10
{
  stack1: 'Error: 1\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:7:20)',
  stack2: 'Error: 2\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:9:20)\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:19:10)',
  stack3: 'Error: 3\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:11:20)\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:19:10)\n' +
    '    at testThrow (file:///home/hiroshi/code/personal/reproductions/vitest-error-stack-perf/repro.js:19:10)'
}

[sheremet-va]

It looks like the extra cost Error.stack is about 0.1ms, so I guess this is negligible territoy?

The time depends on how big the stack is. The file stack is also cached for subsequent access (we don't need to parse the file and the source map multiple times). By the way, it's possible that the time is low because we already do the parsing somewhere

In my tests I managed to trigger a 200ms stack access, for example, and we actually had an issue in Vitest itself that slowed down tests by 2 times when we had vi.mock in a setup file (it now disables the custom Error.prepareStackTrace method while creating the error)

I just want to make sure we are all on the same page here with how Vitest deals with error stacks. In the browser, for example, we don't override the prepareStackTrace, so the performance is negligible there.

[hi-ogawa]

I think we are on the same page. My current understanding is that the error instantiation new Error() itself is negligible, but accessing stack new Error("...").stack (maybe even without Error.prepareStackTrace) is potentially expensive. So in this PR, I still guarded stack access when it's not needed.

With further investigation, it's also possible that we can verify the cost of Error.stack doesn't affect much when considering overall test collection logic already required for each test. I'm not sure how to definitely prove that yet, so that part is pending. We can consider this in the future.

Btw, this is the version with multiple test files and collect duration seems larger (though overall test duration isn't affected much) 🤔 https://github.com/hi-ogawa/reproductions/tree/main/vitest-error-stack-perf2

Number of test files	Duration	Duration with includeTaskLocation	collect	collect with includeTaskLocation
100	1.68s	1.72s	2.10s	2.84s
200	3.25s	3.50s	3.93s	6.19s
300	5.05s	5.48s	6.04s	9.29s