Testing in VS Code

[connor4312]

State of the World

Testing support in VS Code has been a feature request for a long time. The VS Code community has build excellent extensions around testing, for example:

• The Test Explorer UI from @hbenl
• Wallaby.js from the Wallaby team
• Jest from @orta
• ...and many more

Each implementation of testing presents a different set of features, UI, and idiomaticity. Because there is no sanctioned approach to tests in VS Code, extension developers tend to make bespoke implementations, as we've seen in the Python and Java language extensions. Ideally, like in debugging, a VS Code user would have just about the same experience as they work between projects and languages.

VS Code's Approach

Investigate how VS Code can improve the testing support. Several extensions are already providing testing support, explore what APIs/UIs could be added to improve these testing extensions and the test running experience. -- 2020 Roadmap

The Test Explorer UI presents the best point of inspiration for us, as there are many existing extensions built on its API: it's capable and proven. Regardless of the direction we take in VS Code, we should have a way for its Test Adapters to be upgraded to the new world.

Wallaby is an excellent extension, but it's tailored and purpose-built to JavaScript, and includes functionality which is not readily portable to other languages. While it is a good source for inspiration, we're not aiming to encompass Wallaby's feature set in the extension points we provide, at least not yet.

We're prototyping an API in the extension host, but there are a number of approaches we can take:

Extension Host ('traditional' VS Code API)	'Test Protocol' (like DAP/LSP)	Extension (like existing test explorer)
+ Simple to adopt for extension authors + Easier to manage state + Clear way to build 'official' test extensions	+ Encourages keeping expensive work in child processes + Could be theoretically shared with VS and other editors	+ Keep VS Code core slim + Unclear whether there's significant functionality we'd want that's not already possible in exthost api
- The 'obvious path' is doing heavy lifting in the extension host process, which is undesirable	- Additional implementation and maintainence complexity for VS Code - Less friendly, additional complexity than TS APIs for extension authors	- Additional extension and set of libraries to maintain+version for types and implementation - Less clear there's an official pathway for test extensions

API Design

The following is a working draft of an API design. It should not be considered final, or anything close to final. This post will be edited as it evolves.

Changes versus the Test Adapter API

As mentioned, the test adapter API and this one provide a similar end user experience. Here are the notable changes we made:

• The test adapter API does not distinguish between watching a workspace and watching a file. In some cases, there is an existing process that reads workspace tests (such as a language server in Java) or it's not much more expensive to get workspace tests than file tests (such as mocha, perhaps). However, some cases, like Go, providing tests for a single file can be done very cheaply and efficiently without needing to involve the workspace.

  In this API we expect the TestProvider to, after activation, always provide tests for the visible text editors, and we only request tests for the entire workspace when required (i.e. when the UI needs to enumerate them).

• We have modeled the test state more closely after the existing DiagnosticCollection, where the Test Adapter API uses only events to enumerate tests and does not have a central collection.

• The Test Adapter API makes the distinction between suites and tests, we do not. They have almost identical capabilities, and in at least one scenario the 'suites' are more like tests and the leaf 'tests' cannot be run individually.

• We use object identity rather than ID for referencing tests. This is in line with other items in the VS Code API, including Diagnostics.

Ideas and Open Questions

See the testing label for current work, questions, and problems.

API

See the current working proposal in https://github.com/microsoft/vscode/blob/master/src/vs/vscode.proposed.d.ts (ctrl+f for 107467)

[connorshea]

👋 I'm the maintainer of the Ruby Test Adapter extension and just wanted to drop in and give a bit of context on the pain points I've had with the Test Adapter API. The Adapter API works well for the most part, but the biggest problem I've had with it is getting it work well with large codebases.

I developed the Ruby adapter with my Rails app vglist in mind, and at the time it had around 300-400 tests in total (it now has 700-800). My adapter worked fine with that, it took a few seconds (maybe 5-10) to reload all the test suite data whenever a change was made to the test code. However, even 5-10 seconds is annoying to wait for when you just want to run your tests right after you modify them. And now I work at a company with a Rails app that has over 12,000 RSpec tests, so my extension now takes about 45 seconds to reload the tests whenever I change them (and it'd be even longer if I didn't have a pretty well-spec'd laptop). I don't ever really use my test adapter with that codebase because it slows me down far too much.

The problem is that the API doesn't support modifying the existing set of tests, only replacing all of them at once. Which requires RSpec to generate information for every test in the codebase whenever it reloads the tests.

I've considered a workaround where I'd cache the information in a tempfile or something, and have rspec only load information about changed files, and then modify the cached tree and send it up to the explorer UI, but that's complex and I haven't gotten around to actually doing it.

There are other problems - which I'm not really sure how to solve - with being unable to run the tests while it's loading the test suite because it doesn't handle the tests that pass while it's loading (the data can theoretically get out-of-sync if you, say, change a bunch of things between reloads, but >99% of the tests in the codebase will be exactly the same before and after any given change I make to the suite). So it's maybe worth considering that when designing the invalidation logic.

There are also problems with being able to view test logs to figure out issues when tests fail, but that's more of a problem with how the RSpec framework works than with anything the Test Adapter API does.

The current API Sketch looks quite good from what I'm seeing. I especially like the use of a cancellation token, which if I understand correctly seems like a much cleaner solution than the Adapter API where you just blindly attempt to kill any relevant running processes. There are some things missing there (e.g. how do I handle the case where the suite is unloadable because the user has created an invalid test suite via some syntax error?), but overall it looks like it takes a lot of the good decisions from the Adapter extension API.

Anyway, I hope this makes some sense. I understand it's probably not a super cohesive comment since I wrote it on-the-fly, but I've been hoping something like this would be introduced into the core of VS Code eventually, so I wanted to make sure I got my two cents in :)

[connor4312]

Thank you for that feedback, it's super helpful!

Scoping tests and handling large codebases is definitely something we want to tackle from the outset. Like all features we build, we will be dogfooding it in the VS Code repo, so it is a priority to be able to handle that. As you saw in the sketch, I think the collection approach solves the pain points around having to replace all tests.

The idea of running tests during load is interesting. The base case of running all tests is pretty simple from the API perspective -- we would just runTests without any filters and let the extension run as many as it can. So long as it's able to reconcile the collection into the right state running concurrent discovery and execution, everything should be fine.

One scenario that isn't covered is running a subset of tests as discovery is happening (e.g. if I wanted to run all tests with the word "foo" in them) without starting a new test run. This would take some finessing and will not be supported by all adapters, for instance Go could never support it without changes to its toolchain, so is probably not worth designing for...

There are some things missing there (e.g. how do I handle the case where the suite is unloadable because the user has created an invalid test suite via some syntax error?),

I think the right thing to do would be for the test extension to emit a diagnostic for that using our existing API. I've added it to the open questions as well though for discussion.

[connorshea]

@connor4312 generally speaking I'd want to give the user an error notification so they know something has broken and their suite failed to load, rather than creating a diagnostic. But as long as the extension can choose what to do if that occurs, I suppose the Test API in VS Code doesn't need to care about that.

[connor4312]

When I say diagnostic I mean the "Diagnostic" in the VS Code API that appears in the problems view (and can also have an error squiggle in the editor)

The downside there is that if they have a syntax error, having the problem be duplicated by both the language server and test extension will be some ugly clutter.

[connorshea]

The reason I wouldn't use a diagnostic is that 1) I personally rarely ever look at diagnostics/errors in the diagnostics view because the signal to noise ratio in my experience is so high, and 2) there are situations where a diagnostic wouldn't really make sense, for example if the user's tests fail to load because their local Postgres server isn't running or because they forgot to install their Ruby gems so the RSpec gem isn't even available.

[hbenl]

Some thoughts from my side (I'm the author of the Test Explorer UI extension):

• the proposed API looks good, in particular having a central test collection will probably be welcomed by many TestProvider authors. This is obviously necessary if you want to support only detecting tests in the currently opened files (which I had never thought about) but even without that there seem to be some scenarios where TestAdapter authors struggle with this part of the Test Explorer API
• bringing the existing TestAdapters to the new world could be done with a bridge extension (or built into Test Explorer itself) which registers all the tests coming from TestAdapters with the new TestProvider API
• concerning the question whether a TestProvider should run in the Extension Host: I'm not worried about expensive work being done in the Extension Host process here: the most expensive part is usually running the tests and TestProviders should always do that in a child process anyway
• perhaps some other TestAdapter authors would also like to share their thoughts: @Raagh @kavod-io @Gwenio @Testy @marcellourbani @numaru @bneumann @Florin-Popescu @matepek @fredericbonnet @drleq @dampsoft @betwo @zcoinofficial @vshaxe @kondratyev-nv @recca0120 @DEVSENSE @swellaby @prash-wghats @Derivitec @Bochlin @maziac

[bneumann]

Thanks for the mention @hbenl. Looks good to me, I also agree with @connorshea about the notificantion in case something is broken. I had that a lot and having a possibility to tell the users what went wrong might be good.

One thing that I was missing, and I am not sure if it possible, is to match my tests with the source files. My adapter runs C code and I can't simply grep all files for the function names. I could do that if the user registered the toolchain somehow but that would require a lot of setup for the user to make prior to testing. So it would be nice to somehow get cross information from the language server and check which files in my project are actually test files.

Not sure if that is meant by:
In a golden scenario, invalidation of tests would be done by a language server which can intelligently determine specific tests that should be invalidated when a file or a file dependency changes. Maybe this is still handled by an event on the TestProvider, but if we take a "Test Protocol" approach then coordination will be harder.

[hbenl]

One more thought: the central difference between tests and suites in Test Explorer is that the state of a suite is always computed from the states of its children (with one exception: a suite can be marked as errored). TestProvider doesn't distinguish between tests and suites but it should (optionally) provide the capability to let the state of a test be computed from the states of its children. Perhaps this could be flagged by another TestRunState (e.g. INHERIT). Of course it could be left to TestProvider authors to update the parents but this can become a slightly hairy issue if you want to avoid potential performance problems, so it would be nice if VS Code provided that.

[matepek]

Hello I'm the author of C++ TestMate

The Test Adapter API makes the distinction between suites and tests, we do not. They have almost identical capabilities, and in at least one scenario the 'suites' are more like tests and the leaf 'tests' cannot be run individually.

Praise the initiative. The feature will be useful for Catch2 too. (related issue)

invalidation and auto-run

I think only language server isn't enough. We need the flexibility to retire tests in case of external dependencies and file changes.

A request of my user:
I don't see an "output window" concept on the current API but I assume that it is unavoidable. (Some test frameworks can provide useful output which cannot be associated to any tests.)
The person expressed the need to have the output window updated before the test itself finishes. I believe it is a reasonable request. It seems to me that multiple TestState with "Running" can be a solution. But I don't see how the output window fits into the current API.

About test state propagation:
For my extension if a test fails all the ascendants has to be failed too. Even a node has a "Passed" state if one of the descendant's has a "Failed" or "Errored" state than has to be propagated. Some precedence also seems necessary like:
"Running" over "Errored" over "Failed" over "Skipped" over "Unset".

But this raises another question. Scenario: B and C under A. Test B reports failure and test B still running. What should be the state of A? According to previous precedence it should be "Running" but users might would find it useful to see that some tests part of that "Suite"/node has already failed. I have some ideas about it but all of the sacrifices the simplicity and clearness of the API.

Question: The integrated test explorer what extra features will be available for VSCode users? I mean we have a working system now thanks to @hbenl. What is strongest the motivation of the integration?