Language service

[minestarks]

This issue tracks the implementation of the language service and editor features for Q#. Editor features are things like autocomplete, go-to-definition and hover. The language service is the component that uses the compiler internals to keep track of the current state of the program, update the program in response to document updates, and answer questions about it in response to requests from the editor.

Components involved

1. Language service: this is a Rust crate that contains the actual "smarts" of the language service, including the implementation of each individual feature such as completion, hover, etc.
2. JS bindings: this is a trivial WASM wrapper over the language service to expose it as a JavaScript class. It follows a similar pattern to the existing WASM wrapper for the interpreter (the run() method etc)
3. npm package: This is the same package used by the playground that incorporates the Q# compiler. A new interface will be added to provide access to the language service methods, with the option to host the language service in a separate web worker (just like the interpreter today).
4. VS Code extension: Implementation of the VS Code extensibility APIs. These APIs generally map directly to language service methods.
5. Playground: Similar to VS Code, the language service methods just need to be wired up to the Monaco editor extension points in the playground.
6. Python bindings: Very similar to the WASM/JS bindings mentioned above - a trivial wrapper over the language server crate to expose it as a Python class. Also similar to the Python interpreter wrapper that currently exists today in the pip package.
7. Python LSP server: (for JupyterLab support) A standalone LSP (Language Server Protocol) server implemented in Python. While Python may not be the most obvious choice for an LSP server implementation (Node.js libraries maintained by VS Code would have been a more natural fit), it was chosen to make the installation simple (user can just pip install the extension instead of having to separately install Node.js) for a JupyterLab user.
8. Jupyterlab extension: (for JupyterLab support) Includes the LSP server module and registers it with JupyterLab.

%%{init: { "flowchart": { "nodeSpacing": 3 } } }%%
flowchart TD

subgraph JS
  playground[Playground]
  vscode[VS Code extension]
  website[Website]
  npm[npm package]
end

subgraph Python
  jupyterlab[JupyterLab extension]
  lsp[LSP server]
end

subgraph Rust
  wasm["JS bindings (WASM)"]
  pip[Python bindings]
  interpreter[Interpreter]
  ls[Language service]
  compiler[Compiler]
end

playground
vscode
website
jupyterlab

playground-->npm
vscode-->npm
website-->npm
jupyterlab-->lsp

npm-->wasm
lsp-->pip
wasm-->interpreter
wasm-->ls
pip-->ls

interpreter-->compiler
ls-->compiler

Loading

Implementation plan

This list shows the proposed order of implementation, based on complexity (simpler features like hover are prioritized above more complicated ones like completions) and feature priority (VS Code extension being the highest priority deliverable).

The links are to LSP documentation. Even if we don't implement a true LSP server, we'll follow the LSP spec while designing the interface as it closely resembles the extensibility model used by many editors (VS Code, Monaco, JupyterLab, etc).

• Scaffolding:
  • Language service crate with minimal completion, hover, go-to-def and diagnostics (Language service crate #371)
  • npm API and WASM layer (JS bindings for language service #401) (Language service JavaScript API and web worker #426)
  • Wire up feature providers in VS Code and playground (Language service for VS Code and Playground #429)
• P1 features. These will be implemented with support for only the file currently opened in the editor. The whole Q# program gets recompiled with every document edit (i.e. no performance optimizations):
  • Go to definition:
    • locations within the current file. (LS Go to Def for Current File Items #460)
    • For things defined in the standard library, create a virtual document showing the standard library source.
  • Hover: Hover info for callables, newtypes, patterns at the declaration site as well as the usage site showing name, type and documentation.
    • Compiler dependency: Parsing doc comments. (Parse doc comments #386)
    • Hover for UDT definitions and References. (Language Server Hover on Calls and Variables #415, Namespace Info on Hover over Callables #519, Improve Hover Display #464, Only Display Summary on Hover #449)
  • Diagnostics: Error checking is currently implemented in the playground. Expose in VS Code by publishing diagnostics upon every recompilation (i.e. every keystroke in practice, barring any perf optimizations). (Error squiggles in VS Code #394 (preliminary))
  • Completions (basic): Include all known symbols in the completion list. This is a "good enough" temporary implementation until we have improved error recovery in the parser and the ability to provide more accurate completions for each context. (Completion list usability tweaks #443)
  • Compiler dependency: Error recovery in the parser - ability to get information about an imperfect program. (Basic parser error recovery #380)
  • Completions (better): More sophisticated completion list behavior. We could leverage the parser here to narrow down the next expected syntax kind (e.g. type, expression) and populate the completion list based on the tokens and known names (e.g. callables, locals) that are applicable to that syntax.
    • Compiler dependency: parser-provided completions
    • Completion item details (type, documentation). (LS Completions Details #511)
  • Signature help: Signature help for function calls including parameter documentation.
• Notebook cell support. Notebook cells use a variant of Q# that allows top-level items and expressions.
  • Use the "fragment" compiler rather than the "regular" compiler to construct a program.
  • There may be a compiler dependency here (investigate)
• P2 features. Some of these features bring new complexity as they can return text edits back to the client.
  • Format document
  • Format range, format on type - Ability to format the subset of a document, as well the applicable range in response to a trigger character (e.g. })
  • Find all references - essentially reverse go-to-definition. (LS: Find All References #830 )
  • Symbol rename - determine if a rename is valid in the current location, use find-all-references to locate references to symbol, return text edits for rename
  • Quick fixes Contrast with refactorings, which are designated as a P3 feature below. Code fixes are associated with diagnostics (e.g. "unused identifier") reported by the compiler/linter and provide text edits that would "fix" that diagnostic.
    • Compiler dependency: Lint-level diagnostics for desired quick fixes.
• Support for closed files (only necessary when we support file references (modules, imports etc) and/or projects):
  • VS Code Web compatible File I/O using vscode's WASI implementation, mapping of utf-16 line/column offsets
  • File watching - updating compilation in response to workspace changes (e.g. git branch switch)
• LSP server to enable support for other editors
  • Python LSP server and JupyterLab support (blocked on jupyterlab-lsp supporting the latest version of JupyterLab).
• Performance optimizations, as appropriate. These come with tradeoffs, so we should only implement them if we're observing actual performance problems.
  • Debouncing/batching - strategies to reduce the number of recompilations by introducing a slight delay before recompiling the document on each keystroke/file update. This may introduce a slight delay to completions and error squiggles.
  • Cancellation - cancel an outstanding recompilation when a document is updated. May require web workers or some other threading mechanism.
  • Background checking - ability for language service requests to run concurrently with checking. Requires web workers for WASM.
• P3 features:
  • Go to symbol
  • Refactorings - e.g. extract function. Contrast with quick fixes described above.
  • Semantic highlighting - e.g. highlighting of resolved identifiers based on type
  • Document highlights
  • ...and so on.
• Even more expensive performance optimizations (hopefully avoided):
  • Background program update - ability to work with an immutable snapshot of the program as updates are being made in the background (in response to keystrokes or file updates). Requires web workers for WASM.
  • Compiler dependency: Partial recompilation - ability to recompile only parts of a program as the document gets updated.

[minestarks]

Draft PR wiring up basic implementation of P1 features in VS Code and the playground: #364

[sezna]

@minestarks -- is #1038 going to unblock LSP support, or are there other changes required?

Also, perhaps we could file some separate issues for some of these features and use this issue to track them?

[minestarks]

@minestarks -- is #1038 going to unblock LSP support, or are there other changes required?

@sezna well, we'd have to implement an LSP server... but if you're asking if anything in the core language service crate needs to change, I don't think so. I had a prototype for this a few months ago using the https://github.com/python-lsp/python-lsp-jsonrpc package which was working in JupyterLab. I think it's just a matter of wiring pieces together.

Also, perhaps we could file some separate issues for some of these features and use this issue to track them?

Yes this issue is quite big, I don't mind breaking off a few items. I didn't want to create issues for things that aren't even on our radar, but for near-term stuff (e.g. formatting) and for stuff we want a dedicated discussion thread for, it makes sense.