Adding Lua type annotations for yazi

[mikavilpas]

Introduction

There has previously been discussion on adding to yazi for writing lua code. As a (big) fan of types, I was happy to see that there seems to be significant interest in this!

Type annotations for Lua #657
#657

Is there a way to expose yazi's globals to lua-language-server? #934
#934

Benefits

Although I might be preaching to the choir here, I'll state some of the benefits that this can bring:

• it will be easier to write and maintain yazi plugins with features like autocompletion, navigation, and basic type checking
• the plugin ecosystem has huge potential for awesome features, and I think this will encourage many to start writing their own plugins as well as to start contributing to them
  • this also includes me as I would like to write a plugin used by my neovim plugin (https://github.com/mikavilpas/yazi.nvim)

The targets to accomplish this

Since the lua code in yazi is written both in lua files, as well as generated from rust at runtime, ideally the types would be applied to both:

1. the lua files bundled in yazi
2. for the lua code generated from rust

A proposed solution

LuaCATS type annotations are widely in use in the neovim plugin ecosystem. They are a simple "pseudo type system" that is only in use when developing and has no effect at runtime. In my experience with yazi.nvim, the types are not perfect and don't always work correctly, but they still provide a lot of the benefits of a mature type system.

They work nicely with the lua-language-server which I think is the most popular (perhaps the only?) language server for lua.

https://luals.github.io/wiki/annotations/

Technically, I believe it's possible to publish Lua Language Server addons (https://github.com/LuaLS/LLS-Addons) that are totally separate from the lua code. The types can be picked up by the lua language server (according to the addon documentation at https://luals.github.io/wiki/addons/), as there are many examples of this for popular lua libraries in the previous link.

To me that seems like a very good and mature solution. This means the remaining issue is the generation and maintenance of the type annotations.

lua source code files (target 1)

TL;DR: a good solution exists today

To address lua source code files, let's manually add and maintain LuaCATS type annotations to all lua files in yazi.

It will not be difficult to do this, but it will require some work to maintain. However, because the type system is quite simple and can be applied incrementally, I don't see any issues in doing this.

I think this can implemented as long as it's approved and desired as a feature. I think this is a simple task and can also work on it myself.

lua code generated from rust (target 2)

TL;DR: no good solution exists. I have an idea that I can work on if it is approved.

Currently I don't think any good solution exists for this. The problems that I see are:

• rust code is not colocated with the lua code
  • the implementation can easily get out of sync with the type annotations
  • very difficult to maintain reliably
• not easy to test in any meaningful way, which makes the maintenance burder even heavier

I have an idea that I can work on if it is approved:

1. I can write a tool that generates LuaCATS type annotations from rust code.
   I already have written a prototype for a rust crate that implements writing LuaLS annotation files. It implements the LuaCATS type system on a structural level, which means
   • it can represent valid type annotations with rust structs and enums
   • it doesn't do any type checking beyond that as I think it's not necessary and would be very complicated to maintain (and use)
   • it can write the type annotations to a LuaLS annotation file
2. I can see if it's possible to store the type annotations in the rust code itself.
   The idea is to improve colocation which will make it much easier to keep the lua types up to date when they change, and when new lua features are added.
   • my current idea is to use rust attributes that could be attached to various rust constructs. This is similar to how a JSON schema can be generated from rust attributes with the schemars crate. An example of this can be seen here: feat(config): add JSON schema for validation,completion,documentation mfontanini/presenterm#228
     • I'm not sure about this, but it may be that the runtime cost of this is very minimal. It may be possible to force the compiler to #[inline] any extra constructs, although I don't have any experience in doing that at the moment.
3. I can do a proof of concept implementation of this in yazi and open a PR.

@sxyazi I would love to hear your thoughts on all of this to see what direction you would like yazi to go.

[mikavilpas]

Started work on adding types here #986

[mikavilpas]

Also closed it as it did not seem to get traction after a few weeks.

[mikavilpas]

This has been stalled for a while now with no discussion, so I'm hoping to get some going by throwing ideas around.

How about an approach where type annotations could be generated from the documentation? I have seen this approach being used in the neovim community in https://github.com/folke/neodev.nvim?tab=readme-ov-file#-features, providing lua_ls type annotation files for neovim's lua apis.

My understanding of this workflow is that it uses

• github actions to run
• this build file https://github.com/folke/neodev.nvim/blob/main/.github/workflows/types.yml#L30
• which runs this set of lua scripts https://github.com/folke/neodev.nvim/blob/main/lua/neodev/build/init.lua, which seem to use regular expressions for parsing neovim documentations for machine readable patterns
• every hour of every day, forever

[sxyazi]

Most of Yazi's methods and properties are defined in Rust, and there are only a few utility methods in Lua. Unfortunately, it seems there's no reliable way to export types from Rust at the moment.

Do you think it's a good idea to maintain the type definitions in a separate file and generate markdown documentation from that? This way, I'd only need to maintain one set of type definitions, which would give me more time to ensure they're kept up to date.

[mikavilpas]

I agree that currently there seems to be no good solution for rust -> lua_ls type generation. I had a small discussion about this it these two places and some ideas were thrown around. Here they are in case you'd like to track them:

• Lua type annotations wezterm/wezterm#3132 (comment)
• Automatically generate definition files mlua-rs/mlua#392

I'm hoping the situation in the ecosystem will improve in the future.

As of right now, I think what you proposed is the best way to do it. Would you welcome a PR about it?

Do you have some kind of a schema in mind?

[sxyazi]

Now it's supported 🎉🎉

https://github.com/yazi-rs/plugins/tree/main/types.yazi

[mikavilpas]

Looks nice! I looked around and tried to guess how it works. Is the process for building the types (roughly) the following:

• the (markdown) documentation is updated. Maybe this is done manually? I couldn't find where this happens.
• https://github.com/yazi-rs/yazi-rs.github.io/blob/main/scripts/typegen.js parses the markdown, and converts it to LuaLS type definitions

[sxyazi]

Yup! I update the doc manually and include the type info for each property, then use a script to generate Lua type annotations.

Although it's not completely automated, I still only need to maintain just one copy of the API, so it doesn't add too much workflow, and users can more easily understand the types, return values, and other details when they see the doc