[DISCUSS] Execution Environment for Asyncify Lightweight Synchronize System Calls

[tqchen]

Background

One of the primary goal of WASM is to run on the web, and web is async in nature. On the other hand, there is always a need to introduce "synchronize style" system calls that might block, most notable examples includes: file system operations, networking, GPU and accelerator synchronization in machine learning.

Choices

There are several ways to deal with these system calls:

• C0: always turn synchronize calls into asynchronize version(e.g. callbacks). This puts the burden on the compiler and language to implement pause/resume inside WASM, which brings additional overhead.
• C1: Allow synchronize calls in the native environment(e.g. a wasm vm), where synchronization is fine. This cause mismatch between the web and native execution.
• C2: Asynctify the system calls via compiler(save the wasm stack into linear memory every time system call happens), or support ti via corountine
• C3: Standardize mechanism to asynchify synchronize system calls.

Lightweight Asynctification

C2 is certainly one viable option, however it might bring additional execution time overhead. I will elaborate the choice C3 and discuss why it might need standardization (either in WASI or other part of the WebAssembly spec).

Because WebAssembly is executed as a VM, all of its states in the stack are already stored somewhere(e.g. a stack data structure in a WASM VM). When a wasm program calls into a system interface that might synchronize, the execution environment simply "freeze" the state of wasm VM(by storing potential register items if any, into a context). And calls an asynchronize version of the system call, then the execution environment can resume the execution when the callback is done.

This approach certainly have its limitations, since we can only resume at the callsite. And we cannot call into the wasm vm again before the previous system call is resumed.

However, by doing so, we get the asynctification "for free". Because there is no need of saving the stack(as they are part of the linear memory that get frozen).

We believe this is an important design decision for system libraries that will affect the future of machine learning and other applications in WASM. So I hope to use this thread to gather discussions.

Given that C3 requires standardization through of the execution environment, I hope to use this thread to seed the discussion of the topic. It would certainly become part of WASM JS API, but the implications also goes beyond JS, as it could also impact WASM VMs like wasmtime/wasmer.

[devsnek]

There is some work going into the coroutines/generators/effects/etc design space for wasm which would solve this in a somewhat more generic way.

[tqchen]

I agree that adding coroutine support will solve it more generically, that is why it is listed in the C2 as an option. However, there is a tradeoff in here,

• C2: The support for resume/pause certainly comes with additional overhead to save/restore the stack
• C3: If we impose the restriction, then there is no need to save the stack, as we can just freeze the execution state.

As a analogy, think of C3 as hardware thread context-switching support in modern OS. There is exactly a correspondence here, as when hardware thread context switches, its state are preserved via the pages, and you can only resume at the context-switch pt.

[lachlansneff]

I think that C2 is actually made up of two completely distinct options. Saving the stack into linear memory has a very high overhead, whereas "native" wasm coroutines would have a very small overhead.

I'm still not really understanding what C3 is. You can just call back into the wasm from a .then callback on a promise, is that what you mean?

[kripken]

@tqchen I've been thinking about a very related proposal with @RReverser (not for WASI specifically, but for wasm and specifically wasm/JS) - I'll post a design repo issue soon, just trying to polish the description a little more atm.

Our approach very much agrees with your point that there is a fundamental difference between coroutines and async support. Coroutines create stacks explicitly, while async/await can create a stack implicitly, which may have different performance characteristics. So even if coroutines could implement async support (which is likely) it might not be optimal. And it would certainly be less convenient and less explicit, both for users and for the VM.

I think async support in wasm doesn't need to freeze the entire VM, however. That might make sense in some environments (maybe WASI VMs off the web? I don't know), but on the Web at least I think we can pause just a specific wasm instance. That may end up requiring stack copying, but the flexibility may be worth it, and if you wait on say a fetch Promise then it won't matter.

As I said I'm writing this up in detail right now, and will post very soon. I'll be very curious to get your feedback on it @tqchen !

[tqchen]

More elaboration on C3.

Given that wasm itself is a stack machine, its stack must be presented somewhere, and depending on its implementation might be separated from the native OS calling stack as a data structure.

When calling into an async function, a wasm vm can just freeze its stack as well as other states(like a context-switch in a CPU). This will effectively allow resume the execution(context-switch back), the resume could be implemented via a callback(e.g. then in JS case but there can be other implementations in the native platforms). There is no additional action needed as the wasm vm's stack itself is still alive in memory. Of course, if the wasm vm uses the native callstack of the OS, then we will need to save the callstack(so that it can be resumed correctly).

Programming model

One potential aspect here is the programming model. Even after wasm gains native co-rountine support, the compilers would need to generate code that lowers to the co-routine primitives. This may create barriers good old applications (aka C++ and C drivers) that does not yet have good builtin co-rountine primitives. An sync-to-async interface, on the other hand, while has limitations, works out of the box, for most of the current programming models, and might be easier to implement(due to the restrictions) efficiently.

[lachlansneff]

Ah, well, that seems very much like an implementation detail to me. If i'm understanding that correctly, this would appear blocking to the wasm code, but not necessarily block the vm?

[tqchen]

@lachlansneff Yes, your understanding is correct(blocking to wasm, not blocking to vm).

It is certainly an implementation issue. In the meanwhile, it is also part of the interface specifications problem. As sync-to-async is not common concept in normal programming models. It would be harder to push blocking style APIs to WASI because of the need to run on the Web.

Having a guideline/standardization about how the execution environment could be implemented(and asynctified) would enable the same code run on both the Web and native VM.

The end goal is to have blocking-style API(that are asynctified) in WASI that will enable better applications such as GPUs and machine learning(e.g. we need that to bring Machine learning Wasm/WebGPU compiler to more wasm enabled platforms).

[lachlansneff]

This is an opinion, of course, but this seem to me like a repeat of the mistakes of system apis until very recently. Only now are we getting good ways of doing async io on linux. Starting WASI off with the same old, same old, seems shortsighted to me.

Since apis would be blocking to wasm, that means they'll need to create more and more threads, and that's the wrong direction, I think.

[tqchen]

I certainly share some of the opinions (e.g. asyncio is great overall when concurrency outweights other things :) On the other hand, it would be even better if we can have a constructive discussion about how these design decisions translate into solutions to the actual problems and their tradeoffs. Afterall our collective goal is to use these design decisions to solve problems and make wasm/wasi better for everyone.

The particular problem that motivate this thread is that machine learning applications that interacts with GPU and accelerators need synchronization with the device. In such cases, concurrency is not as important(e.g. you don't need to create additional threads for concurrent reuse of the resources like you mentioned), and you just need to block and wait for the compute on the GPU to complete.

Depending on the design decisions are two potential ways, :

• W0: Tell the application writers to rewrite the applications in co-rountines (e.g use async/await). This will restrict the potential langugage and features that people might use(e.g. it is hard to use C as a source language unless you structure the code as callbacks). Most people like to think about ML predictor as simple single threaded programs.
• W1: Support sync-to-async API calls, enable blocking view to the wasm. This would enable larger range of machine applications directly run out of the box, with not restriction on the programming model.

Only taking C2 will certainly entails W0, and C3 means we would enable W1. It certainly does not prevent users to make use of C2 if necessary, and as the system api itself is always async in nature, and it is only a question of whether we are willing to give the user the sync view as in W1.

Notably, it will also have implications in settings such as embedded system, where the simplicity that W1 offers might outweight the complexity bought by concurrency -- you could just implement the blocking API by blocking in such cases

[lachlansneff]

I agree that simplicity is important. But I'm not confident there isn't a way to still be simple to use while still trying to be performant.

[tqchen]

It boils down to the category of the problem we are trying to solve. As we know there is no silver bullet to solve all the problems.

For example, if we want to design system API to the serving http requests, asyncio is certainly better due to the need of concurrency in such cases, and we are certainly not trying to generalize to these cases (co-rountine should be used here).

In the particular category mentioned above, the case of machine learning and wait for GPU compute, where concurrency is less of a problem, synchronize API is both simple and performant (it is the API used by native CUDA/metal/opencl and a few others). In such cases, we will still need async fetch to load the model, however, because fetch is not the bottlebeck and only need to be done once, C2 and C3 does not matter due to Amdahl's law.

Would love to hear about more thoughts and examples about different problems(e.g. the case of http requests where C2 is certainly the right choice, and ML applications where C3 could be better), the tradeoffs and implications of different approaches in terms of the simplicity, performance, runtime size etc.

[tqchen]

Would be great to get more inputs from wasm vm communities.

Related background: we want to bring deep learning to wasm native(wasmer/wasmtime), just like what we did here for browsers, but generalize that to run on webgpu native . There are two items to that needs be resolved:

• webgpu native to be supported on these platforms(should not be too hard as there are already implementations such as wgpu/dawn).
• A common system call convention for GPU synchronization in the wasm vms, and JS version (why this discussion, although this discussion certainly generalizes to other cases).

@sunfishcode @MarkMcCaskey @syrusakbary Would it be something interesting to your community? and would love to see your thoughts about the current discussion.