js_interop changes in 3.6 (master) to JSTypedArray subtypes break passing SharedArrayBuffer as the underlying buffer.

[nullrocket]

A recent change adding constructors to the JSTypedArray subtypes requires a JSArrayBuffer as the buffer argument, however Javascript also allows passing a SharedArrayBuffer.

You can see the change to JSInt32Array here, however it affects all JSTypedArray subtypes.

sdk/sdk/lib/js_interop/js_interop.dart

Line 313 in c008c15

external JSInt32Array([JSArrayBuffer buffer, int byteOffset, int length]);

This situation around SharedArrayBuffers vs ArrayBuffers breaks occasionaly and I have had workarounds that I have used in the past after opening related issues, so I test it regularly now with the latest versions to make sure I have a heads up on future breaking changes.

I get that it is hard to test using the current dart test package so I created a repo that can reproduce the issue and test it using a fork of the test package specified in dependency_overrides that sets the correct headers to allow SharedArrayBuffer to work on browser tests.

Here is the repo with minimal tests if you want a reproduction. try it with dart 3.4 (stable) or 3.5 (beta) ( both pass) . dart 3.6 (master) fails.

https://github.com/nullrocket/js_typed_array_with_sab_and_atomics

[dart-github-bot]

Summary: The recent change adding constructors to JSTypedArray subtypes in Dart 3.6 requires a JSArrayBuffer as the buffer argument, breaking compatibility with SharedArrayBuffer, which is also valid in JavaScript. This issue causes failures when passing SharedArrayBuffer as the underlying buffer for JSTypedArray subtypes.

[mkustermann]

We had a discussion on the review about this. The reason against it was due to the fact that SharedArrayBuffer isn't always available. But maybe we should consider adding it anyway - it's also in the ECMA-262 specification. /cc @srujzs wdyt?

But it's somewhat easy to workaround as everyone can make their own JS interop extension types - also for builtin things like SharedArrayBuffer and typed arrays.

[nullrocket]

I'm not sure that its "easy" to work around with my own extension types, the representation types that JSTypedArray subtypes implement are internal, so to get the same toDart kind of casting like there is in the tests at the link I shared is a lot of brittle work that is going to break regularly with js_interop changes. It would be nice to get it right-ish once and for all,

SharedArrayBuffers, Atomics and TypedArrays are all basic js / web types that I use frequently, they definitely are available in all browsers and on both deno and node.

[srujzs]

Thanks for the repro! The original issue is tangential to that change that adds external constructors. That shouldn't affect any of the dart:typed_data types, and reverting it do not make the tests pass in that repo. e66a0cb is likely the root cause behind the failures in 3.6. It's possible that the change to make buffer polymorphic is the cause. cc @rakudrama

Your tests instantiate a typed array with a SharedArrayBuffer and call toDart, which just casts the result to the equivalent dart:typed_data type. When you call asInt32List and asUint8List on that type's buffer, it fails as it's not a NativeByteBuffer (an ArrayBuffer) or a ByteBuffer but rather a SharedArrayBuffer.

This was likely always somewhat broken. We intercept SharedArrayBuffers as a dart:html type that's unrelated to dart:typed_data, so a call to asInt32List shouldn't ever work as there should be no such method on that type. Before this change, it seemed to avoid the getInterceptor call and therefore this issue, but returning a SharedArrayBuffer where a ByteBuffer is expected is error-prone.

Some options to allow using dart:typed_data types with a SharedArrayBuffer:

• toDart could introduce a separate wrapper type if the underlying buffer is a SharedArrayBuffer. This might reintroduce the polymorphism that that change was avoiding, though, and I'm not really sure what this buys us.
• SharedArrayBuffer in dart:html could implement ByteBuffer and _nativeBuffer could return to returning a ByteBuffer instead of a NativeByteBuffer. It does make ByteBuffer polymorphic, but it looks like it already is.

---

Separate from this is the discussion that @mkustermann is referring to around how to declare the external constructors. I suspect that's not affecting your use case, and the workaround Martin is referring to (declaring your own extension types for SharedArrayBuffer and constructors) is something you're doing already in that repo.

[luigi-rosso]

Would it be possible to provide details around what the extension types work to support SharedArrayBuffer would be for cases like Uint8List.view from a ByteBuffer backed by a js SharedArrayBuffer? Some apparent magic that happens in creating the "asUint8List" methods is unclear from the js_interop source:

Uncaught TypeError: J.getInterceptor$x(...).asUint8List$2 is not a function
    at Object.asUint8List$2$x (main.dart.js?v=DEPLOYMENTVERSION:827:43)

Which occurs when trying to create a Uint8List.view on a ByteBuffer backed by a js SharedArrayBuffer:

Uint8List.view(buffer, buffer.offsetInBytes + readIndex, length);

[srujzs]

Yeah I believe you're coming across the same issue here where the interceptor type for a SharedArrayBuffer doesn't have an asUint8List method because it's just using the dart:html type as the interceptor. If you work entirely within JS interop types, then you can avoid such calls.

The equivalent JS type of Uint8List here would be JSUint8Array, so just use its constructor from dart:js_interop to view and call members on it: JSUint8Array(buffer.toJS, buffer.offsetInBytes + readIndex, length). I'd avoid using a ByteBuffer altogether and just write an extension type for the JSSharedArrayBuffer:

extension type JSSharedArrayBuffer._(JSObject _) implements JSObject {
  external int get byteLength;
  // Add the needed external members
}

// Needed as the constructor for `JSUint8Array` only accepts `JSArrayBuffer`.
@JS('Uint8Array')
extension type JSUint8ArrayWithSharedArrayBuffer._(JSUint8Array _) implements JSUint8Array {
  external JSUint8ArrayWithSharedArrayBuffer([JSArrayBuffer buffer, int byteOffset, int length]);
}

Of course, this doesn't stop someone from calling .toDart and then calling .buffer.asUint8List on it, which would again reproduce the original issue. You could have JSUint8ArrayWithSharedArrayBuffer avoid implementing JSUint8Array so one would need to cast to get that conversion, but the issue remains.

If you really need the Dart typed lists, the safest option is to write Dart class wrappers e.g.

class ByteBufferWithSharedArrayBuffer implements ByteBuffer {
  JSSharedArrayBuffer _arrayWithSharedBuffer;

  ByteBufferWithSharedArrayBuffer(this._arrayWithSharedBuffer);

  // implement the needed methods that forward to `_arrayWithSharedBuffer`'s external members
}

class Uint8ListWithSharedArrayBuffer implements Uint8List {
  ByteBufferWithSharedArrayBuffer _byteBufferWithSharedArrayBuffer;

  Uint8ListWithSharedArrayBuffer(this._byteBufferWithSharedArrayBuffer);

  // implement the needed methods
}

This is a bit of a pain, however, especially because you may need to write all the various views for asUint16List, asUint32List, etc. It may also be slower due to the polymorphic calls, but we may already pay that tax.

---

The general issue of using the .toJS and .toDart conversions is that these conversions just cast back and forth between the default type lists implementations in the JS compilers. The default typed lists implementations assume that the underlying value is a JS typed array with an ArrayBuffer backing it. This is not true when dealing with a SharedArrayBuffer so these implementations are ultimately not going to play nice there.

Hopefully this helps. It's a tricky, annoying problem to work around if you need to use the typed lists.

[luigi-rosso]

Thanks for this @srujzs! Our higher level code (beyond our web/ffi abstraction layer) performs operations like Uint8List.view.

// In our web abstraction layer...
// with extension type JSSharedArrayBuffer defined as you outlined

js.JSObject module = ...;
final heap = module['heap'] as js.JSFunction; // this returns a js SharedArrayBuffer
final sharedArrayBuffer =
    heap.callAsFunction(null, 0.toJS, 4.toJS) as JSSharedArrayBuffer;
final uint8 = JSUint8ArrayWithSharedArrayBuffer(shared).toDart;

// In our higher level "just dart" code we end up with a ref to uint8 and try to make a float32 view on it (we do this as sometimes we deal with serialized byte data which the high level runtime deals with as it can come from different sources, including wasm).
final view = Float32List.view(uint8.buffer, uint8.offsetInBytes, 1);

So the first option still won't work, unless I can somehow define the interceptor with the various asUint8List, asFloat32List, etc. Is that possible?

The second option sounds promising but I don't think I can implement ByteBuffer or Uint8List as they're declared final.

[srujzs]

So the first option still won't work, unless I can somehow define the interceptor with the various asUint8List, asFloat32List, etc. Is that possible?

One can modify the interceptor class in dart:html, but no there's no way for you to define your own interceptor here. Interceptors are inherently 1-1 and therefore can't be defined by users.

The second option sounds promising but I don't think I can implement ByteBuffer or Uint8List as they're declared final.

Right, sorry, and this was true even before they were marked final. I had forgotten that there are implicit exceptions for the Dart SDK to write our own wrapper class definitions like

sdk/sdk/lib/_internal/wasm/lib/js_typed_array.dart

Line 8 in f9c5944

final class JSArrayBufferImpl implements ByteBuffer {

.

---

Since you're forced to use the provided typed data implementations, I think this means that if you need a Dart typed data class to contain a SharedArrayBuffer, there's no recourse until the dart:html class implements ByteBuffer and the JS typed data implementations account for that. I'll want to discuss what this would mean in terms of performance. cc @rakudrama