Linear memory, but the host sees it backwards

[SoniEx2]

This might be the wrong place for this, but:

Usually a wasm memory read looks like this:

fn read_u32(memory: &[u8], position: u32) -> Option<u32> {
  let b0 = *memory.get(position)?;
  let b1 = *memory.get(position + 1)?;
  let b2 = *memory.get(position + 2)?;
  let b3 = *memory.get(position + 3)?;
  // optimized out on little-endian
  Some(u32::from(b3) << 24 | u32::from(b2) << 16 | u32::from(b1) << 8 | u32::from(b0))
}

But why not do it like so instead:

fn read_u32(memory: &[u8], position: u32) -> Option<u32> {
  let b0 = *memory.get(memory.len() - position - 1)?;
  let b1 = *memory.get(memory.len() - position - 2)?;
  let b2 = *memory.get(memory.len() - position - 3)?;
  let b3 = *memory.get(memory.len() - position - 4)?;
  // optimized out on big-endian
  Some(u32::from(b3) << 24 | u32::from(b2) << 16 | u32::from(b1) << 8 | u32::from(b0))
}

It is fairly easy to prove that, as far as a wasm module is concerned, these are equivalent - a wasm module cannot observe any difference between the two of them.

But the host knows the difference.

While wasm modules don't care, the wasm-c-api currently does not support the latter form of linear memory representation. Would it be acceptable to propose that some wasm-c-api engines may use the latter form on big-endian platforms?

[rossberg]

Context: WebAssembly/wasm-c-api#180

[rossberg]

To clarify a little: basically this is asking Wasm implementations on big endian architectures to represent and address their memories backwards, so that numbers can be read off directly in native BE format when accessed through the C API, instead of using helpers, like e.g. the endian conversion types from the boost library.

Would it be acceptable to propose that some wasm-c-api engines may use the latter form on big-endian platforms?

That would break portability. If we decide that engines should be able to expose memory this way on BE architectures, then it would mean that all BE engines must indeed represent their memory this way.

[SoniEx2]

If we decide that engines should be able to expose memory this way on BE architectures, then it would mean that all BE engines must indeed represent their memory this way.

That would suck. Most C APIs use something like config.h for a reason.

You already need to recompile wasm-c-api consumers for BE anyway. Just recompile them twice, once for LE-on-BE, once for native-BE.

We could also deprecate it in the future if we find that one way is significantly more advantageous than the other.

[rossberg]

It is not just a matter of recompiling. If the memory is reversed, then all logic for accessing the memory must be reversed in the client, too, because all data structures in it -- arrays, structs, etc. -- are exposed in reverse. So every client of the API would have to implement two versions of all their memory access logic, perhaps even define compound types in two ways, etc.

Honestly, I think that even for clients, the disadvantage of having to do so would far outweigh the minor benefit they might get from avoiding endianness-agnostic number types when interfacing with Wasm.

[SoniEx2]

Not inherently. If wasm-c-api exposed integer read operations (instead of bulk read), all you'd do is read the individual fields with the "normal" offsets and the wasm-c-api would do the necessary conversions.

For example, if you wanna read a struct like

#[repr(C)]
struct Foo {
  a: u32,
  b: u32,
}

straight from wasm memory, you still need to do endianess conversions for each field. As such, it's much easier to simply read individual fields instead of trying (and failing) to bulk read the whole thing:

// works on both LE and BE
let a = read_u32(memory, offset);
let b = read_u32(memory, offset + 4);
let foo = Foo { a, b };

// but for some reason this is how everyone does it
let foo = read_bulk::<Foo>(memory, offset);
let foo = Foo { a: foo.a.to_le(), b: foo.b.to_le() };

And for something like wasm2c, you cannot get the C compiler to optimize away the le/be conversions as it sees the memory pointer as leaking, so you take a huge performance hit (around 50% slower on some platforms) on wasm memory operations. (this further inhibits other optimizations. but at least wasm2c doesn't support wasm-c-api we guess...)

(apologies for using rust syntax for all these things but we don't remember how to write C)

[rossberg]

Okay, I am no longer sure what you are proposing, to be honest.

From WebAssembly/wasm-c-api#180, I gathered that you are suggesting to globally reverse the entire memory inside the engine and compile all indexing negatively. But then any in-memory array or struct that is indexed in forward order inside Wasm will have to be indexed in reverse order when viewed from C.

Did I misunderstand and you are in fact suggesting to only reverse the local order of reads/writes of number types inside Wasm? Then that would clearly be in violation of Wasm's semantics. Wasm code can perform overlapping reads and writes, so any local change to byte order is a observable and a breaking change. Wasm specifies LE internally, we cannot change that.

Or are you suggesting to merely provide API helpers to reverse the bytes when reading memory from C? But then, how would that differ from functionality that something like boost already provides? Why should we duplicate that?

[SoniEx2]

globally reverse, don't compile negatively (it's mem[size-offset] not mem[-offset]), boost isn't C, C code shouldn't have to know how to read raw wasm memory in order to read data from it, because wasm can't read raw wasm memory either.

[rossberg]

If you globally reverse the memory, then the layout of all data structures is observed as reversed on the outside, as I said above. Imagine you have a struct type:

struct S { char a; double b; };

If you compile the following C++ code to Wasm,

S *f() {
  auto p = new S;
  p->a = 'a'; p->b = 3.14;
  return p;
}

then the host, when given an address meant to be a S*, would have to access it as if it had defined a reversed type

struct S { double b; char a; };

And this reverse type likely won't even work due to padding issues (let alone with objects or inheritance).

Similarly, the host would have to index all arrays with negative indices, which can be a serious issue when doing arithmetics with otherwise unsigned types like size_t.

So contrary to your assumption, host code would have to know even more in that model. For any non-trivial practical purposes, it would be way worse than just abstracting over the endianness of individual numeric types, I'd expect.

boost isn't C

Fair enough, it's C++. I don't know if there is a comparable generic library for raw C, though Linux provides an API, and GNU C has a similar library.

[SoniEx2]

Personally, we wouldn't even expect padding to be generally portable between host and wasm on LE platforms. So the host already has to know what it's doing.

What you need to abstract over is the wasm addresses. If you wanna provide LE-ordered bulk read between C and wasm it's perfectly reasonable to do that:

void wasm_bulk_read(int memory_id, u32 address, restrict void *target, size_t size);
void wasm_bulk_write(int memory_id, u32 address, restrict readonly void *source, size_t size);

tho it'll ofc be slower than just reading the memory directly while being aware of its BE-optimized layout.

We'll keep trying to hammer this for you: Host code needs to know the wasm layout because the wasm layout may differ from the host layout. You cannot simply memcpy out of raw wasm memory and assume it'll fit into a host struct. The ABI is not necessarily the same. The only proper way to do this kind of interop is to have a schema, and to use said schema. Structs are not a schema, because they are ABI-sensitive.

[rossberg]

Sure, this wasn't really meant to be taken as literal code, just to illustrate the problem with relative offsets and indexing.

Host code needs to know the wasm layout because the wasm layout may differ from the host layout.

Right, and that includes endianness. Any magic solution to this part of the problem is just gonna make the rest worse, as far as I can see. Any form of host-side indexing and relative offset calculation would have to be implemented in two versions in such an approach.

Furthermore, there are platforms with mixed endianness. For example, on some ARMs, the words of floats are BE, but the bytes per word are LE. Global memory reversal tricks simply won't fly in such cases.

[SoniEx2]

So maybe the host shouldn't be able to see linear memory at all, instead being forced to go through wasm to interact with it! (interface types/component model would possibly help with that?)

[rossberg]

For clients who want to avoid the hassle of doing endian conversion host-side, going through Wasm could be a reasonable alternative. But the JS API already exposes the memory in raw form for those that want or need it. For the C API, I don't think that falling back behind that is an option. (And the JS API of course requires that this view is LE already.)

Honestly, I think this ship has long sailed. In the early days of Wasm, endianness was of course discussed, and for better or worse, the group at the time decided to go all in on LE for simplicity's sake. The dominating view was that it wasn't worth making amends to the few remaining BE architectures without efficient byteswap instructions.

[SoniEx2]

As far as wasm is concerned, you do have LE, and exposing it as LE to javascript is just as trivial as exposing it as LE to wasm.

Because those are VMs, while the host is not.

So if you're gonna do endian conversions either way, why not... do another form of endian conversions? One that is far more efficient for the average use-case, and only applies to relatively obscure platforms anyway.

[SoniEx2]

There is one case where this proposed ABI has a clear advantage: atomics. Since this proposed ABI changes merely the way address calculation happens, but otherwise accesses memory following the host endianness, atomic accesses can rely on all hardware atomics, instead of having to emulate everything with CAS/CMPXCHG.

Further, after attempting to reimplement s390x support in wabt without this proposed ABI, we can say we'd rather just give up. The added complexity of doing it "properly" really isn't worth it.