export_room_keys fails on big database

[BillCarsonFr]

see element-hq/element-web#26681

=> Current code tries to extract all inbound session in a single session. In case of big accounts probably room

[BillCarsonFr]

see element-hq/element-web#26488 (comment)

[ara4n]

ftr importing large numbers of keys also fails: https://github.com/vector-im/element-web/issues/26692

[andybalaam]

I can reproduce this in my local dev environment, when I have a lot of keys. I see a failure inside:

.get_all()?.await?

On the IndexedDb object store.

The error message I see is:

11:33:38.615 Error exporting e2e keys: Error: DomException UnknownError (0): The operation failed for reasons unrelated to the database itself and not covered by any other error code.
    __wbindgen_error_new http://127.0.0.1:8080/bundles/_dev_/default-matrix-js-sdk_src_rust-crypto_index_ts.js:13432
rageshake.ts:77:27
    fnName rageshake.ts:77
    <anonymous> logger.ts:97
    startExport ExportE2eKeysDialog.tsx:125
    (Async: promise callback)
    startExport ExportE2eKeysDialog.tsx:124
    ExportE2eKeysDialog ExportE2eKeysDialog.tsx:104
    React 14
    unstable_runWithPriority scheduler.development.js:468
    React 15
    reRender Modal.tsx:409
    run setImmediate.js:40
    runIfPresent setImmediate.js:69
    onGlobalMessage setImmediate.js:109
    (Async: EventListener.handleEvent)
    installPostMessageImplementation setImmediate.js:114
    js setImmediate.js:169
    js setImmediate.js:186
    Webpack 8

[andybalaam]

Here is the stack of calls:

• ExportE2eKeysDialog.startExport (matrix-react-sdk)
  • client.exportRoomKeys (matrix-js-sdk) -> JS object
    • RustCrypto.exportRoomKeys (matrix-js-sdk) -> Array<IMegolmSessionData>
      • OlmMachine::export_room_keys (matrix-rust-sdk-crypto-wasm) -> String
        • OlmMachine::export_room_keys (matrix-rust-sdk/crypto) -> Vec<ExportedRoomKey>
          • IndexeddbCryptoStore::get_inbound_group_sessions (matrix-rust-sdk/indexeddb) -> Vec<InboundGroupSession>
          • for loop over InboundGroupSession::export -> Vec<ExportedRoomKey>
        • serde_json::to_string -> String
      • JSON.parse -> JS object
  • JSON.stringify -> JS string
  • MegolmExportEncryption.encryptMegolmKeyFile -> JS ArrayBuffer
    • TextEncoder.encode -> JS UInt8Array (utf-8)
    • subtlecrypto.encrypt -> JS ArrayBuffer
    • new UInt8Array -> JS UInt8Array
    • MegolmExportEncryption.packageMegolmKeyFile -> JS ArrayBuffer
  • new Blob -> JS Blob
  • FileSaver.saveAs

Everything after client.exportRoomKeys, including turning the full list of keys into JSON, then encrypting it as an ArrayBuffer (meaning we have many copies simultaneously) are present in the legacy code as well as the Rust, which works even for heavy users.

So in the Rust case we create a Vec<InboundGroupSession>, copy it into a Vec<ExportedRoomKey>, serialise it with serde into a String, and then parse it as a JS object before we do the above.

[andybalaam]

For future reference, we could base a fully-streaming solution on https://developer.mozilla.org/en-US/docs/Web/API/FileSystemWritableFileStream but it is not currently supported in Safari.

For now, we will focus on creating a Blob of the entire export in a more memory-efficient way.

[andybalaam]

I will consider MegolmExportEncryption outside of the scope of this task because it already exists and works with legacy crypto. But I do think it's worth noting that this could almost certainly be rewritten in streaming form, and doing so would significantly reduce memory usage during export.

[andybalaam]

The most obvious thing to fix first is the fact that the Rust converts the key backup to JSON, and the JS wrapper parses that to return it, only for it to be re-stringified immediately afterwards by the code shared between legacy and Rust crypto.