Explainer: "MediaRecorder allows encoding a MediaStream that has audio and video tracks."

[guest271314]

Explainer presently states

MediaRecorder allows encoding a MediaStream that has audio and video tracks.

Technically, as currently specified MediaRecorder does not provide a means to record multiple video tracks within a single MediaStream

w3c/mediacapture-record#168

Recording multiple tracks was intended to be possible from day one. Many formats handle multiple tracks.

When it was pointed out that a lot of container formats couldn't handle increasing the number of channels mid-recording, we were left with two choices:

• Make the behavior dependent on container format (unpalatable)
• Make the behavior consistent, but not very useful (ie stop).
  The WG chose the latter.
  I am not aware of any change in the landscape of container formats that seems to indicate that varying the number of tracks is a generally available option. If you know of such changes, please provide references.

As for the "replace track" option - I don't think anyone thought about that possibility at the time.

• https://searchfox.org/mozilla-central/source/dom/media/MediaRecorder.cpp#778

• https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/mediarecorder/media_recorder_handler.cc?sq=package:chromium&g=0&l=260

The fact that MediaRecorder specification does not provide a means to record multiple video tracks is a problem that this proposal can resolve.

For example, it should be possible to write code similar to

const merged = await decodeVideoData(["video1.webm#t=5,10", "video2.mp4#t=10,15", "video3.ogv#t=0,5"], {codecs="openh264"});

[pthatcherg]

I think this is now covered by the explainer as a use case.