Djuffin

Allow converting VideoFrame to RGB pixel formats by calling VideoFrame.copyTo()

Problem and Motivation

Converting YUV video frames to RGB is a often required for processing them in libraries like TensorFlow.js and OpenCV.js. Currently the only possible way to achive this is rendering the frame on a canvas.

let cnv = new OffscreenCanvas(frame.visibleRect.width, frame.visibleRect.height);
let ctx = cnv.getContext('2d');
ctx.drawImage(frame, 0, 0);

Djuffin

VideoEncoderConfig.contentHint

Problem and Motivation

Webcam content often require denoising and is often intelligible even when downscaled or with high quantization levels. Screencast content of presentations or webpages with a lot of text content is completely unintelligible if the quantization levels are too high or if the content is downscaled or otherwise blurry.

Video encoding libraries (like libvpx, OpenH264) provide a way for API users to inform the encoder about type of the encoded content.

Djuffin

Example

const options = { 
  format: 'RGBA',
  colorSpace: 'display-p3'
};
const bufSize = frame.allocationSize(options);

const buffer = new Uint8ClampedArray(size);
await frame.copyTo(buffer, options);

Djuffin

Allow transferring ArrayBuffer into VideoFrame, AudioData, EncodedVideoChunk, EncodedAudioChunk, ImageDecoder constructors

Problem and Motivation

Currently when creating any of the VideoFrame, AudioData, EncodedVideoChunk, EncodedAudioChunk, ImageDecoder objects from corresponding *Init objects constructors make a copy of ArrayBuffer contents into an internal memory buffer.

In many cases the ArrayBuffer is never used again for anything else after a WebCodecs object is created. It means that this copy can be avoided by transfering (moving) ArrayBuffer contents into the WebCodecs objects.

Remember, memcpy is murder.

Proposed solutions

Djuffin

Per-frame quantizer in VideoEncoder

Problem and Motivation

Currently there are two rate control modes for video encoding in WebCodecs:

1. Constant bitrate mode
2. Variable bitrate mode

The bitrate is set via VideoEncoderConfig and can be periodically changed. But there is demand for controlling the quality and the size of each of the encoded frames for following reasons:

1. A website might know extra information about the video frames.