Converting YUV video frames to RGB is a often required for processing them in libraries like TensorFlow.js and OpenCV.js. Currently the only possible way to achive this is rendering the frame on a canvas.
let cnv = new OffscreenCanvas(frame.visibleRect.width, frame.visibleRect.height);
let ctx = cnv.getContext('2d');
ctx.drawImage(frame, 0, 0);
let imageData = ctx.getImageData(0, 0, frame.visibleRect.width, frame.visibleRect.height,
{colorSpace: colorSpace});
let buffer = imageData.data.buffer;This can only be done syncronously and requires extra resources (explicit canvas).
Add two new members in VideoFrameCopyToOptions dictionary:
formatpotential values are:RGBA,RGBX,BGRA,BGRX. This field specifies the pixel format for the pixel data incopyTo()destinationcolorSpacepotential values are:srgb,display-p3This field specifies the output color space for the pixel data incopyTo()destination
dictionary VideoFrameCopyToOptions {
VideoPixelFormat format;
PredefinedColorSpace colorSpace = "srgb";
DOMRectInit rect;
sequence<PlaneLayout> layout;
};
With format specified, VideoFrame.copyTo() will be equivalent to rendering on a canvas with a given color
space and calling getImageData() afterwards.
If developers need to save the video frames with BGR channel order, they can specify BGRA or BGRX pixel format
without having to write code for switching channels in the ImageData buffer.
const options = {
format: 'RGBA',
colorSpace: 'display-p3'
};
const bufSize = frame.allocationSize(options);
const buffer = new Uint8ClampedArray(size);
await frame.copyTo(buffer, options);
const image_data = new ImageData(buffer, frame.codedWidth, frame.codedHeight,
{ colorSpace: 'display-p3' });