fix padding image

ARKit-ARFrame-Opencv.mm

// The Y plane represents the luminance component, and the UV plane represents the Cb and Cr chroma components.
// In the case of kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format, you will find the luma plane is 8bpp with the  same dimensions as your video, your chroma plane will be 16bpp, but only a quarter of the size of the original video. You will have one Cb and one Cr component per pixel on this plane.
// so if your input video is 352x288, your Y plane will be 352x288 8bpp, and your CbCr 176x144 16bpp. This works out to be about the same amount of data as a 12bpp 352x288 image, half what would be required for RGB888 and still less than RGB565.
// So in the buffer, Y will look like this [YYYYY . . . ] and UV [UVUVUVUVUV . . .]
// vs RGB being, of course, [RGBRGBRGB . . . ]
// https://stackoverflow.com/questions/13429456/how-seperate-y-planar-u-planar-and-uv-planar-from-yuv-bi-planar-in-ios
#pragma ARSessionDelegate
- (void)session:(ARSession *)session didUpdateFrame:(ARFrame *)frame{
    CVPixelBufferRef buffer=frame.capturedImage;
    CVPixelBufferLockBaseAddress(buffer, 0);

    void *address =  CVPixelBufferGetBaseAddressOfPlane(buffer, 0);
    int bufferWidth = (int)CVPixelBufferGetWidthOfPlane(buffer,0);
    int bufferHeight = (int)CVPixelBufferGetHeightOfPlane(buffer, 0);
    int bytePerRow = (int)CVPixelBufferGetBytesPerRowOfPlane(buffer, 0);
    //Get the pixel format
    OSType pixelFormat = CVPixelBufferGetPixelFormatType(buffer);  

    if (pixelFormat == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {

      //Get the yPlane (Luma values)
      cv::Mat yPlane = cv::Mat(bufferHeight, bufferWidth, CV_8UC1, address, bytePerRow);

      //Get cbcrPlane (Chroma values)
      int cbcrWidth = (int)CVPixelBufferGetWidthOfPlane(buffer,1);
      int cbcrHeight = (int)CVPixelBufferGetHeightOfPlane(buffer, 1);
      void *cbcrAddress = CVPixelBufferGetBaseAddressOfPlane(buffer, 1);

      //Since the CbCr Values are alternating we have 2 channels: Cb and Cr. Thus we need to use CV_8UC2 here.
      cv::Mat cbcrPlane = cv::Mat(cbcrHeight, cbcrWidth, CV_8UC2, cbcrAddress);

      const CvScalar color = ycbcrFromRGB(255, 0, 0);

      cv::circle(yPlane, cv::Point(1000,1000), 100, color, cv::FILLED);
      cv::circle(cbcrPlane, cv::Point(500,500), 50, color, cv::FILLED);
    }

    CVPixelBufferUnlockBaseAddress(buffer, 0);
}

cv::Scalar ycbcrFromRGB(float red,
                     float green,
                     float blue)
{
  // https://developer.apple.com/documentation/arkit/arframe/2867984-capturedimage?language=objc
  // [R, G, B] x ycbcrToRGBTransform
  return cv::Scalar(0.0000 * red - 0.3441 * green + 1.7720 * blue,
                    1.4020 * red - 0.7141 * green + 0.0000 * blue,
                    -0.7010 * red + 0.5291 * green - 0.8860 * blue);
}

Nice ! thx for sharing this ! it helps me !
But don't forget the bytePerRow in your Yplane mat. In case of padding your image will look bad.
cv::Mat yPlane = cv::Mat(bufferHeight, bufferWidth, CV_8UC1, address, bytePerRow);

Nice. Updated my code.

How would you run algorithms like cv::Canny on the Y and CbCr mats?

Hi, just ran across your post as I was looking for some ARKit + YUV references. One thing I noticed is the conversion matrix you're using converts YUV to RGB, but you're using it to convert to YUV. Shouldn't you take the inverse of the Apple matrix? I would imagine M_rgb2yuv = inv(M_yuv2rgb).

Thanks!