Skip to content

Instantly share code, notes, and snippets.

@Alby-o
Last active March 28, 2024 12:50
Show Gist options
  • Star 39 You must be signed in to star a gist
  • Fork 11 You must be signed in to fork a gist
  • Save Alby-o/fe87e35bc21d534c8220aed7df028e03 to your computer and use it in GitHub Desktop.
Save Alby-o/fe87e35bc21d534c8220aed7df028e03 to your computer and use it in GitHub Desktop.
// imgLib -> Image package from https://pub.dartlang.org/packages/image
import 'package:image/image.dart' as imglib;
import 'package:camera/camera.dart';
Future<List<int>> convertImagetoPng(CameraImage image) async {
try {
imglib.Image img;
if (image.format.group == ImageFormatGroup.yuv420) {
img = _convertYUV420(image);
} else if (image.format.group == ImageFormatGroup.bgra8888) {
img = _convertBGRA8888(image);
}
imglib.PngEncoder pngEncoder = new imglib.PngEncoder();
// Convert to png
List<int> png = pngEncoder.encodeImage(img);
return png;
} catch (e) {
print(">>>>>>>>>>>> ERROR:" + e.toString());
}
return null;
}
// CameraImage BGRA8888 -> PNG
// Color
imglib.Image _convertBGRA8888(CameraImage image) {
return imglib.Image.fromBytes(
image.width,
image.height,
image.planes[0].bytes,
format: imglib.Format.bgra,
);
}
// CameraImage YUV420_888 -> PNG -> Image (compresion:0, filter: none)
// Black
imglib.Image _convertYUV420(CameraImage image) {
var img = imglib.Image(image.width, image.height); // Create Image buffer
Plane plane = image.planes[0];
const int shift = (0xFF << 24);
// Fill image buffer with plane[0] from YUV420_888
for (int x = 0; x < image.width; x++) {
for (int planeOffset = 0;
planeOffset < image.height * image.width;
planeOffset += image.width) {
final pixelColor = plane.bytes[planeOffset + x];
// color: 0x FF FF FF FF
// A B G R
// Calculate pixel color
var newVal = shift | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
img.data[planeOffset + x] = newVal;
}
}
return img;
}
@GanZhiXiong
Copy link

@alexcohn Thanks, But converImagetoPng is not very fast, can you still shorten the time to get the image?

@GanZhiXiong
Copy link

@alexcohn
CameraImage to png slow. Do you know the reason for the slowness?

@alexcohn
Copy link

No, I don't have a faster converter for you. You probably need some code in C++ to make this reasonably fast.

@ugu11
Copy link

ugu11 commented Apr 23, 2020

@Hugand the reason the result may be distorted, is that the conversions here don't take into account plane.bytesPerRow. For many CameraImages, bytesPerRow will be equal to width, but some will have some padding, e.g. flutter/flutter#26348 (comment).

Oh! That makes sense then. Thanks man!

@ugu11
Copy link

ugu11 commented Apr 23, 2020

No, I don't have a faster converter for you. You probably need some code in C++ to make this reasonably fast.

Do you think it would be substantially faster?

@alexcohn
Copy link

Do you think it would be substantially faster

I saw complaints that in flutter it can take more than a second. But any Android device can record video at 30 fps.

@ugu11
Copy link

ugu11 commented Apr 23, 2020

Well, I'll try to implement it in C and I'll see how it performs then

@ugu11
Copy link

ugu11 commented Apr 25, 2020

Update: I was able to reduce the the conversion time to less than a second for a CameraImage with ResolutionPreset.high with some C code. I'll write an article/tutorial in the next few days and as soon as it is ready I'll share it with you @alexcohn @GanZhiXiong

@GanZhiXiong
Copy link

@Hugand
Thanks, cool. 👍👍👍
Waiting for your good news.

@ugu11
Copy link

ugu11 commented Apr 28, 2020

Alright @GanZhiXiong, the tutorial is now published and you can check it out. I really hope it helps. Any question you have feel free to contact me :)

https://medium.com/@hugand/capture-photos-from-camera-using-image-stream-with-flutter-e9af94bc2bee

@GanZhiXiong
Copy link

@Hugand

very cool,👍👍👍, First of all, thank you very much for the article you wrote and the open source code, which gave me a lot of help, thank you very much!

Your method is much faster than my previous method
I added Stopwatch to your code. The following is the code I modified.

{
          print('''

tap FloatingActionButton get img
''');
          Stopwatch stopwatch = Stopwatch();
          stopwatch.start();

          // Allocate memory for the 3 planes of the image
          Pointer<Uint8> p = allocate(count: _savedImage.planes[0].bytes.length);
          Pointer<Uint8> p1 = allocate(count: _savedImage.planes[1].bytes.length);
          Pointer<Uint8> p2 = allocate(count: _savedImage.planes[2].bytes.length);

          // Assign the planes data to the pointers of the image
          Uint8List pointerList = p.asTypedList(_savedImage.planes[0].bytes.length);
          Uint8List pointerList1 = p1.asTypedList(_savedImage.planes[1].bytes.length);
          Uint8List pointerList2 = p2.asTypedList(_savedImage.planes[2].bytes.length);
          pointerList.setRange(0, _savedImage.planes[0].bytes.length, _savedImage.planes[0].bytes);
          pointerList1.setRange(0, _savedImage.planes[1].bytes.length, _savedImage.planes[1].bytes);
          pointerList2.setRange(0, _savedImage.planes[2].bytes.length, _savedImage.planes[2].bytes);

          // Call the convertImage function and convert the YUV to RGB
          Pointer<Uint32> imgP = conv(p, p1, p2, _savedImage.planes[1].bytesPerRow, _savedImage.planes[1].bytesPerPixel,
              _savedImage.width, _savedImage.height);

          // Get the pointer of the data returned from the function to a List
          List imgData = imgP.asTypedList((_savedImage.width * _savedImage.height));

          stopwatch.stop();
          int getImgDataTime = stopwatch.elapsedMilliseconds;

          stopwatch.reset();
          stopwatch.start();

          // Generate image from the converted data
          imglib.Image img = imglib.Image.fromBytes(_savedImage.height, _savedImage.width, imgData);

          // Free the memory space allocated
          // from the planes and the converted data
          free(p);
          free(p1);
          free(p2);
          free(imgP);

          stopwatch.stop();
          print('get imgData: ${getImgDataTime}ms, imglib.Image.fromBytes: ${stopwatch.elapsedMilliseconds}ms');

          // The jump has animation, and all the pushing that you see is not going very fast
//          Navigator.push(context, MaterialPageRoute(builder:
//            (context) => new ImagePreview(img: img)));
        }

The following is my test results under different ResolutionPreset

CameraController use ResolutionPreset.medium
04-28 11:29:31.622 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:31.778 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 150ms, imglib.Image.fromBytes: 4ms
04-28 11:29:33.472 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:33.619 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 143ms, imglib.Image.fromBytes: 3ms
04-28 11:29:34.757 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:34.909 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 148ms, imglib.Image.fromBytes: 3ms
04-28 11:29:36.406 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:36.555 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 145ms, imglib.Image.fromBytes: 3ms
04-28 11:29:38.841 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:38.998 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 145ms, imglib.Image.fromBytes: 10ms
04-28 11:29:40.387 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:40.530 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 139ms, imglib.Image.fromBytes: 3ms
04-28 11:29:42.045 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:42.189 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 139ms, imglib.Image.fromBytes: 3ms
04-28 11:29:43.389 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:43.595 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 151ms, imglib.Image.fromBytes: 54ms
04-28 11:29:44.874 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:45.038 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 157ms, imglib.Image.fromBytes: 6ms
04-28 11:29:46.187 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:46.344 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 151ms, imglib.Image.fromBytes: 5ms
04-28 11:29:47.715 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:47.864 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 144ms, imglib.Image.fromBytes: 3ms
04-28 11:29:49.231 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:49.384 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 149ms, imglib.Image.fromBytes: 3ms
04-28 11:29:50.753 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:50.908 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 150ms, imglib.Image.fromBytes: 3ms
04-28 11:29:52.235 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:52.402 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 161ms, imglib.Image.fromBytes: 5ms
04-28 11:29:53.560 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:53.708 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 143ms, imglib.Image.fromBytes: 4ms
04-28 11:29:54.755 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:54.900 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 139ms, imglib.Image.fromBytes: 5ms
04-28 11:29:56.265 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:56.425 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 148ms, imglib.Image.fromBytes: 11ms
04-28 11:29:57.834 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:57.988 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 146ms, imglib.Image.fromBytes: 6ms
04-28 11:29:59.423 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:29:59.573 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 146ms, imglib.Image.fromBytes: 3ms
04-28 11:30:01.993 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:30:02.150 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 153ms, imglib.Image.fromBytes: 2ms
04-28 11:30:03.522 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:30:03.670 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 145ms, imglib.Image.fromBytes: 2ms

CameraController use ResolutionPreset.veryHigh
04-28 11:42:55.601 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:42:56.210 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 545ms, imglib.Image.fromBytes: 56ms
04-28 11:42:58.270 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:42:58.790 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 482ms, imglib.Image.fromBytes: 36ms
04-28 11:42:59.945 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:00.429 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 470ms, imglib.Image.fromBytes: 12ms
04-28 11:43:01.251 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:01.775 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 508ms, imglib.Image.fromBytes: 15ms
04-28 11:43:02.704 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:03.231 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 511ms, imglib.Image.fromBytes: 15ms
04-28 11:43:04.227 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:04.771 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 531ms, imglib.Image.fromBytes: 12ms
04-28 11:43:05.904 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:06.414 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 493ms, imglib.Image.fromBytes: 16ms
04-28 11:43:07.095 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:07.613 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 483ms, imglib.Image.fromBytes: 33ms
04-28 11:43:08.456 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:08.966 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 497ms, imglib.Image.fromBytes: 12ms
04-28 11:43:09.835 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:10.325 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 477ms, imglib.Image.fromBytes: 12ms
04-28 11:43:11.059 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:11.644 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 563ms, imglib.Image.fromBytes: 21ms
04-28 11:43:12.396 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:12.890 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 477ms, imglib.Image.fromBytes: 16ms
04-28 11:43:13.704 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:14.288 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 568ms, imglib.Image.fromBytes: 15ms
04-28 11:43:15.353 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:15.915 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 546ms, imglib.Image.fromBytes: 14ms
04-28 11:43:16.735 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:17.249 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 499ms, imglib.Image.fromBytes: 15ms
04-28 11:43:18.419 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:19.033 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 579ms, imglib.Image.fromBytes: 33ms
04-28 11:43:20.265 10030-10134/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:43:20.791 10030-10134/com.example.camera_tutorial I/flutter: get imgData: 510ms, imglib.Image.fromBytes: 15ms

CameraController use ResolutionPreset.max
04-28 11:34:23.079 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:23.609 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 483ms, imglib.Image.fromBytes: 42ms
04-28 11:34:27.376 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:27.952 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 549ms, imglib.Image.fromBytes: 26ms
04-28 11:34:30.296 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:30.802 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 491ms, imglib.Image.fromBytes: 14ms
04-28 11:34:33.768 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:34.309 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 526ms, imglib.Image.fromBytes: 13ms
04-28 11:34:35.120 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:35.691 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 559ms, imglib.Image.fromBytes: 11ms
04-28 11:34:36.777 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:37.257 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 470ms, imglib.Image.fromBytes: 9ms
04-28 11:34:38.217 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:38.765 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 536ms, imglib.Image.fromBytes: 10ms
04-28 11:34:39.591 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:40.112 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 493ms, imglib.Image.fromBytes: 27ms
04-28 11:34:41.130 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:41.623 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 481ms, imglib.Image.fromBytes: 11ms
04-28 11:34:42.274 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:42.791 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 504ms, imglib.Image.fromBytes: 11ms
04-28 11:34:43.312 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:43.836 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 513ms, imglib.Image.fromBytes: 10ms
04-28 11:34:44.439 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:45.103 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 569ms, imglib.Image.fromBytes: 94ms
04-28 11:34:45.584 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:46.070 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 470ms, imglib.Image.fromBytes: 14ms
04-28 11:34:46.775 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:47.337 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 544ms, imglib.Image.fromBytes: 17ms
04-28 11:34:47.910 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:48.465 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 493ms, imglib.Image.fromBytes: 61ms
04-28 11:34:49.228 3347-3465/com.example.camera_tutorial I/flutter: tap FloatingActionButton get img
04-28 11:34:49.809 3347-3465/com.example.camera_tutorial I/flutter: get imgData: 565ms, imglib.Image.fromBytes: 15ms

@GanZhiXiong
Copy link

@Hugand
Can't ffi be used on the flutter stable branch?
I wanted to use it in a production environment, so I didn't want to switch to flutter channel dev

@khal-it
Copy link

khal-it commented Dec 14, 2020

Hey I just implemented a plugin that converts the CameraImage to JPEG. This could be easily adjusted to support PNG.
https://pub.dev/packages/camera_image_converter/versions/0.0.2. Its pretty fast too.

@hxtruong6
Copy link

image
I use the same your way, but I got the image containing nothing?

Now CameraImage also supports jpeg, so I do the same in your code, just add a new type of converting.

imglib.Image? _convertJpeg(CameraImage image) {
  return imglib.Image.fromBytes(
      image.width, image.height, image.planes[0].bytes,
      format: imglib.Format.rgb, channels: imglib.Channels.rgb);
}

@Ammobag
Copy link

Ammobag commented Jul 16, 2021

image
I use the same your way, but I got the image containing nothing?

Now CameraImage also supports jpeg, so I do the same in your code, just add a new type of converting.

imglib.Image? _convertJpeg(CameraImage image) {
  return imglib.Image.fromBytes(
      image.width, image.height, image.planes[0].bytes,
      format: imglib.Format.rgb, channels: imglib.Channels.rgb);
}

thnx it helped a great deal

@SelanDeemantha
Copy link

@GanZhiXiong,
Hi, I faced the same issue with using latest ffi: ^1.1.2 version on my code for image conversion, can you please share your solution here.

@SebghatYusuf
Copy link

SebghatYusuf commented Sep 23, 2021

@Hugand Trying to use your C converter implementation, for android it's running perfectly fine, but when I'm running on iOS, I get below error

   /Users/Username/Library/Developer/Xcode/DerivedData/Runner-bapuesqdvyewspdpyvpssxebolee/Build/Intermediates.noindex/Runner.build/Debug-iphonesimulator/Runner.build/Objects-normal/x86_64/custom_image_converter.o
   /Users/Username/Library/Developer/Xcode/DerivedData/Runner-bapuesqdvyewspdpyvpssxebolee/Build/Intermediates.noindex/Runner.build/Debug-iphonesimulator/Runner.build/Objects-normal/x86_64/AppDelegate.o
  ld: 1 duplicate symbol for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    note: Using new build system
    note: Building targets in parallel
    note: Planning build
    note: Analyzing workspace
    note: Constructing build description
    note: Build preparation complete
Could not build the application for the simulator.
Error launching application on iPhone 12 Pro.


@sikandernoori
Copy link

Could not build the application for the simulator.

The Error is Simulator related, not pertaining to C Code.

@SebghatYusuf
Copy link

@sikandernoori Actually I've resolved the issue, and it was not simulator related issue, it was code related issue.
The issue was in with the main function in C script.

@yh4922
Copy link

yh4922 commented Dec 8, 2021

The output is a grayscale image. How do I get a color image?

@sikandernoori
Copy link

@yh4922 can you provide reproduce able code segment ?

@yh4922
Copy link

yh4922 commented Dec 9, 2021

imglib.Image _convertYUV420(CameraImage image) {
  var img = imglib.Image(image.width, image.height); // Create Image buffer

  final int width = image.width;
  final int height = image.height;
  final int uvRowStride = image.planes[1].bytesPerRow;
  final int uvPixelStride = image.planes[1].bytesPerPixel;
  const shift = (0xFF << 24);

  for(int x=0; x < width; x++) {
    for(int y=0; y < height; y++) {
      final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
      final int index = y * width + x;

      final yp = image.planes[0].bytes[index];
      final up = image.planes[1].bytes[uvIndex];
      final vp = image.planes[2].bytes[uvIndex];
      // Calculate pixel color
      int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
      int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
      int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);     
      // color: 0x FF  FF  FF  FF 
      //           A   B   G   R
      img.data[index] = shift | (b << 16) | (g << 8) | r;
    }
  }

  return img;
}

@sikandernoori this way, it can be converted into color images, but it takes more than 1000ms to convert on a mobile phone with Snapdragon 870 CPU, and it will block the UI.

@alexcohn
Copy link

alexcohn commented Dec 9, 2021

@yh4922 this is the time it takes dart to convert a YUV 420 image (which comes from Android camera) to RGB. Performance could be much better if you do it in C, possibly enabling NEON or GPU. You can do this conversion in OpenCV, it's nicely optimized.

@sikandernoori
Copy link

@yh4922 solution proposed by @alexcohn is a good option.
But if you want the simpler one than I would suggest use isolates to convert image ...

Send CameraImage to isolate and convert image within isolate and use ...

@alexcohn
Copy link

@ramsmart-inno if you are using OpenCV anyway, this image conversion does not cost you APK size

@luomo-pro
Copy link

Thanks for your answers, I see many kinds of solutions.
I now want to convert yuv420 images to color, can you tell me which solution can do it?
I've seen many that only convert to black and white, and that's not what I want.
In addition, the camera's imageFormatGroup parameter is set to jpeg, you can easily convert the color image, but I found that it will cause the preview screen lag, the experience is very bad.
So I may only be able to convert yuv420.
Thank you very much!

@krzaklus
Copy link

krzaklus commented Feb 22, 2022

About a few days of struggling with the CamerImage to Image conversion, I managed to improve the method to include paddnig on different devices. I tested on several devices, checking the conversion at different camera resolutions. And I think it works.

imglib.Image convertYUV420ToImage(CameraImage cameraImage) {
    final imageWidth = cameraImage.width;
    final imageHeight = cameraImage.height;

    final yBuffer = cameraImage.planes[0].bytes;
    final uBuffer = cameraImage.planes[1].bytes;
    final vBuffer = cameraImage.planes[2].bytes;

    final int yRowStride = cameraImage.planes[0].bytesPerRow;
    final int yPixelStride = cameraImage.planes[0].bytesPerPixel!;

    final int uvRowStride = cameraImage.planes[1].bytesPerRow;
    final int uvPixelStride = cameraImage.planes[1].bytesPerPixel!;

    final image = imglib.Image(imageWidth, imageHeight);

    for (int h = 0; h < imageHeight; h++) {
      int uvh = (h / 2).floor();

      for (int w = 0; w < imageWidth; w++) {
        int uvw = (w / 2).floor();

        final yIndex = (h * yRowStride) + (w * yPixelStride);

        // Y plane should have positive values belonging to [0...255]
        final int y = yBuffer[yIndex];

        // U/V Values are subsampled i.e. each pixel in U/V chanel in a
        // YUV_420 image act as chroma value for 4 neighbouring pixels
        final int uvIndex = (uvh * uvRowStride) + (uvw * uvPixelStride);

        // U/V values ideally fall under [-0.5, 0.5] range. To fit them into
        // [0, 255] range they are scaled up and centered to 128.
        // Operation below brings U/V values to [-128, 127].
        final int u = uBuffer[uvIndex];
        final int v = vBuffer[uvIndex];

        // Compute RGB values per formula above.
        int r = (y + v * 1436 / 1024 - 179).round();
        int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
        int b = (y + u * 1814 / 1024 - 227).round();

        r = r.clamp(0, 255);
        g = g.clamp(0, 255);
        b = b.clamp(0, 255);

        // Use 255 for alpha value, no transparency. ARGB values are
        // positioned in each byte of a single 4 byte integer
        // [AAAAAAAARRRRRRRRGGGGGGGGBBBBBBBB]
        final int argbIndex = h * imageWidth + w;

        image.data[argbIndex] = 0xff000000 |
            ((b << 16) & 0xff0000) |
            ((g << 8) & 0xff00) |
            (r & 0xff);
      }
    }

    return image;
  }

@neoacevedo
Copy link

This code has some deprecated classes for Flutter 3.7.1:
imglib.Image _convertBGRA8888(CameraImage image) has to be rewritten because of imglib.Image.fromBytes doesn't accept direct parameters and the Format enum doesn't have the bgra value.

@Holofox
Copy link

Holofox commented Feb 17, 2023

@neoacevedo,

imglib.Image _convertBGRA8888(CameraImage image) {
  return imglib.Image.fromBytes(
    image.width,
    image.height,
    image.planes[0].bytes,
    order: ChannelOrder.bgra,
  );
}

@rraayy
Copy link

rraayy commented Mar 15, 2023

Hi, 'img.data[index]' seems no longer use, how to modify the code when using convertYUV420ToImage?

@juanlabrador
Copy link

Hi, 'img.data[index]' seems no longer use, how to modify the code when using convertYUV420ToImage?

I have the same problem

@rraayy
Copy link

rraayy commented Mar 17, 2023

Hi, 'img.data[index]' seems no longer use, how to modify the code when using convertYUV420ToImage?

I have the same problem

Hi juanlabrador

I found a way and it's work well. Replace 'img.data[index]' to 'img.setPixelRgba(x, y, r, g, b, hexFF);'

@Serdar1112
Copy link

Hi, 'img.data[index]' seems no longer use, how to modify the code when using convertYUV420ToImage?

I have the same problem

Hi juanlabrador

I found a way and it's work well. Replace 'img.data[index]' to 'img.setPixelRgba(x, y, r, g, b, hexFF);'

And what should go into each value?

@federico-amura-kenility
Copy link

@neoacevedo,

imglib.Image _convertBGRA8888(CameraImage image) {
  return imglib.Image.fromBytes(
    image.width,
    image.height,
    image.planes[0].bytes,
    order: ChannelOrder.bgra,
  );
}

the image result is all broken.

@krzaklus
Copy link

krzaklus commented May 1, 2023

@neoacevedo I used this function.

  static imglib.Image convertBGRA8888ToImage(CameraImage cameraImage) {
    return imglib.Image.fromBytes(
      width: cameraImage.planes[0].width!,
      height: cameraImage.planes[0].height!,
      bytes: cameraImage.planes[0].bytes.buffer,
      order: imglib.ChannelOrder.bgra,
    );
  }

@federico-amura-kenility
Copy link

federico-amura-kenility commented May 1, 2023

I used this function.

static imglib.Image convertBGRA8888ToImage(CameraImage cameraImage) { return imglib.Image.fromBytes( width: cameraImage.planes[0].width!, height: cameraImage.planes[0].height!, bytes: cameraImage.planes[0].bytes.buffer, order: imglib.ChannelOrder.bgra, ); }

its working for iOS, but not on Android.
What imageFormatGroup did you use to create the Camera Controller?

@krzaklus
Copy link

krzaklus commented May 1, 2023

federico-amura-kenility I think on Android you should use convertYUV420ToImage. I modify my class after update Image lib, please try this code.

part of object_detection;

/// ImageUtils
class ImageUtils {
  ///
  /// Converts a [CameraImage] in YUV420 format to [image_lib.Image] in RGB format
  ///
  static imglib.Image convertCameraImage(CameraImage cameraImage) {
    if (cameraImage.format.group == ImageFormatGroup.yuv420) {
      return convertYUV420ToImage(cameraImage);
    } else if (cameraImage.format.group == ImageFormatGroup.bgra8888) {
      return convertBGRA8888ToImage(cameraImage);
    } else {
      throw Exception('Undefined image type.');
    }
  }

  ///
  /// Converts a [CameraImage] in BGRA888 format to [image_lib.Image] in RGB format
  ///
  static imglib.Image convertBGRA8888ToImage(CameraImage cameraImage) {
    return imglib.Image.fromBytes(
      width: cameraImage.planes[0].width!,
      height: cameraImage.planes[0].height!,
      bytes: cameraImage.planes[0].bytes.buffer,
      order: imglib.ChannelOrder.bgra,
    );
  }

  ///
  /// Converts a [CameraImage] in YUV420 format to [image_lib.Image] in RGB format
  ///
  static imglib.Image convertYUV420ToImage(CameraImage cameraImage) {
    final imageWidth = cameraImage.width;
    final imageHeight = cameraImage.height;

    final yBuffer = cameraImage.planes[0].bytes;
    final uBuffer = cameraImage.planes[1].bytes;
    final vBuffer = cameraImage.planes[2].bytes;

    final int yRowStride = cameraImage.planes[0].bytesPerRow;
    final int yPixelStride = cameraImage.planes[0].bytesPerPixel!;

    final int uvRowStride = cameraImage.planes[1].bytesPerRow;
    final int uvPixelStride = cameraImage.planes[1].bytesPerPixel!;

    final image = imglib.Image(width: imageWidth, height: imageHeight);

    for (int h = 0; h < imageHeight; h++) {
      int uvh = (h / 2).floor();

      for (int w = 0; w < imageWidth; w++) {
        int uvw = (w / 2).floor();

        final yIndex = (h * yRowStride) + (w * yPixelStride);

        // Y plane should have positive values belonging to [0...255]
        final int y = yBuffer[yIndex];

        // U/V Values are subsampled i.e. each pixel in U/V chanel in a
        // YUV_420 image act as chroma value for 4 neighbouring pixels
        final int uvIndex = (uvh * uvRowStride) + (uvw * uvPixelStride);

        // U/V values ideally fall under [-0.5, 0.5] range. To fit them into
        // [0, 255] range they are scaled up and centered to 128.
        // Operation below brings U/V values to [-128, 127].
        final int u = uBuffer[uvIndex];
        final int v = vBuffer[uvIndex];

        // Compute RGB values per formula above.
        int r = (y + v * 1436 / 1024 - 179).round();
        int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
        int b = (y + u * 1814 / 1024 - 227).round();

        r = r.clamp(0, 255);
        g = g.clamp(0, 255);
        b = b.clamp(0, 255);

        image.setPixelRgb(w, h, r, g, b);
      }
    }

    return image;
  }
}

@saad-palapa
Copy link

saad-palapa commented May 21, 2023

After some trial and error, I found the perfect solution for iOS:

const IOS_BYTES_OFFSET = 28;

static Image _convertBGRA8888ToImage(CameraImage cameraImage) {
  final plane = cameraImage.planes[0];

  return Image.fromBytes(
    width: cameraImage.width,
    height: cameraImage.height,
    bytes: plane.bytes.buffer,
    rowStride: plane.bytesPerRow,
    bytesOffset: IOS_BYTES_OFFSET,
    order: ChannelOrder.bgra,
  );
}

The other solution produced a 1088 wide image with a 8px black bar. By adding rowStride and bytesOffset, it is now 1080 width with no black bars.

I have no idea where the offset of 28 comes from. Does anyone know why 28 works?

@jmealo
Copy link

jmealo commented May 22, 2023 via email

@alexcohn
Copy link

@saad-palapa Does anyone know why 28 works?

Now that we know the answer, the explanation is rather easy. 28 bytes is 8 pixels of BGRA. The image in memory is 1088 pixel wide with a black bar before the first column (the illustration keeps the 8 "black" extra pixels, but does not go keep the real dimensions):

XXXXXXXX......................
XXXXXXXX.......... _ .........
XXXXXXXX........ _( )_ .......
XXXXXXXX....... (_(%)_) ......
XXXXXXXX......... (_)\ .......
XXXXXXXX............. | __ ...
XXXXXXXX............. |/_/ ...
XXXXXXXX............. | ......
XXXXXXXX............. | ......
XXXXXXXX......................
XXXXXXXX......................

By adding the offset, you feed to Image.fromBytes() something like

......................XXXXXXXX
.......... _ .........XXXXXXXX
........ _( )_ .......XXXXXXXX
....... (_(%)_) ......XXXXXXXX
......... (_)\ .......XXXXXXXX
............. | __ ...XXXXXXXX
............. |/_/ ...XXXXXXXX
............. | ......XXXXXXXX
............. | ......XXXXXXXX
......................XXXXXXXX
......................

The function is smart enough to throw away the extra pixels on the right when the width parameter is 1080 and rowStride is 1088.

@rraayy
Copy link

rraayy commented Jun 1, 2023

Hello, I meet another problem. Does anyone try to convert NV21 from cameraImage to image ?
Some device (Xiaomi, Motorola) will have the NV21 format. Anyone can help?

@alexcohn
Copy link

alexcohn commented Jun 3, 2023

@rraayy you can see how this can be efficiently done with ffi on medium. The sample code is available at https://github.com/Hugand/camera_tutorial

@rraayy
Copy link

rraayy commented Jun 4, 2023

@rraayy you can see how this can be efficiently done with ffi on medium. The sample code is available at https://github.com/Hugand/camera_tutorial

Thanks so much @alexcohn , I will try it with ffi!

@DmitrySikorsky
Copy link

federico-amura-kenility I think on Android you should use convertYUV420ToImage. I modify my class after update Image lib, please try this code.

This works on Android, thank you very much.

@owjoh
Copy link

owjoh commented Jun 16, 2023

@rraayy Any luck converting NV21 to Image? I'm having the same difficulties. The NV21 formatted CameraImage object only has one plane.

@alexcohn
Copy link

@owjoh here @camsim99 claims that the issue with NV21 for Motorola and Xiaomi has been fixed in a recent CameraX implementation.

Alternatively, convertImage function could be tuned to handle single-plane images, but I don't know their actual layout.

@prkhrv
Copy link

prkhrv commented Aug 30, 2023

can anyone help me with yuv420 to RGB conversion on iOS ?

@prkhrv
Copy link

prkhrv commented Aug 30, 2023

About a few days of struggling with the CamerImage to Image conversion, I managed to improve the method to include paddnig on different devices. I tested on several devices, checking the conversion at different camera resolutions. And I think it works.

imglib.Image convertYUV420ToImage(CameraImage cameraImage) {
    final imageWidth = cameraImage.width;
    final imageHeight = cameraImage.height;

    final yBuffer = cameraImage.planes[0].bytes;
    final uBuffer = cameraImage.planes[1].bytes;
    final vBuffer = cameraImage.planes[2].bytes;

    final int yRowStride = cameraImage.planes[0].bytesPerRow;
    final int yPixelStride = cameraImage.planes[0].bytesPerPixel!;

    final int uvRowStride = cameraImage.planes[1].bytesPerRow;
    final int uvPixelStride = cameraImage.planes[1].bytesPerPixel!;

    final image = imglib.Image(imageWidth, imageHeight);

    for (int h = 0; h < imageHeight; h++) {
      int uvh = (h / 2).floor();

      for (int w = 0; w < imageWidth; w++) {
        int uvw = (w / 2).floor();

        final yIndex = (h * yRowStride) + (w * yPixelStride);

        // Y plane should have positive values belonging to [0...255]
        final int y = yBuffer[yIndex];

        // U/V Values are subsampled i.e. each pixel in U/V chanel in a
        // YUV_420 image act as chroma value for 4 neighbouring pixels
        final int uvIndex = (uvh * uvRowStride) + (uvw * uvPixelStride);

        // U/V values ideally fall under [-0.5, 0.5] range. To fit them into
        // [0, 255] range they are scaled up and centered to 128.
        // Operation below brings U/V values to [-128, 127].
        final int u = uBuffer[uvIndex];
        final int v = vBuffer[uvIndex];

        // Compute RGB values per formula above.
        int r = (y + v * 1436 / 1024 - 179).round();
        int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
        int b = (y + u * 1814 / 1024 - 227).round();

        r = r.clamp(0, 255);
        g = g.clamp(0, 255);
        b = b.clamp(0, 255);

        // Use 255 for alpha value, no transparency. ARGB values are
        // positioned in each byte of a single 4 byte integer
        // [AAAAAAAARRRRRRRRGGGGGGGGBBBBBBBB]
        final int argbIndex = h * imageWidth + w;

        image.data[argbIndex] = 0xff000000 |
            ((b << 16) & 0xff0000) |
            ((g << 8) & 0xff00) |
            (r & 0xff);
      }
    }

    return image;
  }

Does this work for iOS ?

@rraayy
Copy link

rraayy commented Aug 31, 2023

Hi @prkhrv,
Please take a look at nv21 to Image
In my project, the solution is work for me.

@min23asdw
Copy link

min23asdw commented Feb 24, 2024

i fix it with use image.setPixelRgb it got valid image
by
"
imglib.Image convertYUV420ToImage(CameraImage cameraImage) {
final imageWidth = cameraImage.width;
final imageHeight = cameraImage.height;

final yBuffer = cameraImage.planes[0].bytes;
final uBuffer = cameraImage.planes[1].bytes;
final vBuffer = cameraImage.planes[2].bytes;

final int yRowStride = cameraImage.planes[0].bytesPerRow;
final int yPixelStride = cameraImage.planes[0].bytesPerPixel!;

final int uvRowStride = cameraImage.planes[1].bytesPerRow;
final int uvPixelStride = cameraImage.planes[1].bytesPerPixel!;

final image = imglib.Image(width: imageWidth, height: imageHeight);

for (int h = 0; h < imageHeight; h++) {
  int uvh = (h / 2).floor();

  for (int w = 0; w < imageWidth; w++) {
    int uvw = (w / 2).floor();

    final yIndex = (h * yRowStride) + (w * yPixelStride);

    // Y plane should have positive values belonging to [0...255]
    final int y = yBuffer[yIndex];

    // U/V Values are subsampled i.e. each pixel in U/V chanel in a
    // YUV_420 image act as chroma value for 4 neighbouring pixels
    final int uvIndex = (uvh * uvRowStride) + (uvw * uvPixelStride);

    // U/V values ideally fall under [-0.5, 0.5] range. To fit them into
    // [0, 255] range they are scaled up and centered to 128.
    // Operation below brings U/V values to [-128, 127].
    final int u = uBuffer[uvIndex];
    final int v = vBuffer[uvIndex];

    // Compute RGB values per formula above.
    int r = (y + v * 1436 / 1024 - 179).round();
    int g = (y - u * 46549 / 131072 + 44 - v * 93604 / 131072 + 91).round();
    int b = (y + u * 1814 / 1024 - 227).round();

    r = r.clamp(0, 255);
    g = g.clamp(0, 255);
    b = b.clamp(0, 255);

    // Use 255 for alpha value, no transparency.
    image.setPixelRgb(w, h, r, g, b);
  }
}

return image;

}
"
but it still rotate the image by 270 degrees
you can fix it by rotate but i don't know how to to i just rotate in python backend !

@KevinCCucumber
Copy link

imglib.Image _convertYUV420(CameraImage image) {
  var img = imglib.Image(image.width, image.height); // Create Image buffer

  final int width = image.width;
  final int height = image.height;
  final int uvRowStride = image.planes[1].bytesPerRow;
  final int uvPixelStride = image.planes[1].bytesPerPixel;
  const shift = (0xFF << 24);

  for(int x=0; x < width; x++) {
    for(int y=0; y < height; y++) {
      final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
      final int index = y * width + x;

      final yp = image.planes[0].bytes[index];
      final up = image.planes[1].bytes[uvIndex];
      final vp = image.planes[2].bytes[uvIndex];
      // Calculate pixel color
      int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
      int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
      int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);     
      // color: 0x FF  FF  FF  FF 
      //           A   B   G   R
      img.data[index] = shift | (b << 16) | (g << 8) | r;
    }
  }

  return img;
}

@sikandernoori this way, it can be converted into color images, but it takes more than 1000ms to convert on a mobile phone with Snapdragon 870 CPU, and it will block the UI.

This works with 2 planes like the iPad camera does. But for me, "final int uvPixelStride = image.planes[1].bytesPerPixel;" is always null, so I cannot use this code. Any Idea what I can change if that is null?

@alexcohn
Copy link

@KevinCCucumber what device are you working with?

@KevinCCucumber
Copy link

@KevinCCucumber what device are you working with?

@alexcohn I am using an ipad air 5th gen

@alexcohn
Copy link

alexcohn commented Mar 28, 2024

@KevinCCucumber what does it report on ipad air 5th gen for image.planes[0].bytesPerPixel? image.planes[1].bytesPerRow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment