Skip to content

[camera] document how to use ImageStream #26348

Open

Description

@eduardkieser

First of all thank you so mush for adding access to the image stream in 0.2.8, I'm sure may people will be very happy about this.
I know the latest commit is fresh off the press, bit I would love to see a little more documentation on how to use the camera image stream (other than: "use: cameraController.startImageStream(listener) to process the images")

Activity

  1. added
    d: api docsIssues with https://api.flutter.dev/
    on Jan 10, 2019
  2. added this to the milestone on Jan 10, 2019
  3. lewixlabs commented on Jan 10, 2019

    @lewixlabs

    Yes agree, great feature but it needs documentation.
    It seems working, i added this new method and its callback in initState inside demo project

      @override
      void initState() {
        super.initState();
        controller = CameraController(cameras[0], ResolutionPreset.medium);
        controller.initialize().then((_) {
          if (!mounted) {
            return;
          }
    
          controller.startImageStream((CameraImage availableImage) {
            controller.stopImageStream();
            _scanText(availableImage);
          });
    
          setState(() {});
        });
      }
    
    void _scanText(CameraImage availableImage) async {
      
      final FirebaseVisionImage visionImage = FirebaseVisionImage.fromBytes(availableImage.planes[0].bytes, null);
      final TextRecognizer textRecognizer = FirebaseVision.instance.textRecognizer();
      final VisionText visionText = await textRecognizer.processImage(visionImage);
    
      print(visionText.text);
      for (TextBlock block in visionText.blocks) {
    
        // final Rectangle<int> boundingBox = block.boundingBox;
        // final List<Point<int>> cornerPoints = block.cornerPoints;
        print(block.text);
        final List<RecognizedLanguage> languages = block.recognizedLanguages;
    
        for (TextLine line in block.lines) {
          // Same getters as TextBlock
          print(line.text);
          for (TextElement element in line.elements) {
            // Same getters as TextBlock
            print(element.text);
          }
        }
      }
    }

    Now i'm learning how to handle method FirebaseVisionImage.fromBytes to set metadata

  4. changed the title [-]Camera documentation[/-] [+]Camera document how to use ImageStream[/+] on Jan 10, 2019
  5. lewixlabs commented on Jan 10, 2019

    @lewixlabs

    This is a code sample just update
    https://github.com/lewixlabs/mlkit_ocr_realtime/blob/96a749c0568247fc0063d52a4f03f0c6385f52cc/lib/main.dart#L118-L168
    but not working because i get this error

    PlatformException (PlatformException(textRecognizerError, Internal error has occurred when executing Firebase ML tasks, null))

    when

    final VisionText visionText = await textRecognizer.processImage(visionImage);

    is called.
    Full sample is here
    https://github.com/lewixlabs/mlkit_ocr_realtime/blob/master/lib/main.dart

    Waiting for the right direction, thank you! :)

  6. lewixlabs commented on Jan 11, 2019

    @lewixlabs

    Uuuuh, my updated code works on Android! :)
    Tomorrow i test it on iOS...

      void _scanText(CameraImage availableImage) async {
        _isScanBusy = true;
    
        print("scanning!...");
    
        /*
         * https://firebase.google.com/docs/ml-kit/android/recognize-text
         * .setWidth(480)   // 480x360 is typically sufficient for
         * .setHeight(360)  // image recognition
         */
    
        final FirebaseVisionImageMetadata metadata = FirebaseVisionImageMetadata(
          rawFormat: availableImage.format.raw,
          size: Size(availableImage.width.toDouble(),availableImage.height.toDouble()),
          planeData: availableImage.planes.map((currentPlane) => FirebaseVisionImagePlaneMetadata(
            bytesPerRow: currentPlane.bytesPerRow,
            height: currentPlane.height,
            width: currentPlane.width
            )).toList(),
          rotation: ImageRotation.rotation90
          );
    
        final FirebaseVisionImage visionImage = FirebaseVisionImage.fromBytes(availableImage.planes[0].bytes, metadata);
        final TextRecognizer textRecognizer = FirebaseVision.instance.textRecognizer();
        final VisionText visionText = await textRecognizer.processImage(visionImage);
    
        print("--------------------visionText:${visionText.text}");
        for (TextBlock block in visionText.blocks) {
          // final Rectangle<int> boundingBox = block.boundingBox;
          // final List<Point<int>> cornerPoints = block.cornerPoints;
          print(block.text);
          final List<RecognizedLanguage> languages = block.recognizedLanguages;
    
          for (TextLine line in block.lines) {
            // Same getters as TextBlock
            print(line.text);
            for (TextElement element in line.elements) {
              // Same getters as TextBlock
              print(element.text);
            }
          }
        }
    
        _isScanBusy = false;
      }

    Full demo project here
    https://github.com/lewixlabs/mlkit_ocr_realtime

  7. BoHellgren commented on Jan 30, 2019

    @BoHellgren

    I am trying to display an image from an ImageStream. One one button I have

    onPressed: () async  {
                    await controller.startImageStream((CameraImage availableImage)  {
                      print(availableImage.planes[0].bytes.toString().substring(0,200));
                      setState(() {
                        theImage = availableImage.planes[0].bytes;
                      });
                    });

    where theImage is declared as Uint8List.

    A list of numbers between 0 and 255 is displayed. Like

    I/flutter (18471): [16, 118, 115, 112, 114, 117, 118, 119, 120, 120, 120, 120, 118, 120, etc
    

    But if I try

    Image.memory(theImage) or
    MemoryImage(theImage) I get

    [ERROR:flutter/lib/ui/painting/codec.cc(97)] Failed decoding image. Data is either invalid, or it is encoded using an unsupported format.
    

    What am I doing wrong?

    (I also have great problems stopping the ImageStream with controller.stopImageStream() ).

  8. lewixlabs commented on Jan 31, 2019

    @lewixlabs
  9. eduardkieser commented on Jan 31, 2019

    @eduardkieser
    Author
  10. zoechi commented on Jan 31, 2019

    @zoechi
    Contributor

    @lewixlabs @eduardkieser Using "add reaction" on the initial comment would increase priority while +1 comments are rather pointless and cause lots of people to get notified for no reason.
    To get notified yourself use the [Subscribe] button to the top right instead.

  11. BoHellgren commented on Feb 1, 2019

    @BoHellgren

    This is a sample of what the CameraImage contains:

    I/flutter (27030): Format group: ImageFormatGroup.yuv420
    I/flutter (27030): height: 480
    I/flutter (27030): width: 640
    I/flutter (27030): Number of planes: 3
    I/flutter (27030): plane 0 length: 307200
    I/flutter (27030): plane 0 data: [16, 21, 21, 19, 19,
    I/flutter (27030): plane 1 length: 76800
    I/flutter (27030): plane 1 data: [130, 126, 126, 125,
    I/flutter (27030): plane 2 length: 76800
    I/flutter (27030): plane 2 data: [125, 131, 131, 132,
    

    How can I convert this to a Uint8List acceptable to Image.memory()??
    Or is there another widget which can display a CameraImage?

  12. rockingelevator commented on Feb 6, 2019

    @rockingelevator
  13. 100 remaining items

  14. BoHellgren commented on Jan 9, 2023

    @BoHellgren

    CameraImage to imageLib.Image performance
    I made the function below which converts a CameraImage to a imageLib.Image using a) C code and Dart.ffi, and b) plain Dart. I expected the C code to be much faster, but, to my surprise, the Dart code is some 25% faster! Can anyone explain why? Any ideas on how to do this conversion really fast?

    import 'dart:typed_data';
    import 'package:ffi/ffi.dart';
    import 'package:image/image.dart' as imageLib;
    import 'package:camera/camera.dart';
    
    typedef convert_func = Pointer<Uint32> Function(
        Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, Int32, Int32, Int32, Int32);
    typedef Convert = Pointer<Uint32> Function(
        Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, int, int, int, int);
    
    imageLib.Image convertCameraImage(CameraImage image, Convert conv) {
    //*********************FFI VERSION******************************
      /* Put this code in main.dart
      final DynamicLibrary convertImageLib =
          DynamicLibrary.open("libconvertImage.so");
      Convert? conv;
      ... and in initState
      conv = convertImageLib
          .lookup<NativeFunction<convert_func>>('convertImage')
          .asFunction<Convert>(); */
      var t1 = DateTime.now().millisecondsSinceEpoch;
      // Allocate memory for the 3 planes of the image
      Pointer<Uint8> p = malloc.allocate(image.planes[0].bytes.length);
      Pointer<Uint8> p1 = malloc.allocate(image.planes[1].bytes.length);
      Pointer<Uint8> p2 = malloc.allocate(image.planes[2].bytes.length);
      // Assign the planes data to the pointers of the image
      Uint8List pointerList = p.asTypedList(image.planes[0].bytes.length);
      Uint8List pointerList1 = p1.asTypedList(image.planes[1].bytes.length);
      Uint8List pointerList2 = p2.asTypedList(image.planes[2].bytes.length);
      pointerList.setRange(0, image.planes[0].bytes.length, image.planes[0].bytes);
      pointerList1.setRange(0, image.planes[1].bytes.length, image.planes[1].bytes);
      pointerList2.setRange(0, image.planes[2].bytes.length, image.planes[2].bytes);
    
      // Call the convertImage function and convert the YUV to RGB
      Pointer<Uint32> imgP = conv(
          p,
          p1,
          p2,
          image.planes[1].bytesPerRow,
          image.planes[1].bytesPerPixel!,
          image.planes[0].bytesPerRow,
          image.height);
    
      // Get the pointer of the data returned from the function to a List
      List imgData = imgP.asTypedList((image.planes[0].bytesPerRow * image.height));
      // Generate image from the converted data
      imageLib.Image img1 = imageLib.Image.fromBytes(
          image.height, image.planes[0].bytesPerRow, imgData as List<int>);
      // Free the memory space allocated
      // from the planes and the converted data
      malloc.free(p);
      malloc.free(p1);
      malloc.free(p2);
      malloc.free(imgP);
    
      var t2 = DateTime.now().millisecondsSinceEpoch;
      // return img1;  //Comment to use Dart
    //******************DART VERSION***********************
      var t3 = DateTime.now().millisecondsSinceEpoch;
      int width = image.width;
      int height = image.height;
      var img2 = imageLib.Image(width, height); // Create Image buffer
      const int hexFF = 0xFF000000;
      final int uvyButtonStride = image.planes[1].bytesPerRow;
      final int? uvPixelStride = image.planes[1].bytesPerPixel;
      for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
          final int uvIndex = uvPixelStride! * (x / 2).floor() +
              uvyButtonStride * (y / 2).floor();
          final int index = y * width + x;
          final yp = image.planes[0].bytes[index];
          final up = image.planes[1].bytes[uvIndex];
          final vp = image.planes[2].bytes[uvIndex];
    // Calculate pixel color
          int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
          int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
              .round()
              .clamp(0, 255);
          int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
    // color: 0x FF  FF  FF  FF
    //           A   B   G   R
          img2.data[index] = hexFF | (b << 16) | (g << 8) | r;
        }
      }
    // Rotate 90 degrees to upright
      var img3 = imageLib.copyRotate(img2, 90);
      var t4 = DateTime.now().millisecondsSinceEpoch;
      print('*********** conv took ${t2 - t1} milliseconds, Dart took ${t4 - t3} milliseconds');
    
      return img3;
    
    }
    

    Here is the C code (by @hugand)

    #include <stdio.h>
    #include "converter.h"
    #include <math.h>
    #include <stdlib.h>
    
    int clamp(int lower, int higher, int val){
        if(val < lower)
            return 0;
        else if(val > higher)
            return 255;
        else
            return val;
    }
    
    int getRotatedImageByteIndex(int x, int y, int rotatedImageWidth){
        return rotatedImageWidth*(y+1)-(x+1);
    }
    
    uint32_t *convertImage(uint8_t *plane0, uint8_t *plane1, uint8_t *plane2, int bytesPerRow, int bytesPerPixel, int width, int height){
         int hexFF = 255;
        int x, y, uvIndex, index;
        int yp, up, vp;
        int r, g, b;
        int rt, gt, bt;
    
        uint32_t *image = malloc(sizeof(uint32_t) * (width * height));
    
        for(x = 0; x < width; x++){
            for(y = 0; y < height; y++){
                
                uvIndex = bytesPerPixel * ((int) floor(x/2)) + bytesPerRow * ((int) floor(y/2));
                index = y*width+x;
    
                yp = plane0[index];
                up = plane1[uvIndex];
                vp = plane2[uvIndex];
                rt = round(yp + vp * 1436 / 1024 - 179);
                gt = round(yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91);
                bt = round(yp + up * 1814 / 1024 - 227);
                r = clamp(0, 255, rt);
                g = clamp(0, 255, gt);
                b = clamp(0, 255, bt);
                image[getRotatedImageByteIndex(y, x, height)] = (hexFF << 24) | (b << 16) | (g << 8) | r;
            }
        }
        return image;
    }
    
  15. faslurrajah commented on Jun 21, 2023

    @faslurrajah

    This works for me.

    Future<Uint8List> convertImageToPng(CameraImage image) async {
        Uint8List bytes;
        try {
          imglib.Image img;
          if (image.format.group == ImageFormatGroup.yuv420) {
            bytes = await convertYUV420toImageColor(image);
          } else if (image.format.group == ImageFormatGroup.bgra8888) {
            img = _convertBGRA8888(image);
            imglib.PngEncoder pngEncoder = new imglib.PngEncoder();
            bytes = pngEncoder.encodeImage(img);
          }
          return bytes;
        } catch (e) {
          print(">>>>>>>>>>>> ERROR:" + e.toString());
        }
        return null;
      }
      imglib.Image _convertBGRA8888(CameraImage image) {
        return imglib.Image.fromBytes(
          image.width,
          image.height,
          image.planes[0].bytes,
          format: imglib.Format.bgra,
        );
      }
      Future<Uint8List> convertYUV420toImageColor(CameraImage image) async {
        try {
          final int width = image.width;
          final int height = image.height;
          final int uvRowStride = image.planes[1].bytesPerRow;
          final int uvPixelStride = image.planes[1].bytesPerPixel;
          print("uvRowStride: " + uvRowStride.toString());
          print("uvPixelStride: " + uvPixelStride.toString());
          var img = imglib.Image(width, height); // Create Image buffer
          // Fill image buffer with plane[0] from YUV420_888
          for (int x = 0; x < width; x++) {
            for (int y = 0; y < height; y++) {
              final int uvIndex =
                  uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
              final int index = y * width + x;
              final yp = image.planes[0].bytes[index];
              final up = image.planes[1].bytes[uvIndex];
              final vp = image.planes[2].bytes[uvIndex];
              int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
              int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
                  .round()
                  .clamp(0, 255);
              int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
              img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;
            }
          }
          imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
          Uint8List png = pngEncoder.encodeImage(img);
          final originalImage = imglib.decodeImage(png);
          final height1 = originalImage.height;
          final width1 = originalImage.width;
          imglib.Image fixedImage;
          if (height1 < width1) {
            fixedImage = imglib.copyRotate(originalImage, 90);
          }
          final path =
              join((await getTemporaryDirectory()).path, "${DateTime.now()}.jpg");
          print(path);
          File(path).writeAsBytesSync(imglib.encodeJpg(fixedImage));
          return imglib.encodeJpg(fixedImage);
        } catch (e) {
          print(">>>>>>>>>>>> ERROR:" + e.toString());
        }
        return null;
      }

    I'm trying to the same. But I cannot edit ImageData variable's index. im1.data is a ImageData type. I'm trying to do as you do img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;

    image

    I'm getting compiling error The operator '[]=' isn't defined for the type 'ImageData'.

  16. caohy7 commented on Jul 28, 2023

    @caohy7

    This is how CameraImage is handled in the iOS environment

    Under the startImageStream function

    var bytes = cameraImg.planes[0].bytes;
    Map<String, dynamic> json = {};
    json["bytes"] = bytes;
    json['bytesPerRow'] = cameraImg.planes[0].bytesPerRow;
    json['height'] = cameraImg.planes[0].height;
    json['width'] = cameraImg.planes[0].width;
    json['raw'] = cameraImg.format.raw as int;
    Uint8List? result = await channel.invokeMethod('invokeConvert', json);
    

    Called in the callback of iOS's MethodChannel

    if([method isEqualToString:@"invokeConvert"]){
        NSData *data = [ConvertCameraImg inputDataWithJson:call.arguments];
        result(data);
    }
    

    Specific treatment methods

    import Foundation
    import Accelerate
    
    struct Image: Codable {
        let width: Int
        let height: Int
        let bytesPerRow: Int
        let raw: Int
        var data: Data!
    }
    
    class ConvertCameraImg: NSObject {
        @objc public static func inputData(json: [String: Any]) -> Data? {
            var j = json
            let data = (j.removeValue(forKey: "bytes") as? FlutterStandardTypedData)!.data
            guard let jsonData = try? JSONSerialization.data(withJSONObject: j) else { return nil }
            guard var image = try? JSONDecoder().decode(Image.self, from: jsonData) else { return nil }
            image.data = data
            var buffer = [UInt8](data)
            let uint8Pointer = UnsafeMutablePointer<UInt8>.allocate(capacity: buffer.count)
            uint8Pointer.initialize(from: &buffer, count: buffer.count)
            defer {
                uint8Pointer.deallocate()
            }
    
            ConvertCameraImg.swizzleBGRA8toRGBA8(uint8Pointer, width: image.width, height: image.height)
            
            guard let colorSpace = CGColorSpace(name: CGColorSpace.linearSRGB) else { return nil }
            let bitmapInfo = CGImageAlphaInfo.premultipliedLast.rawValue
            guard let bitmapContext = CGContext(data: nil,
                                                width: image.width,
                                                height: image.height,
                                                bitsPerComponent: 8,
                                                bytesPerRow: image.bytesPerRow,
                                                space: colorSpace,
                                                bitmapInfo: bitmapInfo) else { return nil }
            bitmapContext.data?.copyMemory(from: buffer, byteCount: buffer.count)
            
            guard let img = bitmapContext.makeImage() else { return nil }
            
            
            guard let imageData = CFDataCreateMutable(kCFAllocatorDefault, 0) else { return nil }
            guard let dest = CGImageDestinationCreateWithData(imageData, "public.jpeg" as CFString, 1, nil) else { return nil }
            CGImageDestinationAddImage(dest, img, nil)
            let result = CGImageDestinationFinalize(dest)
            if result {
                return imageData as Data
            }
            return nil
        }
        
        static func swizzleBGRA8toRGBA8(_ bytes: UnsafeMutableRawPointer, width: Int, height: Int) {
            var sourceBuffer = vImage_Buffer(data: bytes,
                                             height: vImagePixelCount(height),
                                             width: vImagePixelCount(width),
                                             rowBytes: width * 4)
            var destBuffer = vImage_Buffer(data: bytes,
                                           height: vImagePixelCount(height),
                                           width: vImagePixelCount(width),
                                           rowBytes: width * 4)
            var swizzleMask: [UInt8] = [ 2, 1, 0, 3 ] // BGRA -> RGBA
            vImagePermuteChannels_ARGB8888(&sourceBuffer, &destBuffer, &swizzleMask, vImage_Flags(kvImageNoFlags))
        }
    }
    

    Android processing can use the YUVTransform plugin

  17. guyluz11 commented on Nov 5, 2023

    @guyluz11

    This is a sample of what the CameraImage contains:

    I/flutter (27030): Format group: ImageFormatGroup.yuv420
    I/flutter (27030): height: 480
    I/flutter (27030): width: 640
    I/flutter (27030): Number of planes: 3
    I/flutter (27030): plane 0 length: 307200
    I/flutter (27030): plane 0 data: [16, 21, 21, 19, 19,
    I/flutter (27030): plane 1 length: 76800
    I/flutter (27030): plane 1 data: [130, 126, 126, 125,
    I/flutter (27030): plane 2 length: 76800
    I/flutter (27030): plane 2 data: [125, 131, 131, 132,
    

    How can I convert this to a Uint8List acceptable to Image.memory()?? Or is there another widget which can display a CameraImage?

    I am using

    Codec codec = await instantiateImageCodec(ulist)
    image = (await codec.getNextFrame()).image

    and you can show the image like this

    RawImage(image: image)
    And it works for normal images.

    But like others here I am getting errors in the conversion when using the images from the stream

    [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Exception: Invalid image data
    #0      _futurize (dart:ui/painting.dart:6950:5)
    #1      ImageDescriptor.encoded (dart:ui/painting.dart:6764:12)
    #2      instantiateImageCodecWithSize (dart:ui/painting.dart:2307:60)
    #3      instantiateImageCodecFromBuffer (dart:ui/painting.dart:2251:10)
    #4      instantiateImageCodec (dart:ui/painting.dart:2198:10)
    <asynchronous suspension>
    #5      _ImageStreamPageState.videoStream.<anonymous closure> (package:cbj_smart_device_flutter/presentation/client/image_stream_page.dart:39:11)
    <asynchronous suspension>
  18. GleammerRay commented on Jul 2, 2024

    @GleammerRay

    @faslurrajah this solution no longer works on latest package version due to API changes. Here's an adjusted series of methods from my project, ready for general use and should work on both Android and iOS:

    // CameraImage BGRA8888 -> PNG
    // Color
    imglib.Image imageFromBGRA8888(CameraImage image) {
      return imglib.Image.fromBytes(
        width: image.width,
        height: image.height,
        bytes: image.planes[0].bytes.buffer,
        order: imglib.ChannelOrder.bgra,
      );
    }
    
    // CameraImage YUV420_888 -> PNG -> Image (compresion:0, filter: none)
    // Black
    imglib.Image imageFromYUV420(CameraImage image) {
      final uvRowStride = image.planes[1].bytesPerRow;
      final uvPixelStride = image.planes[1].bytesPerPixel ?? 0;
      final img = imglib.Image(width: image.width, height: image.height);
      for (final p in img) {
        final x = p.x;
        final y = p.y;
        final uvIndex =
            uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
        final index = y * uvRowStride +
            x; // Use the row stride instead of the image width as some devices pad the image data, and in those cases the image width != bytesPerRow. Using width will give you a distored image.
        final yp = image.planes[0].bytes[index];
        final up = image.planes[1].bytes[uvIndex];
        final vp = image.planes[2].bytes[uvIndex];
        p.r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255).toInt();
        p.g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
            .round()
            .clamp(0, 255)
            .toInt();
        p.b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255).toInt();
      }
    
      return img;
    }
    
    imglib.Image? imageFromCameraImage(CameraImage image) {
      try {
        imglib.Image img;
        switch (image.format.group) {
          case ImageFormatGroup.yuv420:
            img = imageFromYUV420(image);
            break;
          case ImageFormatGroup.bgra8888:
            img = imageFromBGRA8888(image);
            break;
          default:
            return null;
        }
        return img;
      } catch (e) {
        //print(">>>>>>>>>>>> ERROR:" + e.toString());
      }
      return null;
    }
  19. thandal commented on Jul 6, 2024

    @thandal

    FYI: I think I have an explanation. While some phones produce yuv420p, others actually quietly are producing nv21/nv12, but then creating interleaved U and V planes.

    • Y plane -- always good, always the expected size
    • if U and V planes have 2 bytesPerPixel (better named pixelStride in the Android CameraX documentation), they are actually interleaved U and V channels

    This means the U and V bytes are interleaved in memory as follows, and the U and V plane buffers are just offset by one byte:

    [ U1 V1 U2 V2 ....  Un Vn]  <-- actual layout in memory, with length (w/2) * (h/2) * 2
     ^-- U buffer starts, pixelStride is 2
         ^-- V buffer starts, pixelStride is 2
                         ^-- U buffer ends, size is odd
                            ^-- V buffer ends, size is odd
    

    Which is actually nv21 (or nv12 because U is first) "faking it" as yuv420p! So everything makes sense, and you can actually get nv21 just by using the whole (interleaved) U buffer, and the last byte of the V buffer! Of course, it would be best if we could actually get "explicit" n21.

  20. alexcohn commented on Jul 7, 2024

    @alexcohn

    @thandal I am not sure what you want to achieve this way. Is there some API that expects an nv12 contiguous buffer?

  21. angelru commented on Mar 13, 2025

    @angelru

    Hi guys i just fixed it by a simple argument in controller: (jpeg format is now supported) cameraController = CameraController( await _getCamera(_direction), ResolutionPreset.low, imageFormatGroup: ImageFormatGroup.jpeg); so if you just specify image format you don't have to mess with yuv420

    in streamer you just have to do something like this: bytes = cameraImage.planes[0].bytes; then you can simply use it in your opencv or image.memory and after all I think this way it has a much better performance

    Does this work?

  22. changed the title [-]Camera document how to use ImageStream[/-] [+][camera] document how to use ImageStream[/+] on Jul 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listcustomer: crowdAffects or could affect many people, though not necessarily a specific customer.d: api docsIssues with https://api.flutter.dev/p: cameraThe camera pluginpackageflutter/packages repository. See also p: labels.team-ecosystemOwned by Ecosystem teamtriaged-ecosystemTriaged by Ecosystem team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions