Skip to content

Instantly share code, notes, and snippets.

@Tanapruk

Tanapruk/Camera2.md

Last active October 20, 2022 07:49
Show Gist options
  • Save Tanapruk/3924061c29875dad9cd7ace302904185 to your computer and use it in GitHub Desktop.
Save Tanapruk/3924061c29875dad9cd7ace302904185 to your computer and use it in GitHub Desktop.
Download ZIP
Camera2 API
Raw
Camera2.md

TextureView

You need to initialize TextureView before opening a camera. TextureView is for displaying the image from camera on the device. However, you cannot use it unless the texture is ready, either because the screen is currently off or it is initializing.

You should check mTextureView.isAvailable() before opening your camera. Else you will mTextureView.setSurfaceTextureListener(). And open your camera in the callback method onSurfaceTextureListener.

Camera

Before opening your camera you should check for a camera permission first.

Next, you initialize a cameraManager through CameraManager manager = (CameraManager) activity.getSystemService(Context.CAMERA_SERVICE);

This manager is our hero.

  • It knows metadata of your camera devices.
  • It can initialize camera.

Metadata

E.g., How many camera you have?

manager.getCameraIdList(); Usually, it will return 0 and 1.

  • 0 means it is your back camera.
  • 1 for your front camera.
What is the characteristics of your camera?

CameraCharacteristics characteristics = manager.getCameraCharacteristics(cameraId);

Available characteristics including:

CameraCharacteristics.Key(android.colorCorrection.availableAberrationModes)
CameraCharacteristics.Key(android.control.aeAvailableAntibandingModes)
CameraCharacteristics.Key(android.control.aeAvailableModes)
CameraCharacteristics.Key(android.control.aeAvailableTargetFpsRanges)
CameraCharacteristics.Key(android.control.aeCompensationRange)
CameraCharacteristics.Key(android.control.aeCompensationStep)
CameraCharacteristics.Key(android.control.aeLockAvailable)
CameraCharacteristics.Key(android.control.afAvailableModes)
CameraCharacteristics.Key(android.control.availableEffects)
CameraCharacteristics.Key(android.control.availableModes)
CameraCharacteristics.Key(android.control.availableSceneModes)
CameraCharacteristics.Key(android.control.availableVideoStabilizationModes)
CameraCharacteristics.Key(android.control.awbAvailableModes)
CameraCharacteristics.Key(android.control.awbLockAvailable)
CameraCharacteristics.Key(android.control.maxRegionsAe)
CameraCharacteristics.Key(android.control.maxRegionsAf)
CameraCharacteristics.Key(android.control.maxRegionsAwb)
CameraCharacteristics.Key(android.flash.info.available)
CameraCharacteristics.Key(android.info.supportedHardwareLevel)
CameraCharacteristics.Key(android.jpeg.availableThumbnailSizes)
CameraCharacteristics.Key(android.lens.facing)
CameraCharacteristics.Key(android.lens.info.availableFocalLengths)
CameraCharacteristics.Key(android.noiseReduction.availableNoiseReductionModes)
CameraCharacteristics.Key(android.request.availableCapabilities)
CameraCharacteristics.Key(android.request.maxNumOutputProc)
CameraCharacteristics.Key(android.request.maxNumOutputProcStalling)
CameraCharacteristics.Key(android.request.maxNumOutputRaw)
CameraCharacteristics.Key(android.request.partialResultCount)
CameraCharacteristics.Key(android.request.pipelineMaxDepth)
CameraCharacteristics.Key(android.scaler.availableMaxDigitalZoom)
CameraCharacteristics.Key(android.scaler.croppingType)
CameraCharacteristics.Key(android.scaler.streamConfigurationMap)
CameraCharacteristics.Key(android.sensor.availableTestPatternModes)
CameraCharacteristics.Key(android.sensor.info.activeArraySize)
CameraCharacteristics.Key(android.sensor.info.physicalSize)
CameraCharacteristics.Key(android.sensor.info.pixelArraySize)
CameraCharacteristics.Key(android.sensor.info.timestampSource)
CameraCharacteristics.Key(android.sensor.orientation)
CameraCharacteristics.Key(android.statistics.info.availableFaceDetectModes)
CameraCharacteristics.Key(android.statistics.info.maxFaceCount)
CameraCharacteristics.Key(android.sync.maxLatency)

you can get this key list from characteristics.getKeys();

How good is your hardware can be obtained by

characteristics.get(CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL)

You can check it by comparing with built-in integer. e.g.,

characteristics.get(CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL) == CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL_FULL

From my observation, Pixel Phone 2016 is CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL_3.

My OnePlus One is CameraCharacteristics.INFO_SUPPORTED_HARDWARE_LEVEL_LEGACY.

Various hardware supports allow different capabilities.

  • LEGACY devices operate in a backwards-compatibility mode for older Android devices, and have very limited capabilities.
  • LIMITED devices represent the baseline feature set, and may also include additional capabilities that are subsets of FULL.
  • FULL devices additionally support per-frame manual control of sensor, flash, lens and post-processing settings, and image capture at a high rate.
  • LEVEL_3 devices additionally support YUV reprocessing and RAW image capture, along with additional output stream configurations.

Image size

StreamConfigurationMap map = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);

   StreamConfiguration(
       Outputs(
          [w: 4160, h: 3120, format: JPEG(256),       min_duration: 0, stall: 1121523200],
          [w: 4160, h: 2340, format: JPEG(256),       min_duration: 0, stall: 891142400],
          [w: 4000, h: 3000, format: JPEG(256),       min_duration: 0, stall: 1052000000],
          [w: 4096, h: 2160, format: JPEG(256),       min_duration: 0, stall: 828162560],
          [w: 3200, h: 2400, format: JPEG(256),       min_duration: 0, stall: 745280000],
          [w: 3200, h: 1800, format: JPEG(256),       min_duration: 0, stall: 608960000],
          [w: 2592, h: 1944, format: JPEG(256),       min_duration: 0, stall: 557758208],
          [w: 2048, h: 1536, format: JPEG(256),       min_duration: 0, stall: 423346688],
          [w: 1920, h: 1080, format: JPEG(256),       min_duration: 0, stall: 347225600],
          [w: 1600, h: 1200, format: JPEG(256),       min_duration: 0, stall: 336320000],
          [w: 1280, h: 768,  format: JPEG(256),       min_duration: 0, stall: 269795840],
          [w: 1280, h: 720,  format: JPEG(256),       min_duration: 0, stall: 265433600],
          [w: 1024, h: 768,  format: JPEG(256),       min_duration: 0, stall: 255836672],
          [w: 800,  h: 600,  format: JPEG(256),       min_duration: 0, stall: 234080000],
          [w: 800,  h: 480,  format: JPEG(256),       min_duration: 0, stall: 227264000],
          [w: 720,  h: 480,  format: JPEG(256),       min_duration: 0, stall: 224537600],
          [w: 640,  h: 480,  format: JPEG(256),       min_duration: 0, stall: 221811200],
          [w: 320,  h: 240,  format: JPEG(256),       min_duration: 0, stall: 205452800],
          [w: 1440, h: 1080, format: PRIVATE(34),     min_duration: 0, stall: 0],
          [w: 1280, h: 720,  format: PRIVATE(34),     min_duration: 0, stall: 0],
          [w: 720,  h: 480,  format: PRIVATE(34),     min_duration: 0, stall: 0],
          [w: 640,  h: 480,  format: PRIVATE(34),     min_duration: 0, stall: 0],
          [w: 320,  h: 240,  format: PRIVATE(34),     min_duration: 0, stall: 0],
          [w: 1440, h: 1080, format: YUV_420_888(35), min_duration: 0, stall: 0],
          [w: 1280, h: 720,  format: YUV_420_888(35), min_duration: 0, stall: 0],
          [w: 720,  h: 480,  format: YUV_420_888(35), min_duration: 0, stall: 0],
          [w: 640,  h: 480,  format: YUV_420_888(35), min_duration: 0, stall: 0],
          [w: 320,  h: 240,  format: YUV_420_888(35), min_duration: 0, stall: 0],
          [w: 1440, h: 1080, format: YV12(842094169), min_duration: 0, stall: 0],
          [w: 1280, h: 720,  format: YV12(842094169), min_duration: 0, stall: 0],
          [w: 720,  h: 480,  format: YV12(842094169), min_duration: 0, stall: 0],
          [w: 640,  h: 480,  format: YV12(842094169), min_duration: 0, stall: 0],
          [w: 320,  h: 240,  format: YV12(842094169), min_duration: 0, stall: 0]),
          HighResolutionOutputs(),
          Inputs(),
          ValidOutputFormatsForInput(),
          HighSpeedVideoConfigurations()
     )

Handle Rotation

When you rotate your screen, your camera angle will not automatically rotate the preview picture with you. You need to transform your picture accordingly, specifically image size, aspect ratio and angle.

  • Matrix The Matrix class is the same as the Matrix(maths) that you may have a brief glance and forget it long ago.
  • This matrix will be applied with your picture/your bitmap/your area or in our case our TextureView.
  • With this you can change its aspect ratio, rotate image or move it.
  • My assumption is that this 3x3 dimension gets to do matrix multiplication with each pixel on your picture.
  • Each pixel will have a matrix of 1x3 dimension:

At the upper left most pixel of the picture. x=0, y=0. 1 is for its math purpose. Nothing comprehensible here. Check my source if you have doubt about the 1.


[0] 
[0]
[1]
  • This matrix is 3x3 dimension.
[1.0 0.0 0.0]

[0.0 1.0 0.0]

[0.0 0.0 1.0]

Which has an explanation as follows:


[MSCALE_X MSKEW_X MTRANS_X]

[MSKEW_Y MSCALE_Y MTRANS_Y]

[MPERSP_0 MPERSP_1 MPERSP_2]
  • Scale X and Scale Y = How will you scale x-axis and y-axis. E.g., Write the 1.5 and you have your picture zoom in 150%
  • Skew X and Skew Y = How will you rotate your picture. This one is some kind of trigonometry. We cannot make sense it in normal programmer language. Good news. We have a proxy method named matrix.setRotation(angle);. And the matrix will add this skew value for you.
  • __Translation X and Translation Y = How will you move your picture horizontally (X) or vertically (Y). E.g., After we rotate picture we should move our picture to the right so that it won't stay at the left side of the screen.
  • Perspective 0, 1, 2 = This is for 3D Image. This is out of our scope here.

Matrix1, Matrix2

Opening your Camera

  • First param is cameraId 0 for back and 1 for front camera
  • Second is CameraDevice.StateCallback which contains callback of CameraDevice.
  • onOpened() onDisconnected() and onError
  • Third is Handler. This can be null. In this case we use it to handle photo saving.
manager.openCamera(mCameraId, mStateCallback, mBackgroundHandler);

CameraDevice.StateCallback

After you open your camera, it won't open right away. You have to handle it with its callback inside. When the camera opened, we create a camerePreview which is rendered on TextureView.

 private final CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() {

       @Override
       public void onOpened(@NonNull CameraDevice cameraDevice) {
           // This method is called when the camera is opened.  We start camera preview here.
           mCameraOpenCloseLock.release();
           mCameraDevice = cameraDevice;
           createCameraPreviewSession();
       }

       @Override
       public void onDisconnected(@NonNull CameraDevice cameraDevice) {
           mCameraOpenCloseLock.release();
           cameraDevice.close();
           mCameraDevice = null;
       }

       @Override
       public void onError(@NonNull CameraDevice cameraDevice, int error) {
           mCameraOpenCloseLock.release();
           cameraDevice.close();
           mCameraDevice = null;
           Activity activity = getActivity();
           if (null != activity) {
               activity.finish();
           }
       }

   };
   ````
   

### createCameraPreviewSession
Camera Preview won't be so simple, we can apply camera effect, adjust exposure and apply them to preview
To build those effects you need a `CaptureRequest` which is built by `CaptureRequest.Builder`

This is done inside CaptureSession.


####HARD LAZY TO STUDY MORE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment