Skip to content

Instantly share code, notes, and snippets.

@Scott31393

Scott31393/imx8mp-media.md

Last active September 9, 2024 14:41
Show Gist options
  • Save Scott31393/077a10024a6058536d3f2fdde476265a to your computer and use it in GitHub Desktop.
Save Scott31393/077a10024a6058536d3f2fdde476265a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
imx8mp-media.md

A Quick Tour into I.MX8MP Media Subsystem

Introduction

Required hw:

ref: AN13430

There are more pipeline allowed from imx8mp media subsystem in this article we present

  • isp pipeline
  • isi pipeline

I.MX8MP ISP Pipeline

The ISP IP cores in the Rockchip RK3399 (known as the "Rockchip ISP1") and in the NXP i.MX8MP have the same origin, and have slightly diverged over time as they are now independently developed (afaik) by Rockchip and VeriSilicon. The latter is marketed under the name "ISP8000Nano", and is close enough to the RK3399 ISP that it can easily be supported by the same driver. [Paul Elder paul.elder@ideasonboard.com]

Tested media topology:

media-ctl --print-dot > graph.dot
dot -Tpng graph.dot > graph.png

Testing ov5640 Camera Sensor on imx8mp-evk

Working on top of Ideas on Board mainline branch ideasonboard/v6.2/isp I collect some patches from mailing list to enable HDMI on imx8mp-evk comes from Pengutronix in particular the following branch pengutronix-imx8mp-hdmi. Cherry-picking a bit I collect all the patchset required to enable ISI + ISP + HDMI on imx8mp into the following branch from Avnet Software and Service team avs-sas tm/media_stage/master/master/alvium_drv Thanks all for the work!

Add ov5640 MINISASTOCSI support

Based on the following two commit Jacopo Mondi debix-a and Laurent Pinchart. We develop the imx8mp-evk-isp0-ov5640.dtbo all the work is done under Linux version 6.3.0-rc4 mainline kernel.

Boot Linux Kernel and .dtbo for imx8mp-evk ov5640 MINISASTOCSI

Boot the kernel using U-Boot with the following cmds to enable minisastocsi ov5640 board support:

setenv serverip 10.0.0.1
setenv ipaddr 10.0.0.100
setenv fdtovaddr 0x430c0000
setenv resize_dim 60000
setenv bootargs console=ttymxc1,115200 debug rootwait root=/dev/nfs ip=10.0.0.100:::::eth0 nfsroot=10.0.0.1:/targetfs,v4,tcp rw
setenv bootcmd "tftp $kernel_addr_r Image; tftpboot $fdt_addr_r imx8mp-evk.dtb; tftp $fdtovaddr imx8mp-evk-isp0-ov5640.dtbo; fdt addr $fdt_addr_r; fdt resize $resize_dim; fdt apply $fdtovaddr; booti $kernel_addr_r - $fdt_addr_r;""

Then

boot

Note: As rootfs image we adopt multimedia-libcamera-image this to enable libcamera stuff into our rootfs using Yocto layer from NXP vendor.

bitbake multimedia-libcamera-image

Configure Media topology using media-ctl

To configure i.mx8mp Media topology in the right way we use the following cmds:

# MEDIA CONFIGURATION

# query sensor color format
v4l2-ctl -d /dev/v4l-subdev3 --list-subdev-mbus-codes 0

# query csis-32e40000.csi color format
v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 1

# query rkisp1_isp color format
v4l2-ctl -d /dev/v4l-subdev0 --list-subdev-mbus-codes 2

# query rkisp1_resizer_mainpath color format
v4l2-ctl -d /dev/v4l-subdev1 --list-subdev-mbus-codes 1

# query rkisp1_mainpath color format
v4l2-ctl --device=/dev/video0 --list-formats

# SETUP TOPOLOGY LINKS
media-ctl --links "'ov5640 1-003c':0->'csis-32e40000.csi':0[1]"
media-ctl --links "'csis-32e40000.csi':1->'rkisp1_isp':0[1]"
media-ctl --links "'rkisp1_isp':2->'rkisp1_resizer_mainpath':0[1]"
media-ctl --links "'rkisp1_resizer_mainpath':1->'rkisp1_mainpath':0[1]"

# SET SGRBG8 FORMAT 640X480
media-ctl --set-v4l2 "'ov5640 1-003c':0[fmt:SBGGR8_1X8/2592x1944@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]"
media-ctl --set-v4l2 "'csis-32e40000.csi':0[fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb quantization:full-range]"
media-ctl --set-v4l2 "'csis-32e40000.csi':1[fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb quantization:full-range]"
media-ctl --set-v4l2 "'rkisp1_isp':0[fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:full-range]"
media-ctl --set-v4l2 "'rkisp1_isp':0[crop:(0,0)/2592x1944]"
media-ctl --set-v4l2 "'rkisp1_isp':2[fmt:YUYV8_2X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range]"
media-ctl --set-v4l2 "'rkisp1_isp':2[crop:(0,0)/2592x1944]"
media-ctl --set-v4l2 "'rkisp1_resizer_mainpath':0[fmt:YUYV8_2X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range]"
media-ctl --set-v4l2 "'rkisp1_resizer_mainpath':0[crop:(0,0)/2592x1944]"
media-ctl --set-v4l2 "'rkisp1_resizer_mainpath':1[fmt:YUYV8_2X8/640x480 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range]"

v4l2-ctl --device /dev/video0 --list-formats-ext
v4l2-ctl --device /dev/video0 --list-framesizes=UYVY

v4l2-ctl --device /dev/video0 --set-fmt-video=width=640,height=480,pixelformat=UYVY

With obtain the following Topology configuration:

root@imx8mp-lpddr4-evk:~# media-ctl -p
Media controller API version 6.2.0

Media device information
------------------------
driver          rkisp1
model           rkisp1
serial
bus info        platform:rkisp1
hw revision     0xe
driver version  6.2.0

Device topology
- entity 1: rkisp1_isp (4 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/2592x1944
                 crop:(0,0)/2592x1944]
                <- "csis-32e40000.csi":1 [ENABLED]
        pad1: Sink
                [fmt:unknown/0x0 field:none]
                <- "rkisp1_params":0 [ENABLED,IMMUTABLE]
        pad2: Source
                [fmt:YUYV8_2X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/2592x1944
                 crop:(0,0)/2592x1944]
                -> "rkisp1_resizer_mainpath":0 [ENABLED]
        pad3: Source
                [fmt:unknown/0x0 field:none]
                -> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

- entity 6: rkisp1_resizer_mainpath (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:YUYV8_2X8/2592x1944 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range
                 crop.bounds:(0,0)/2592x1944
                 crop:(0,0)/2592x1944]
                <- "rkisp1_isp":2 [ENABLED]
        pad1: Source
                [fmt:YUYV8_2X8/640x480 field:none colorspace:rec709 xfer:srgb ycbcr:601 quantization:lim-range]
                -> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE]

- entity 9: rkisp1_mainpath (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "rkisp1_resizer_mainpath":1 [ENABLED,IMMUTABLE]

- entity 13: rkisp1_stats (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "rkisp1_isp":3 [ENABLED,IMMUTABLE]

- entity 17: rkisp1_params (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video2
        pad0: Source
                -> "rkisp1_isp":1 [ENABLED,IMMUTABLE]

- entity 29: csis-32e40000.csi (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb quantization:full-range]
                <- "ov5640 1-003c":0 [ENABLED]
        pad1: Source
                [fmt:SBGGR8_1X8/2592x1944 field:none colorspace:rec709 xfer:srgb quantization:full-range]
                -> "rkisp1_isp":0 [ENABLED]

- entity 34: ov5640 1-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:SBGGR8_1X8/2592x1944@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/2624x1964
                 crop:(0,0)/2624x1952]
                -> "csis-32e40000.csi":0 [ENABLED]

Get some Frames from ov5640 using v4l2-ctl

To get some frames on the first stage bring-up we use v4l2-ctl with the following cmds:

# GET SOME FRAMES
v4l2-ctl --verbose --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=15 --set-fmt-video=width=640,height=480 --stream-to=./frame.raw

# DISPLAY FRAMES CAPTURED
sudo cp /targetfs/home/root/frame.raw .
convert -size 640x480 -depth 16 uyvy:frame.raw frame.png
gimp frame.png

Raw frames can be checke also using YUView tool

Stream Frames from ov5640 using GStreamer

Another way to get fram from ov5640 camera senso is using GStreamer with the following cmd:

gst-launch-1.0 v4l2src  io-mode=dmabuf blocksize=76800 ! video/x-raw,format=UYVY,width=640,height=480,framerate=30/1 ! videoconvert ! fbdevsink sync=false

TODO: Testing Libcamera

Some test are done using Libcamera below some good cmds:

gst-launch-1.0 libcamerasrc camera-name="/base/soc\@0/bus\@30800000/i2c\@30a30000/ov5640\@3c" ! video/x-raw,format=UYVY,width=640,height=480,framerate=30/1 ! videoconvert ! fbdevsink sync=false
cam -c 1 -C --file=frame.raw

I.MX8MP ISI Pipeline

The Image Sensing Interface (ISI) combines image processing pipelines with DMA engines to process and capture frames originating from a variety of sources. The inputs to the ISI go through Pixel Link interfaces, and their number and nature is SoC-dependent. They coverboth capture interfaces (MIPI CSI-2 RX, HDMI RX) and memory inputs. [Laurent Pinchartl laurent.pinchart@ideasonboard.com]

Tested media topology:

media-ctl --print-dot > graph.dot
dot -Tpng graph.dot > graph.png

Testing ov5640 Camera Sensor on imx8mp-evk

Working on top of Ideas on Board mainline branch ideasonboard/v6.2/isi I collect some patches from mailing list to enable HDMI on imx8mp-evk comes from Pengutronix in particular the following branch pengutronix-imx8mp-hdmi. Cherry-picking a bit I collect all the patchset required to enable ISI + ISP + HDMI on imx8mp into the following branch from Avnet Software and Service team avs-sas tm/media_stage/master/master/alvium_drv Thanks all for the work!

Add ov5640 MINISASTOCSI support

Based on the following two commit Jacopo Mondi debix-a and Laurent Pinchart. We develop the imx8mp-evk-isi0-ov5640.dtbo all the work is done under Linux version 6.3.0-rc4 mainline kernel.

Boot Linux Kernel and .dtbo for imx8mp-evk ov5640 MINISASTOCSI

Boot the kernel using U-Boot with the following cmds to enable minisastocsi ov5640 board support:

setenv serverip 10.0.0.1
setenv ipaddr 10.0.0.100
setenv fdtovaddr 0x430c0000
setenv resize_dim 60000
setenv bootargs console=ttymxc1,115200 debug rootwait root=/dev/nfs ip=10.0.0.100:::::eth0 nfsroot=10.0.0.1:/targetfs,v4,tcp rw
setenv bootcmd "tftp $kernel_addr_r Image; tftpboot $fdt_addr_r imx8mp-evk.dtb; tftp $fdtovaddr imx8mp-evk-isi0-ov5640.dtbo; fdt addr $fdt_addr_r; fdt resize $resize_dim; fdt apply $fdtovaddr; booti $kernel_addr_r - $fdt_addr_r;"

Then

boot

Note: As rootfs image we adopt multimedia-libcamera-image this to enable libcamera stuff into our rootfs using Yocto layer from NXP vendor.

bitbake multimedia-libcamera-image

Configure Media topology using media-ctl

To configure i.mx8mp Media topology in the right way we use the following cmds:

# MEDIA CONFIGURATION

# query sensor color format
v4l2-ctl -d /dev/v4l-subdev3 --list-subdev-mbus-codes 0

# query csis-32e40000.csi color format
v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 0

# query crossbar color format
v4l2-ctl -d /dev/v4l-subdev0 --list-subdev-mbus-codes 1

# query mxc_isi.0 color format
v4l2-ctl -d /dev/v4l-subdev1 --list-subdev-mbus-codes 0

# query mxc_isi.0.capture color format
v4l2-ctl -d /dev/video0 --list-formats-ext

# SETUP TOPOLOGY LINKS
media-ctl --links "'ov5640 1-003c':0->'csis-32e40000.csi':0[1]"
media-ctl --links "'csis-32e40000.csi':1->'crossbar':0[1]"
media-ctl --links "'crossbar':3->'mxc_isi.0':0[1]"
media-ctl --links "'mxc_isi.0':1->'mxc_isi.0.capture':0[1]"

# SET UYVY8 FORMAT 1920x1080
media-ctl -d /dev/media0 --set-v4l2 '"ov5640 1-003c":0[fmt:UYVY8_1X16/1920x1080@1/30 field:none]'
media-ctl -d /dev/media0 --set-v4l2 '"csis-32e40000.csi":0[fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]'
media-ctl -d /dev/media0 --set-v4l2 '"crossbar":0[fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]'
media-ctl -d /dev/media0 --set-v4l2 '"mxc_isi.0":0[fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]'

v4l2-ctl --device /dev/video0 --list-formats-ext
v4l2-ctl --device /dev/video0 --list-framesizes=YUYV

v4l2-ctl --device /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=YUYV

With obtain the following Topology configuration:

root@imx8mp-lpddr4-evk:~# media-ctl -p
Media controller API version 6.3.0

Media device information
------------------------
driver          mxc-isi
model           FSL Capture Media Device
serial
bus info        platform:32e00000.isi
hw revision     0x0
driver version  6.3.0

Device topology
- entity 1: crossbar (5 pads, 4 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]
                <- "csis-32e40000.csi":1 [ENABLED,IMMUTABLE]
        pad1: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
        pad2: Sink
                <- "mxc_isi.output":0 [ENABLED,IMMUTABLE]
        pad3: Source
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]
                -> "mxc_isi.0":0 [ENABLED,IMMUTABLE]
        pad4: Source
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
                -> "mxc_isi.1":0 [ENABLED,IMMUTABLE]

- entity 7: mxc_isi.0 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg
                 compose.bounds:(0,0)/1920x1080
                 compose:(0,0)/1920x1080]
                <- "crossbar":3 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:YUV8_1X24/1920x1080 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/1920x1080
                 crop:(0,0)/1920x1080]
                -> "mxc_isi.0.capture":0 [ENABLED,IMMUTABLE]

- entity 10: mxc_isi.0.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video0
        pad0: Sink
                <- "mxc_isi.0":1 [ENABLED,IMMUTABLE]

- entity 18: mxc_isi.1 (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev2
        pad0: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
                 compose.bounds:(0,0)/1920x1080
                 compose:(0,0)/1920x1080]
                <- "crossbar":4 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:YUV8_1X24/1920x1080 field:none colorspace:jpeg xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/1920x1080
                 crop:(0,0)/1920x1080]
                -> "mxc_isi.1.capture":0 [ENABLED,IMMUTABLE]

- entity 21: mxc_isi.1.capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "mxc_isi.1":1 [ENABLED,IMMUTABLE]

- entity 29: mxc_isi.output (1 pad, 1 link)
             type Node subtype V4L flags 0
        pad0: Source
                -> "crossbar":2 [ENABLED,IMMUTABLE]

- entity 36: csis-32e40000.csi (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]
                <- "ov5640 1-003c":0 [ENABLED]
        pad1: Source
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:jpeg]
                -> "crossbar":0 [ENABLED,IMMUTABLE]

- entity 41: ov5640 1-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev4
        pad0: Source
                [fmt:UYVY8_1X16/1920x1080@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range
                 crop.bounds:(0,0)/2624x1964
                 crop:(336,434)/1952x1088]
                -> "csis-32e40000.csi":0 [ENABLED]

Get some Frames from ov5640 using v4l2-ctl

To get some frames on the first stage bring-up we use v4l2-ctl with the following cmds:

# GET SOME FRAMES
v4l2-ctl --verbose --device /dev/video0 --stream-mmap --stream-to=frame.raw --stream-count=15 --set-fmt-video=width=1920,height=1080 --stream-to=./frame.raw

# DISPLAY FRAMES CAPTURED
sudo cp /targetfs/home/root/frame.raw .
convert -size 1920x1080 -depth 16 uyvy:frame.raw frame.png
gimp frame.png

Raw frames can be checke also using YUView tool

Stream Frames from ov5640 using GStreamer

Another way to get fram from ov5640 camera sensor is using GStreamer with the following cmd:

gst-launch-1.0 v4l2src  io-mode=dmabuf blocksize=76800 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! videoconvert ! fbdevsink sync=false

TODO: Testing Libcamera

Some test are done using Libcamera below some good cmds:

gst-launch-1.0 libcamerasrc camera-name="/base/soc\@0/bus\@30800000/i2c\@30a30000/ov5640\@3c" ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! videoconvert ! fbdevsink sync=false
cam -c 1 -C --file=frame.raw

Some References:

ref https://github.com/IENT/YUView/releases/tag/v.2.13
ref https://www.marcusfolkesson.se/blog/v4l2-and-media-controller/
ref http://trac.gateworks.com/wiki/linux/v4l2#capturingvideoframes
ref https://www.linux4sam.org/bin/view/Linux4SAM/MediaController
ref https://www.kernel.org/doc/html/v5.0/media/uapi/v4l/crop.html
ref https://www.phytec.de/no/cdocuments/?doc=IwCgG
ref https://www.kernel.org/doc/html/v5.0/media/kapi/mc-core.html
ref https://www.kernel.org/doc/html/v4.8/media/uapi/v4l/colorspaces.html
ref https://git.libcamera.org/libcamera/jmondi/libcamera.git

@Scott31393
Copy link
Author

graph

@Scott31393
Copy link
Author

image

@Scott31393
Copy link
Author

image

@Scott31393
Copy link
Author

graph-isi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment