Understanding vimc

vimc.md

• When you configure /dev/videoX, it is always the memory layout (thus the pixel format), when configuring /dev/v4l2-subdevX, you configure the bus format (but in virtual drivers, buses doesn't exist, because it is all emulated, there is no physical bus)
• Hence, since VIMC is a virtual driver, we don't work with mbus codes. For example: SGRBG8_1X8 is associated with a pixelformat, that is used by the sensor, and we can see that by checking vimc_pix_map_list[]

   {
   .code = { MEDIA_BUS_FMT_SGRBG8_1X8 },
   .pixelformat = V4L2_PIX_FMT_SGRBG8,
   .bpp = 1,
   .bayer = true,
   },
  

-Function vimc_sen_tpg_s_format()..tpg is the test-pattern-generator (tpg generates the image)..and we set which pixelformat we want the tpg to generate in the sensor, using this: tpg_s_fourcc(&vsen->tpg, vpix->pixelformat);

• It doesn't make much sense for the debayer to produce alpha, because in the raw bayer formats there is no alpha

• Each entity has to implement 3 callbacks:

  1. "add" : this registers the entity when the driver is probed
  2. release: this free space allocted for the entity when the driver is removed
  3. process_frame: when the streaming starts, each entity process the frame, so it recieve an input frame and produce an output frame.for example, the scaler scales the frame
  4. the 'process_frame' of the capture entity just sends the frame to userspace by calling vb2_buffer_done

• formats in each of vimc's entities is configured using: s_fmt callback: https://www.kernel.org/doc/html/v5.6/media/uapi/v4l/vidioc-subdev-g-fmt.html#c.VIDIOC_SUBDEV_S_FMT

Streamer:

• vimc_stream: represents stream
• pipe: pipeline obj associated
• ved_pipeline: array containing all the entities participating in the stream -->order: video device (ex capture) where stream_on is called, to sensor.
• kthread: thread that generates the frames of the stream.
• vimc_get_source_entity: Gives source connected to sink -->pipeline_init calls it to get the entire pipepline backwards from capture's sink and initialises it.
• vimc_streamer_pipeline_terminate: takes in vimc_stream and disables the stream in each entity of the pipeline in given stream.
• vimc_streamer_pipeline_init - Initializes the stream structure using ved (the pointer to the vimc entity initializing the stream) and pointer to the stream structure to be initialized.
• vimc_streamer_thread - Process frames through the pipeline (takes in current stream){ From the source to the sink, gets a frame from each subdevice and send to the next one of the pipeline at a fixed framerate.} --> runs forever (for ( ( ; ; ) )
• vimc_streamer_s_stream - Start/stop the streaming on the media pipeline --> Catpture calls it to start streaming, and the stream gets passed on the basis of capture device

Core:

• vimc_ent_link: describes links between entities --> source entity, source pad, sink entity, sink pad, flag
• vimc_pipeline_config: describes the whole topology --> vimc_ent_config (defining topology), num_ents (number of entities), links, number of links.
• vimc_create_links: initialises links between entities

Common:

• struct vimc_pix_map - maps media bus code with v4l2 pixel format --> When talking about memory layout, we talk about pixelformat, and when talking about subdevices configuration, we talk about media bus format. So, in a virtual driver, we always map media bus format to pixel formats as actual subdevices are not present, we can just define memory layout (pixelformat)
• Validate() validates for all the capture nodes ( we can see that it outputs for RAW capture 0 and RGB/YUV capture but not for Raw capture 1, which shows that one sensor can have a single stream validated for now..[before multi stream implementation])

Debayer:

The sensor of a camera is composed my many color sensors, each of this color sensors just captures a single color. Thus a sensor may be composed with many color sensors in the order Red and Green in the odd lines and Green and Blue in the even lines like this --

|Red Sensor.....| Green Sensor.|Red Sensor.....| Green Sensor.|
|Green Sensor..| Blue Sensor...|Green Sensor..| Blue Sensor...|

This is called RGGB Usually our eye are more sensible to green, that is why we are te double of green sensors compared to the other color components

Debayer algo:

To get each color component of each pixel we will calculate the mean of the color components around a given color sensor, i.e. a Blue color sensor is inside a 3x3 square (or mean window):

R G R
G B G
R G R

The Red component will be the mean of all four Red components in this square, the Green component will be the man of all four Green component in this square and the Blue is the value read from the Blue sensor being evaluated. The mean window can be any SxS size, where S is an odd value to allow the pixel being evaluated to be in the center of the square.

• So the sensor actually generates a frame with V4L2_PIX_FMT_SGRBG8, and the debayer should transform it to BGR888_1X24.
• Now, lets see how the debayer maps the pixels from V4L2_PIX_FMT_SGRBG8 to MEDIA_BUS_FMT_BGR888_1X24:
  • vimc_deb_calc_rgb_sink() function has this line that gets one specific color from the sink image (SGRBG8) :

     /* Check which color this pixel is */  
     color = vdeb->sink_pix_map->order[wlin % 2][wcol % 2];  
    

  • this sink_pix_map[] array defines the layout of the sink image (in this case V4L2_PIX_FMT_SGRBG8)..lets see where this is assigned:

     /* Get the corresponding pixel map from the table */  
     vdeb->sink_pix_map = vimc_deb_pix_map_by_code(vdeb->sink_fmt.code);  
    

• Now, we see that it takes an entry from the array vimc_deb_pix_map_list[]
  .code = MEDIA_BUS_FMT_SGRBG8_1X8,
  .order = { { VIMC_DEB_GREEN, VIMC_DEB_RED },
  { VIMC_DEB_BLUE, VIMC_DEB_GREEN } }