video: ov5640 support for vga , qvga and qqvga not there

[adityaganesh]

Describe the bug

The current driver for 0v5640 does not support vga , qvga or qqvga resolutions

Impact
The ov5640 camera supports the above mentioned resolutions but the driver does not support so using this driver with microcontrollers which have ram lesser than 614.4kB in its current state is not possible

Environment (please complete the following information):

• Windows 11
• Zephyr SDK release version 4.0

[adityaganesh]

Looks like the issue is resolved in main branch but not committed to the release branch of 4.0

[KurtE]

Quick note: Today I started experimenting to see if I could get this camera to work with the ArduinoCore-zephyr project, in my case
I was trying to configure it for a Portenta H7...

Note: The main camera code in camera.cpp, is looking at the table of Width, Height, Format for a match. And the current stuff up
in main still only has:

static const struct video_format_cap csi2_fmts[] = {
	OV5640_VIDEO_FORMAT_CAP(1280, 720, VIDEO_PIX_FMT_RGB565),
	OV5640_VIDEO_FORMAT_CAP(1280, 720, VIDEO_PIX_FMT_YUYV),
	OV5640_VIDEO_FORMAT_CAP(640, 480, VIDEO_PIX_FMT_RGB565),
	OV5640_VIDEO_FORMAT_CAP(640, 480, VIDEO_PIX_FMT_YUYV),
	{0}};

Whereas the OV2640 has a much more complete table:

static const struct video_format_cap fmts[] = {
	OV2640_VIDEO_FORMAT_CAP(160, 120, VIDEO_PIX_FMT_RGB565),   /* QQVGA */
	OV2640_VIDEO_FORMAT_CAP(176, 144, VIDEO_PIX_FMT_RGB565),   /* QCIF  */
	OV2640_VIDEO_FORMAT_CAP(240, 160, VIDEO_PIX_FMT_RGB565),   /* HQVGA */
	OV2640_VIDEO_FORMAT_CAP(240, 240, VIDEO_PIX_FMT_RGB565),   /* 240x240 */
	OV2640_VIDEO_FORMAT_CAP(320, 240, VIDEO_PIX_FMT_RGB565),   /* QVGA  */
	OV2640_VIDEO_FORMAT_CAP(352, 288, VIDEO_PIX_FMT_RGB565),   /* CIF   */
	OV2640_VIDEO_FORMAT_CAP(640, 480, VIDEO_PIX_FMT_RGB565),   /* VGA   */
	OV2640_VIDEO_FORMAT_CAP(800, 600, VIDEO_PIX_FMT_RGB565),   /* SVGA  */
	OV2640_VIDEO_FORMAT_CAP(1024, 768, VIDEO_PIX_FMT_RGB565),  /* XVGA  */
	OV2640_VIDEO_FORMAT_CAP(1280, 1024, VIDEO_PIX_FMT_RGB565), /* SXGA  */
	OV2640_VIDEO_FORMAT_CAP(1600, 1200, VIDEO_PIX_FMT_RGB565), /* UXGA  */
	OV2640_VIDEO_FORMAT_CAP(160, 120, VIDEO_PIX_FMT_JPEG),     /* QQVGA */
	OV2640_VIDEO_FORMAT_CAP(176, 144, VIDEO_PIX_FMT_JPEG),     /* QCIF  */
	OV2640_VIDEO_FORMAT_CAP(240, 160, VIDEO_PIX_FMT_JPEG),     /* HQVGA */
	OV2640_VIDEO_FORMAT_CAP(320, 240, VIDEO_PIX_FMT_JPEG),     /* QVGA  */
	OV2640_VIDEO_FORMAT_CAP(352, 288, VIDEO_PIX_FMT_JPEG),     /* CIF   */
	OV2640_VIDEO_FORMAT_CAP(640, 480, VIDEO_PIX_FMT_JPEG),     /* VGA   */
	OV2640_VIDEO_FORMAT_CAP(800, 600, VIDEO_PIX_FMT_JPEG),     /* SVGA  */
	OV2640_VIDEO_FORMAT_CAP(1024, 768, VIDEO_PIX_FMT_JPEG),    /* XVGA  */
	OV2640_VIDEO_FORMAT_CAP(1280, 1024, VIDEO_PIX_FMT_JPEG),   /* SXGA  */
	OV2640_VIDEO_FORMAT_CAP(1600, 1200, VIDEO_PIX_FMT_JPEG),   /* UXGA  */
	{ 0 }
};

Will try more with 2640, but wish the OV5640 was updated.

Thanks.

[josuah]

I think I missed this message earlier!

The OV5640 has a built-in GPU that pemits to emmit JPEG data directly, with an integrated auto-focus, auto-exposure, auto-white-balance, which makes it a popular image sensor for makers and firmware developers, who can skip all the effort of doing all this image post processing on a low-power platform.

This however makes the OV5640 very complex (and in turn increases noise on the output image), and makes it the biggest image sensor driver of all Linux, so it is normal that it appears particularly complex.

~/Code/Linux/drivers/media/i2c master.$ wc -l *.c | sort -nr | head
   4058 ds90ub960.c <- not an image sensor, FDP-Link
   4031 ov5640.c <- this one
   3697 adv7604.c
   3690 ov64a40.c <- 64 MP image sensor
    [...]

Still, the basics are:

1. Undocumented registers permits to select a "mode of operation" that determines the base resolution and framerate
2. On top of this, extra configuration permits to control further the extra resolution.

The Zephyr drivers seems to combine them in just one big block without annotation:

zephyr/drivers/video/ov5640.c

Lines 469 to 474 in ed45960

	static const struct ov5640_reg csi2_low_res_params[] = {
	{0x3800, 0x00}, {0x3801, 0x00}, {0x3802, 0x00}, {0x3803, 0x04}, {0x3804, 0x0a},
	{0x3805, 0x3f}, {0x3806, 0x07}, {0x3807, 0x9b}, {0x3808, 0x02}, {0x3809, 0x80},
	{0x380a, 0x01}, {0x380b, 0xe0}, {0x380c, 0x07}, {0x380d, 0x68}, {0x380e, 0x03},
	{0x380f, 0xd8}, {0x3810, 0x00}, {0x3811, 0x10}, {0x3812, 0x00}, {0x3813, 0x06},
	{0x3814, 0x31}, {0x3815, 0x31}, {0x3824, 0x02}, {0x460c, 0x22}};

One way would be to directly add the extra registers for the missing resolution by looking at the now-public datasheet.

One way would be to first document the registers (with #define REG_THIS 0x2345) to help writing a new one.

[josuah]

For Portenta and very small resolutions, there is also the HM01B0 image sensor, 320x320 array, available on a shield.

It is not supported yet, but offers an alternative.
I am interested in porting it to Zephyr, but cannot give any time frame yet.

[ngphibang]

In fact, the ov5640 supports 2 modes:

• DVP (parallel) mode : in this mode, QVGA (320x240) and QQVGA (160x120) are recently supported (from Zephyr 4.1).
• CSI2 mode : It is in our plan to support QVGA and QQVGA in the near future

[KurtE]

For Portenta and very small resolutions, there is also the HM01B0 image sensor, 320x320 array, available on a shield.

I believe their current V2 shield now has the HM0360 on it. I believe both of these are gray scale cameras. For
lower resolution color cameras there are also OV767x cameras. Currently I believe that Zephyr only has support in it
for the OV7670 and not the OV7675, which Arduino currently sells for the Portenta and GIGA.

There is support in the Arduino library arducam_dvp, for both of these Monochrome cameras as well as OV7670 and OV7675 on the MBED releases for these boards. Hopefully should not be too hard to hopefully migrate this support into this code base.

In fact, the ov5640 supports 2 modes:

• DVP (parallel) mode : in this mode, QVGA (320x240) and QQVGA (160x120) are recently supported (from Zephyr 4.1).
• CSI2 mode : It is in our plan to support QVGA and QQVGA in the near future
  Will try out some of these modes soon, once the Arduino beta setup for zephyr is setup for Zephyr 4.1 (or beyond).

Side notes about the OV5640/OV2640 for setups, wishing to use smaller memory. There are a couple of other capabilities of these
cameras, that are useful. The ability to capture images using JPEG. The other is the ability to define a window of the image to be returned to the processor. One might be able to pan and zoom through the image

A few of us have played around with some of these capabilities on Teensy 4.x board (mainly T4.1 and MicroMod or custom).
The code is up at: https://github.com/mjs513/ArduCAM_t4

[ngphibang]

QVGA and QQVGA in CSI2 mode is now done in #87911.

The ability to capture images using JPEG.

I asked @CharlesDias last time if he can add support for JPEG as I don't have HW which can receive JPEG.

A few of us have played around with some of these capabilities on Teensy 4.x board (mainly T4.1 and MicroMod or custom).
The code is up at: https://github.com/mjs513/ArduCAM_t4

It would be nice if you can upstream these code to support these features

[KurtE]

A few of us have played around with some of these capabilities on Teensy 4.x board (mainly T4.1 and MicroMod or custom).
The code is up at: https://github.com/mjs513/ArduCAM_t4

It would be nice if you can upstream these code to support these features

Oops, I actually put the wrong link in for our camera stuff, it was:
https://github.com/mjs513/Teensy_Camera

Porting this code over, is currently beyond my abilities using zephyr. My experience so far has been testing and trying to contribute
to the Arduino conversion of some of their boards like (Giga, Portenta H7, Portenta C33) over to Zephyr.

The Teensy_Camera code, which @mjs513 and I (and maybe a few others) modified and extended code originally by:
Copyright (c) 2013-2021 Ibrahim Abdelkader [email protected] Copyright (c) 2013-2021

Background on this code: - sorry in advance, probably way too much information, but...

Was done to run using Arduino, with the PJRC Teensy boards. Our Lower-level code for reading data in from the different cameras depended on which of the Teensy boards we were running it on. With all of them (T4, T4.1 and Micromod) as well as a couple of custom boards. we have support and settings to do reads using GPIO registers.

With the T4.1 it uses the CMOS Sensor Interface(CSI) in 4 bit mode as enough of the CSI capable pins were exported on this board.
We can do this, either using polling or DMA.

Supporting JPEG in polling mode is pretty easy to look for the End Marker. In DMA mode it is more difficult. We have hacks, in that we chain multiple DMA setting objects, and Interrupt as each one completes, and then we try to scan this buffer.

With the MicroMod - We are using FlexIO pin and defined a FlexIO sequence that more or less emulates the CSI. Like CSI, easier do wo when we poll versus using DMA.

With the T4 - Use GPIO1 and maybe FlexIO3 (Does not support DMA), and bits are not all consecutive, so have to manipulate each item as it is read in.

Porting to Zephyr: As I mentioned beyond my Zephyr knowledge. I should probably try programming some boards directly using zephyr. Looking at the Teensy Page (https://docs.zephyrproject.org/latest/boards/pjrc/teensy4/doc/index.html) I don't get the impression that there is currently support for the subsystems I mentioned (CSI, FlexIO). I Could be wrong, but...

[josuah]

I am very glad to see this image sensor implementation effort! Also, thank you for explaining the history and rationale behind the project.

I just realized that you meant CSI as Camera Sensor Interface which OmninVision calls DVP, not MIPI-CSI (Camera Serial Interface) which is much more costly and higher speed. Not sure about FlexIO support in Zephyr yet.

I wanted to test #76124 with the MCXN947 (a different NXP chip with DVP support), but we have some work to do first. :) I ordered an Arduino Giga.

As I mentioned beyond my Zephyr knowledge.

I just opened a PR trying to simplify new image sensor driver contributions, so that it requires fewer Zephyr knowledge, and facilitate reviews. This is a proposal, no idea if that will land on Zephyr or when, but I this helps with porting more things.

When I have time, I will try converting a few sensors to this simplified model as an example, and ping you back.

• drivers: video: imager: move the boilerplate in a library #88003

P.S. I misclicked, did not mean to close it at all!

[josuah]

By the way, you might soon be able to have the devboard detected as a regular webcam, which means no need for a custom software to extract the images. Just opening the video stream with any webcam app. This should help image sensor driver development too. This was tested with the GC2145 on the Arducam Nicla.

Just like the awesome OpenMV team did lately.

[ngphibang]

I wanted to test #76124 with the MCXN947 (a different NXP chip with DVP support), but #80514 (comment). :) I ordered an Arduino Giga.

I will try to complete the fix on MCXN947 this weekend, thanks for reminding.

I just opened a PR trying to simplify new image sensor driver contributions, so that it requires fewer Zephyr knowledge, and facilitate reviews. This is a proposal, no idea if that will land on Zephyr or when, but I this helps with porting more things.

Thanks for adding me in this PR as well as some others. I noticed that recently, the zephyr-bot didn't add me as a reviewer anymore for all recent video-related PRs, don't really know the reason :(.