Skip to content

ports/stm32: Dual CSI support (for N6). #2691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

Conversation

iabdalkader
Copy link
Member

@iabdalkader iabdalkader commented May 16, 2025

A first draft of dual-CSI support (see #2676). This is based on N6 support, see commits starting at imlib/framebuffer: Add support for a second frame buffer.

The main changes are:

  • Support for defining a second framebuffer. The main/default FB is used for drawing, fb_alloc etc...
  • All CSI functions now accept an instance with the exception of clock config, which can be made an op and passed an instance too.
  • A new CSI API (highly subject to change) is introduced. The API allows using the main CSI, secondary CSI or both at the same time, with the ability to control which one updates the JPEG buffer.
  • The old sensor module is still supported and both the new and old APIs are enabled.
  • Note to minimize changes, only the enable sensors were fixed/updated

I've enabled Lepton, both PixArt sensors and Soft-CSI. Lepton + PAG should work, but it's untested. If a single sensor is connected, soft-csi gets added automatically, so you could run the following example as is:

import time
import csi

csi0 = csi.CSI()
csi0.reset(hard=True)
csi0.pixformat(csi.RGB565)
csi0.framesize(csi.QVGA)
csi0.snapshot(time=100)

csi1 = csi.CSI(cid=csi.SOFTCSI)
csi1.reset(hard=False)
csi1.pixformat(csi.RGB565)
csi1.framesize(csi.QVGA)
csi0.snapshot(time=100)

clock = time.clock()
while True:
    clock.tick()
    img0 = csi0.snapshot()
    img1 = csi1.snapshot(update=False)
    img0.draw_image(img1, 0, 0, x_scale=0.5, y_scale=0.5)
    print(clock.fps())

Technically, you could also enable 3 CSIs, but you'll need a third, statically-defined FB. So I'm considering making FBs dynamically allocated, possibly only for secondary CSIs. This will also fix the FB size issue. Thoughts?

Also, I'm considering replacing most of the functions with attributes. For example:

csi0 = csi.CSI()
csi0.reset(hard=True)
csi0.pixformat = csi.RGB565
csi0.framesize = csi.QVGA
csi0.window = (0, 0, 100, 100)
csi0.delays = False
csi0.snapshot(time=100)

They're smaller, faster, easier to maintain. Thoughts?


A note about the ops: More functions have been made ops (such as, abort, config and snapshot).

1- In port's main CSI driver (for example: ports/omv_csi.c), the main driver's ops are initially set as defaults:

for (size_t i=0; i<OMV_CSI_MAX_DEVICES; i++) {
    omv_csi_t *csi = &csi_all[i];

    memset(csi, 0, sizeof(omv_csi_t));
    csi->fb = framebuffer_get(i);
    csi->i2c_bus = &i2c_bus;

    csi->abort = stm_csi_abort;         // main driver's abort (disables DCMI, DMA etc..)
    csi->config = stm_csi_config;       // main driver's config 
    csi->snapshot = stm_csi_snapshot;   // main driver's snapshot.

    csi->color_palette = rainbow_table;
}

This allows something like Lepton and Soft-CSI to set abort/config to NULL, which will not configure DCMI/DMA and will do the general abort() part but Not disable DMA/DCMI. These driver were never supposed to call stm32's config/abort anyway, because they don't use DCMI/DMA, but now with dual-csi config, these structs will not even be initialized for the secondary/aux sensors.

While we're at it, made main driver's snapshot an op too. Drivers like boson that need to both override the main snapshot, and still call it later, can get the main snapshot by stashing the pointer in their init functions:

typedef int (*omv_csi_snapshot_t)(omv_csi_t *csi, image_t *image, uint32_t flags);

static int snapshot(omv_csi_t *csi, image_t *image, uint32_t flags) {
    // Call `csi->priv()` to get an image using the main driver's snapshot.
    int ret = ((omv_csi_snapshot_t) csi->priv)(csi, image, flags);
    ....
}

int boson_init(omv_csi_t *csi) {
    // Initialize csi structure
    csi->reset = reset;
    csi->set_pixformat = set_pixformat;
    //etc...

    csi->priv = csi->snapshot
    csi->snapshot = snapshot;

@kwagyeman
Copy link
Member

Technically, you could also enable 3 CSIs, but you'll need a third, statically-defined FB. So I'm considering making FBs dynamically allocated, possibly only for secondary CSIs. This will also fix the FB size issue. Thoughts?

Yes, that's perfect! Also, with the new information we received from ST about the AI hyperRAM not needed to be pegged we can just make the heap even bigger.

Note, to support 5MP camera sensors we want the framebuffer+fb_alloc to hold 20MB+.

Also, I'm considering replacing most of the functions with attributes. For example:

Prefer functions. The new pythonic methods are perfect.

While we're at it, made main driver's snapshot an op too. Drivers like boson that need to both override the main snapshot, and still call it later, can get the main snapshot by stashing the pointer in their init functions:

Nice.

@iabdalkader iabdalkader force-pushed the n6_port_dual_csi branch 3 times, most recently from 9d6aec1 to d622087 Compare May 19, 2025 10:32
Signed-off-by: iabdalkader <[email protected]>
- Fix I2C bus.
- Override default abort/config.
- Set aux flag for dual-csi config.

Signed-off-by: iabdalkader <[email protected]>
A modern API that supports multiple CSI instances.
The API is a work in progress and is not backwards
compatible, but it can be enabled and used at the
same time as the old "sensor" module.

Signed-off-by: iabdalkader <[email protected]>
- Temporary PR, drop later.

Signed-off-by: iabdalkader <[email protected]>
@iabdalkader iabdalkader added this to the v5.0.0 milestone Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants