IMX412-FPV

tom

@AidanGallagher If you want to try using the imx412-fpv sensor default settings you can modify the high level config file for whatever camera config you're using in /usr/share/modalai/voxl-camera-server/standard_configs/platform_config_* and change the sensor type from imx412 to imx412-fpv and run voxl-configure-cameras N where N is the camera config you're using

AidanGallagher

Cheers @tom ill give it a try

AidanGallagher

@tom I was giving imx412-fpv-misp a try and was wondering can the resolution of the stream be increased?
Changing the values in the voxl-camera-server.conf file results in a corrupted stream

tom

@AidanGallagher If you run voxl-camera-server -l you should be able to see a list of supported resolutions

AidanGallagher

@tom I have done that are there are three potential resolutions.
However, only the 1936x1080 produces a valid stream.

When I capture a raw bayer at 4056x3040 then the majority of the image is black. It is as if the 1080p image is put into a 4k container.

Any ideas what I could be doing wrong?

Alex Kushleyev

Hi @AidanGallagher , can you please clarify what resolution and fps you are trying to achieve using the "fpv" version of the IMX412 drivers? Please note this version of drivers is still "experimental" but I can help you get the results that you need.

When using MISP (Modalai ISP) pipeline, the 4056 resolution currently does not work (long story). But we have a workaround for using 4040x3040, which is just 16 pixel crop from the full frame. If that would work for you, you should pick up the latest drivers for MISP here :

https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250311/imx412_fps_eis_20250311_drivers.zip

You will simply need to replace the sensormodule file for the slot number that you have for your imx412 (in /usr/lib/camera/). and might as well update the .so that's in the zip to overwrite the existing one in the same location)

The following RAW resolutions are supported by these drivers:

   - 4056x3040 @ 30                     16.5ms readout time
   - 4040x3040 @ 30, **60**             16.5ms readout time
   - 3840x2160 @ 30, 60, 80             11.8ms readout time
   - 1996x1520 @ 30, 60, 120            5.5ms readout time
   - 1936x1080 @ 30, 60, 90, 120, 240   4.0ms readout time
   - 1996x480  @ 30, 480                1.8ms readout time
   - 1996x240  @ 30, 800                0.9 - 1.0 ms readout time

*** Please note 4040x3040 60FPS is not stable, please use slightly lower FPS like 55 or even 57-58 should be OK if needed.

1996x1520 is the 2x2 binned resolution of full frame with a small horizontal crop to make it compatible with MISP.

The readout times are noted as they are important if you are concerned with overall latency.

These drivers are configured to use IMX412 at its maximum throughput of 2.1Gbps per MIPI lane (4 lanes), which ensures the lowest latency and maximum possible FPS.

When you use this driver, please set the following value in /vendor/etc/camera/camxoverridesettings.txt - this is a workaround for camera pipeline not showing all possible RAW resolutions / fps.

maxRAWSizes=20

Also, we recently (I believe SDK 1.4.1) disabled debug prints from camera pipeline to logcat, so if you wanted to confirm that correct resolution is selected (see below), enable the following (in the same file), if the param is set to 0 (need to set to 1):

systemLogEnable=1

After confirming the resolution, you can set the param back to 0 to reduce cpu usage.

You should not use 4056x4030, but instead use 4040x3040. if you use 1936x1080, then you probably want to specify misp_width as 1920, not 1936. The reason for 1936 is the same as for 4056 -- MISP does not work with 1920, but we just crop 1936 down to 1920. There are no performance issues associated with this.

Please note the 480 and 800fps modes are experimental, may cause camera server to crash, depending on what you are doing (especially 800fps). also if you try to stream these high fps to voxl-portal, you will have really large frame drop, but you will see some frames. (maybe up to 120fps, since frames are small). For high fps modes, set cpu to perf using voxl-set-cpu-mode perf.

Finally, for best performance you should use the latest camera server from our nightly builds, if you do not, please specify which version you are using and I can check if there are any relevant issues related to what you are trying to do.

In camera server config, you should specify one of the above resolutions as your preview_width and preview_height and misp_width and misp_height can actually be pretty much any resolution you want. Based on the misp resolution that you request, the source image will be down-scaled and cropped to match your dimensions and maintain the proper aspect ratio (no stretching, etc).

Finally, if you are experimenting with different modes and FPS, and the image you get looks completely wrong (zigzags, etc), you should verify that the camera pipeline indeed selects correct resolution by running the following before you start camera server:

this will show something like this:

logcat | grep -i selected
03-19 20:26:09.694  1894  1894 I CHIUSECASE: [CONFIG ] chxsensorselectmode.cpp:635 FindBestSensorMode() Selected Usecase: 7, SelectedMode W=4040, H=3040, FPS:30, NumBatchedFrames:0, modeIndex:0
03-19 20:26:09.694  1894  1894 I CHIUSECASE: [CONFIG ] chxpipeline.cpp:371 CreateDescriptor() Pipeline[Raw] Pipeline pointer 0x55ae2d6620 Selected sensor Mode W=4040, H=3040
03-19 20:26:09.694  1894  1894 I CamX    : [CONFIG][CORE   ] camxsession.cpp:5907 SetRealtimePipeline() Session 0x55ae411ea0 Pipeline[Raw] Selected Sensor Mode W=4040  H=3040

if the selected resolution is not what you reqested as preview (this may happen if you select 1996x240 resolution but choose FPS equal or below 480, the pipeline will choose 1996x480 resolution (because it is closer in FPS...), which may cause the board to crash (because buffers are allocated for 1996x240 resolution). So if you use use high FPS modes, either use the exact FPS that i listed or use FPS above the the higher resolution modes. I hope that makes sense.. This is an issue we are going to work out soon. If you want to avoid crashing the board (which is temporary, just power cycle to recover). you should start the camera server but do not have any clients for the hires camera stream, which would not actually start the camera streaming, but you will see the resolution selected and can stop the camera server before it would cause a crash.

In the near future we will update these experimental binaries and make them the default, as they are backwards compatible with existing drivers that are shipped with the SDK.

Alex

AidanGallagher

@Alex-Kushleyev thanks a million for this information (and the beta drivers) Alex it is very helpful.

I am using voxl-camera-server version 2.1.1 (voxl-suite 1.4.1)
From your information I will try 3840x2160 and 4040x3040 at 60fps to see what the performance is like.

Aidan

Alex Kushleyev

@AidanGallagher , you are welcome.

Just one note, if you are going to try 4040x3040 maximum fps, please choose something slightly under 60, as 60 is sometimes causing issues due to very little time between end of frame and start of frame, but even 55 is going to be stable. When I use this resolution, i test at 50 or 55FPS. I will look into 60FPS at full resolution a bit later, but may need to cut down on the vertical size a bit to give some time between EOF and SOF.

Alex

AidanGallagher

@Alex-Kushleyev understood,

Thanks again I will give this a try - fingers crossed

Alex Kushleyev

@AidanGallagher , I have updated the IMX412 drivers to support stable 4040x3040 60FPS.

Just for full disclosure, the camera in this mode is slightly overclocked from 2.1Gbps / lane to 2.2Gbps MIPI bit rate (voxl2 supports up to 2.5Gbps) in order to allow sufficient time between EOF and SOF. This is within 5% of the camera spec and seems to work fine.. If you do not feel comfortable using this mode, don't :). (only the specific 4040x3040 60fps mode has been changed).

We are going to continue testing this mode because it allows doing things like EIS at 4K60 without losing any vertical stabilization margin. If there are any issues with it, I will report back.

By the way, for best image quality, even if you don't need 4K output, but lets say 1080p, in the case of unstabilized video (no EIS), you should select 3840x2160 input and 1920x1080 output. The debayering process loses sharpness, so going from 1920x1080 RAW -> 1920x1080 RGB is going to be worse quality than 3840x2160 RAW -> 1920x1080 RGB. For unstabilized video you do not need the full frame because the top and bottom will be cropped out anyway, so reduce the waste and not select the full frame resolution (4040x3040). You can experiment using the different input resolutions to see what output quality (sharpness) you get. There is currently no option to do output sharpening when using MISP, but we will look into in the future. The down side of using higher resolution input is increased GPU usage (should not be an issue) and longer readout time (the rolling shutter artifacts will be more pronounced).

https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250409/imx412_fps_eis_20250409_drivers.zip

   - 4056x3040 @ 30                     16.5ms readout time
   - 4040x3040 @ 30, 60                 16.5ms readout time
   - 3840x2160 @ 30, 60, 80             11.8ms readout time
   - 1996x1520 @ 30, 60, 120            5.5ms readout time
   - 1936x1080 @ 30, 60, 90, 120, 240   4.0ms readout time
   - 1996x480  @ 30, 480                1.8ms readout time
   - 1996x240  @ 30, 800                0.9 - 1.0 ms readout time