IMX412-FPV

AidanGallagher

Hi,

I see in the voxl camera server gitlab project it mentions imx412-fpv camera - what is required to enable this for testing?
It doesn't appear as an option on the voxl-configure-camera script.

tom

@AidanGallagher If you want to try using the imx412-fpv sensor default settings you can modify the high level config file for whatever camera config you're using in /usr/share/modalai/voxl-camera-server/standard_configs/platform_config_* and change the sensor type from imx412 to imx412-fpv and run voxl-configure-cameras N where N is the camera config you're using

AidanGallagher

Cheers @tom ill give it a try

AidanGallagher

@tom I was giving imx412-fpv-misp a try and was wondering can the resolution of the stream be increased?
Changing the values in the voxl-camera-server.conf file results in a corrupted stream

tom

@AidanGallagher If you run voxl-camera-server -l you should be able to see a list of supported resolutions

AidanGallagher

@tom I have done that are there are three potential resolutions.
However, only the 1936x1080 produces a valid stream.

When I capture a raw bayer at 4056x3040 then the majority of the image is black. It is as if the 1080p image is put into a 4k container.

Any ideas what I could be doing wrong?

Alex Kushleyev

Hi @AidanGallagher , can you please clarify what resolution and fps you are trying to achieve using the "fpv" version of the IMX412 drivers? Please note this version of drivers is still "experimental" but I can help you get the results that you need.

When using MISP (Modalai ISP) pipeline, the 4056 resolution currently does not work (long story). But we have a workaround for using 4040x3040, which is just 16 pixel crop from the full frame. If that would work for you, you should pick up the latest drivers for MISP here :

https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250311/imx412_fps_eis_20250311_drivers.zip

You will simply need to replace the sensormodule file for the slot number that you have for your imx412 (in /usr/lib/camera/). and might as well update the .so that's in the zip to overwrite the existing one in the same location)

The following RAW resolutions are supported by these drivers:

   - 4056x3040 @ 30                     16.5ms readout time
   - 4040x3040 @ 30, **60**             16.5ms readout time
   - 3840x2160 @ 30, 60, 80             11.8ms readout time
   - 1996x1520 @ 30, 60, 120            5.5ms readout time
   - 1936x1080 @ 30, 60, 90, 120, 240   4.0ms readout time
   - 1996x480  @ 30, 480                1.8ms readout time
   - 1996x240  @ 30, 800                0.9 - 1.0 ms readout time

*** Please note 4040x3040 60FPS is not stable, please use slightly lower FPS like 55 or even 57-58 should be OK if needed.

1996x1520 is the 2x2 binned resolution of full frame with a small horizontal crop to make it compatible with MISP.

The readout times are noted as they are important if you are concerned with overall latency.

These drivers are configured to use IMX412 at its maximum throughput of 2.1Gbps per MIPI lane (4 lanes), which ensures the lowest latency and maximum possible FPS.

When you use this driver, please set the following value in /vendor/etc/camera/camxoverridesettings.txt - this is a workaround for camera pipeline not showing all possible RAW resolutions / fps.

maxRAWSizes=20

Also, we recently (I believe SDK 1.4.1) disabled debug prints from camera pipeline to logcat, so if you wanted to confirm that correct resolution is selected (see below), enable the following (in the same file), if the param is set to 0 (need to set to 1):

systemLogEnable=1

After confirming the resolution, you can set the param back to 0 to reduce cpu usage.

You should not use 4056x4030, but instead use 4040x3040. if you use 1936x1080, then you probably want to specify misp_width as 1920, not 1936. The reason for 1936 is the same as for 4056 -- MISP does not work with 1920, but we just crop 1936 down to 1920. There are no performance issues associated with this.

Please note the 480 and 800fps modes are experimental, may cause camera server to crash, depending on what you are doing (especially 800fps). also if you try to stream these high fps to voxl-portal, you will have really large frame drop, but you will see some frames. (maybe up to 120fps, since frames are small). For high fps modes, set cpu to perf using voxl-set-cpu-mode perf.

Finally, for best performance you should use the latest camera server from our nightly builds, if you do not, please specify which version you are using and I can check if there are any relevant issues related to what you are trying to do.

In camera server config, you should specify one of the above resolutions as your preview_width and preview_height and misp_width and misp_height can actually be pretty much any resolution you want. Based on the misp resolution that you request, the source image will be down-scaled and cropped to match your dimensions and maintain the proper aspect ratio (no stretching, etc).

Finally, if you are experimenting with different modes and FPS, and the image you get looks completely wrong (zigzags, etc), you should verify that the camera pipeline indeed selects correct resolution by running the following before you start camera server:

this will show something like this:

logcat | grep -i selected
03-19 20:26:09.694  1894  1894 I CHIUSECASE: [CONFIG ] chxsensorselectmode.cpp:635 FindBestSensorMode() Selected Usecase: 7, SelectedMode W=4040, H=3040, FPS:30, NumBatchedFrames:0, modeIndex:0
03-19 20:26:09.694  1894  1894 I CHIUSECASE: [CONFIG ] chxpipeline.cpp:371 CreateDescriptor() Pipeline[Raw] Pipeline pointer 0x55ae2d6620 Selected sensor Mode W=4040, H=3040
03-19 20:26:09.694  1894  1894 I CamX    : [CONFIG][CORE   ] camxsession.cpp:5907 SetRealtimePipeline() Session 0x55ae411ea0 Pipeline[Raw] Selected Sensor Mode W=4040  H=3040

if the selected resolution is not what you reqested as preview (this may happen if you select 1996x240 resolution but choose FPS equal or below 480, the pipeline will choose 1996x480 resolution (because it is closer in FPS...), which may cause the board to crash (because buffers are allocated for 1996x240 resolution). So if you use use high FPS modes, either use the exact FPS that i listed or use FPS above the the higher resolution modes. I hope that makes sense.. This is an issue we are going to work out soon. If you want to avoid crashing the board (which is temporary, just power cycle to recover). you should start the camera server but do not have any clients for the hires camera stream, which would not actually start the camera streaming, but you will see the resolution selected and can stop the camera server before it would cause a crash.

In the near future we will update these experimental binaries and make them the default, as they are backwards compatible with existing drivers that are shipped with the SDK.

Alex

AidanGallagher

@Alex-Kushleyev thanks a million for this information (and the beta drivers) Alex it is very helpful.

I am using voxl-camera-server version 2.1.1 (voxl-suite 1.4.1)
From your information I will try 3840x2160 and 4040x3040 at 60fps to see what the performance is like.

Aidan

Alex Kushleyev

@AidanGallagher , you are welcome.

Just one note, if you are going to try 4040x3040 maximum fps, please choose something slightly under 60, as 60 is sometimes causing issues due to very little time between end of frame and start of frame, but even 55 is going to be stable. When I use this resolution, i test at 50 or 55FPS. I will look into 60FPS at full resolution a bit later, but may need to cut down on the vertical size a bit to give some time between EOF and SOF.

Alex

AidanGallagher

@Alex-Kushleyev understood,

Thanks again I will give this a try - fingers crossed