IMX412-Flip

[robertociuch]

Hi everyone,

I'm working with a VOXL2 setup using the microcoax adapter (MDK-M0173-1-00), and I have three AR0144 and IMX412 cameras connected. (same connection used on starling2, without ToF sensor - image attached)

I need to flip the IMX412 camera image upside down, but I haven't found a way to do this via the voxl-camera-server.conf file. In the past, I've managed this using the "imx412-flip" drivers found under /usr/share/modalai/chi-cdk, but the problem is that my IMX412 camera is assigned to ID 1, and there is no driver for that ID

Is there a way to solve this problem?

Thanks in advance for any help

[Alex Kushleyev]

@robertociuch ,

I just built the sensormodules for ID 1 and 5. Please test it out. A while ago we did not have hires camera support in these camera slots, so the sensormodules were not built. Please let me know if that works for you.

Alex

https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250430/com.qti.sensormodule.imx412_flip_1.bin

https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250430/com.qti.sensormodule.imx412_flip_5.bin

[robertociuch]

@Alex-Kushleyev

Hi Alex, it works fine

thank you very much!

[Alex Kushleyev]

@robertociuch , thanks for letting me know! I will make sure this update makes it into a future SDK release.

Alex

[robertociuch]

@Alex-Kushleyev , one more question;

I'm using two AR0144 cameras as a stereo setup. Should I use the "combo driver" to synchronize them?
I remember that about a year ago I had an issue with a different setup where the cameras weren't perfectly synchronized, and the images were captured with a slight time offset.
In this case, how should I set the IDs in the camera server, and what connections should I use to ensure they work correctly?

[Alex Kushleyev]

Hi @robertociuch ,

There are no sync issues with AR0144 that we are aware of right now. If you want the cameras to by sync'ed, you have to use the "combo" or "fsin" drivers for the AR0144 camera (see below re fsin). The non-combo / non-fsin drivers do not use the sync line as the sync input and will be free-running.

The only additional limitation for the sync'ed AR0144 vs non-synced, the maximum exposure is limited to 12ms (compared to 33ms in non-synced use case). The actual limitation is slightly higher than 12ms (maybe 14-15) and it is coming from the way the camera operates in the sync'ed mode (this is specific to AR0144 only). This 12ms limit is set in voxl-camera-server (per each camera) "exposure_max_us": 12000, so you want to double check that. If you don't limit the exposure, then if the exposure setting extceeds this limit (well, exceeds 15ms or so), the frame rate will drop in half from 30 to 15. When you run voxl-configure-cameras 26, the correct sensormodules will be coped to /usr/lib/camera/ and correct config with appropriate exposure limits applied to /etc/modalai/voxl-camera-server.conf.

In terms of sensormodule IDs for AR0144, you need to look where they are connected.

If we look at C27 use case, from the following link:
https://docs.modalai.com/voxl2-coax-camera-bundles/

there are 3 AR0144 cameras connected and the following sensormodules are present:

com.qti.sensormodule.ar0144_combo_0.bin
com.qti.sensormodule.ar0144_combo_6.bin
com.qti.sensormodule.ar0144_fsin_2.bin

Actually the combo sensormodules are not only sync'ed but they also use a single MIPI interface to connect two cameras (nice trick). That is really whey they are called combo (the streams are combined into a single mipi interface). The third camera takes up the whole 4-lane mipi camera port and that is why it is not a combo mode driver. fsin means it is using frame sync input.

So all 3 of those camera drivers will use the sync input, generated by VOXL 2 for sync signal. BTW, the voxl-camera-server.conf contains information about the frame sync pin and whether it is enabled or not. You have to make sure that you enable it and use correct pin when using M0173:

	"fsync_en":	true,
	"fsync_gpio":	109,

This is also mentioned here : https://docs.modalai.com/M0173/

If you are only connecting two AR0144 cameras, you should connect them as shown in C26 config and you would want to have the following sensormodules:

com.qti.sensormodule.ar0144_combo_0.bin
com.qti.sensormodule.ar0144_combo_6.bin

Finally, in order to confirm time sync. You can do it indirectly by looking at the timestamp of the frames coming from two cameras. I actually wrote a tool which subscribes to two camera streams and will compare the incoming timestamps to tell you how much out of sync they are. I will find the tool and share it.

There is also a direct way of checking sync of looking at the camera images. We have a special LED array where we can light up each LED for a fixed number of microseconds. Each LED is lit up for that short amount of time and the LEDs light up in a scanning pattern across a 2D array. And then we set the camera exposure to a small value (1ms lets say) for both cameras. In the image below, each LED lights up for 50us and the camera exposure was set to 1ms, so we see 20 LEDs lit up at the same time (50*20 = 1000us) and they are also aligned in time to within 50us). This specific capture was from an older ov7251 test, but we have also done ar0144 sync test.

I just did a quick test using two sync'ed AR144, showing 0 delay in timestamps:

voxl2:~$ ./voxl-cam-sync-check -a tracking_front -b tracking_rear
camera a: /run/mpa/tracking_front/
camera b: /run/mpa/tracking_rear/
ab delta: 0.000000

Of course the sync cannot be EXACTLY zero, but it is as close to zero as the system can measure (way below 100us), just based on the timestamp test.

If I do a comparison with a non-sync'ed hires cameras:

voxl2:~$ ./voxl-cam-sync-check -a tracking_front -b hires_bayer      
camera a: /run/mpa/tracking_front/
camera b: /run/mpa/hires_bayer/
ab delta: 0.053434a ahead, trying to re-sync
ab delta: -0.013322  dt too large!
ab delta: -0.013265  dt too large!
ab delta: -0.013148  dt too large!
ab delta: -0.013226  dt too large!

Please let me know if you have any more questions.

Alex

[John.Almasi]

Hi @Alex-Kushleyev ,

Following up on this thread ... I have nearly the same setup as @robertociuch and need the same flip, but with one twist.

Our setup:

VOXL 2 (M0054), SDK 1.5.1, system-image 1.8.04, with the M0173 microcoax front-end; three AR0144 tracking cameras + an IMX412 hires, same layout as Starling 2 without the ToF.

The IMX412 is on camera ID 1 and is physically mounted inverted (only way we were able to), so the image comes through upside down. As you noted here, en_rotate has no effect on the IMX412.

The difference from the earlier case: we're running the FPV low-latency IMX412 driver on that slot, not the standard one:

com.qti.sensor.imx412_fpv.so
com.qti.sensormodule.imx412_fpv_1.bin
com.qti.tuned.imx412_fpv.bin

The IMX412 is the pilot's live video feed on a commercial cleaning drone, so we need to keep the low-latency FPV behavior ... the imx412_flip_1.bin you built earlier would flip the image but drop us back to the standard (non-FPV) driver.

Would you be able to build a combined FPV + 180° flip sensormodule for camera ID 1 ( i.e. com.qti.sensormodule.imx412_fpv_flip_1.bin ) so we get the flip while keeping the FPV variant?

Any install guidance would also be appreciated: where to place it (/usr/lib/camera/?) and whether the imx412_fpv.so / tuned files need a matching update, or if swapping the sensormodule .bin alone is enough.

Thanks!
John