EIS merge

JP Drone

Hi @Alex-Kushleyev
Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
Would it be possible to include a replacement lens with my current order?
Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

Alex Kushleyev

@JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

Here is an example of EIS running with bad and good lenses (just a short test):

bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

Alex

Alex Kushleyev

The EIS branch is ready for initial tested. Please see the doc file:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

This release also supports eis on 3 additional MISP channels (can be different resolutions).

We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

Please try it out, i would be excited to get initial feedback.

Alex

JP Drone

Hi @Alex-Kushleyev
Happy to test it.
My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

Thanks a lot for all the support!

Alex Kushleyev

@JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/voxl2-eis.md?ref_type=heads#extrinsic-calibration

Instead of right side up

"RPY_parent_to_child":    [0, 90, 90]

your extrinsic file should be

"RPY_parent_to_child":    [0, 90, -90]

In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

{
    "parent": "imu_apps",
    "child":  "<camera_name>",
    "T_child_wrt_parent": [0.0, 0.0, 0.0],
    "RPY_parent_to_child":    [0, 90, -90]
}

(I am assuming VOXL2 is mounted in normal orientation, right side up)

Please try.

Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

Alex

Alex Kushleyev

Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/

SKA

Hi @Alex-Kushleyev! Where can I find the IMX214 drivers for EIS? Thanks!

Alex Kushleyev

Hello @SKA,

I just posted the IMX214 sensormodules here : https://storage.googleapis.com/modalai_public/temp/imx214_test_bins/imx214_eis_drivers_20250815.zip

These support the new resolutions 4196x3120 and 2096x1560 needed for MISP / EIS.

IMX214 (M0025-2)

4196x3120 - calibration file resolution 4196x3120 or 2098x1560
2096x1560 - calibration file resolution 2096x1560 or 2098x1560

(calibration file is optional, EIS will run without one)

I also just updated the voxl camera server (dev branch) to enable MISP support for these resolutions and also a new feature to allow specifying the bayer pattern (in case you use the flipped sensormodule).

Here is an example snipped of voxl-camera-server.conf. The parameter bayer_type is optional and hidden by default. you can use rggb or bggr for IMX214. You can change this parameter if you see that your colors are flipped (red and blue).

                        "type": "imx214",
                        "name": "hires0",
                        "bayer_type": "rggb"

The performance of EIS with IMX214 has not been tuned - specifically the behavior of the ROI controller. We can come back to that if there is interest.. Mostly focusing on IMX412 with the fisheye lens for now.

Please try and make sure you can at least make it work and we can discuss performance tuning.

Alex

SKA

@Alex-Kushleyev The drivers worked, and the EIS as expected isn't tuned well. The MISP brightness looks alright outdoors, but indoors its very low. Increasing exposure_max_us helps but adds blur. Not sure if that's to be expected with this sensor, since it isn't as good in low-light conditions as the IMX412. Let me know if there's anything I can tune in the conf file to help with the brightness, and EIS output. Thanks!

Alex Kushleyev

Hi @SKA,

The down side of using IMX214 vs IMX412 is that IMX214 is a much older sensor and it has several attributes that make it less desirable for using with EIS:

lower pixel sensitivity and lower maximum gain range, requiring increasing exposure to achieve desired brightness, but this creates motion blur. The image brightness can be increased via digital gain, but will add some pixel noise
smaller package / lens -- smaller lens does not allow as much light to enter the sensor
smaller FOV (larger FOV with fisheye lens allows for more compensation margin)
lower readout speed (about 2x slower). This increases the readout time, which is related to rolling shutter effect.

If the image brightness is the most concern to you right now, I can add a digital gain feature to boost the image brightness, but we are going to see some more pixel noise. I wanted to do that anyway, so may be a good chance to try it. After boosting the digital gain, ideally you would want to run some de-noising algorithm, but we don't have that yet in the MISP pipeline.

Are you working with an existing system that has IMX214 already integrated and want to see if you can enable EIS on that set up?

Alex

Federico Wyrwal

I have the starling 2 drone with voxl2 and sdk 1.5. I have the imx412 sensor mounted. The EIS requires a config file that unfortunately does not match the available resolutions for my hires sensor: 4056x3040. Is there a way around this?
Another valid configuration exists for 1024x768

Alex Kushleyev

@Federico-Wyrwal , please check the instructions for setting up EIS, specifically

https://docs.modalai.com/camera-video/electronic-image-stabilization/#minimal-software-setup

The latest drivers can be downloaded from here

Please let me know if you have any other questions.

Alex