EIS merge

[h3robotics]

@Alex-Kushleyev said in EIS merge:

You will need to build the voxl-camera-server and voxl-portal packages from the appropriate branches. Hopefully you can do that by yourself. If not, i can post the test debs.

Hi @Alex-Kushleyev I want to use the EIS features. Would you be able to help me guide in the right direction of how to setup a build environment and build from the voxl-camera-server eis branch? Thank you.

[Alex Kushleyev]

@h3robotics ,

I am working on merging the current EIS features to our main branch, this should be done sometime next week.

Meanwhile, it would help if you briefly describe your use case for EIS, so I can make sure it is supported in the release. Are you using IMX412 camera?

The Rolling Shutter correction won't yet be supported, but I wanted to release what we have so far, which works pretty under low vibration conditions.

If you don't have the build environment set up for building voxl-camera-server, i can provide the deb package for you to test what is on EIS branch right now.

Alex

[h3robotics]

@Alex-Kushleyev thank you so much. I can wait till the changes are merged and available to update directly on the voxl2.

Yes, I am using the IMX412 camera.

I can start off with whatever you have got for now and hope for a good video feed on my drone without any gimbal. As new changes come along, I will update.

I have never setup a build environment for voxl repos. I am used to building PX4 etc. with some of my custom changes. I would love to learn how to setup the build environment for the voxl repos. So if you can point me in the right direction, that would be great! I see in the docs, under "voxl SDK > custom voxl applications", there are some instructions.

Thank you!

[h3robotics]

Hi @Alex-Kushleyev ,

Any update on the EIS feature merge?

Thanks.

[Alex Kushleyev]

Hi @h3robotics, Thanks for your patience.

Actually as of right now, the EIS branch has been updated to have all the latest features from dev.

One big improvement from prior EIS version is that now for each camera we support up to 4 MISP outputs of different resolution, fps, encoder params and each stream can have EIS on or off.

Please hang tight, i just need to tie up some loose ends and add documentation. I will post an update most likely tomorrow. I will also share a voxl-camera-server.conf with the usage examples.

Meanwhile please make sure you can build the eis branch from here, https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis , using voxl-cross:4.0 docker image, which is required.

Alex

[Alex Kushleyev]

Hi All,

Sorry for the delay again. We were chasing down a vibration issue which was apparently causing significant rolling shutter artifacts on the front camera in my tests. Apparently some of the lenses for IMX412 cameras have loose internal components, causing the vibration which is impossible to correct. On my test platform (Starling 2 Max), I had this issue and no frame modifications improved anything. The last thing to try was to switch out the lens and the issue was completely gone.

We will investigate the reason for the loose lenses and the solution. In my case, shaking the lens would result in audible rattle inside the lens. If anyone has "shaky" lens, we will send a replacement.

I will post some sample videos of what a bad lens might look like.

The good news is that there is actually no major vibration issue on the Starling 2 Max and I can move forward with the EIS sw release.. stay tuned for another day or so!

Alex

[JP Drone]

Hi @Alex-Kushleyev
Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
Would it be possible to include a replacement lens with my current order?
Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

[Alex Kushleyev]

@JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

Here is an example of EIS running with bad and good lenses (just a short test):

bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

Alex

[Alex Kushleyev]

The EIS branch is ready for initial tested. Please see the doc file:

voxl2-eis.md · eis · voxl / VOXL SDK / Services / voxl-camera-server · GitLab

GitLab.com

GitLab (gitlab.com)

You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

This release also supports eis on 3 additional MISP channels (can be different resolutions).

We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

Please try it out, i would be excited to get initial feedback.

Alex

[JP Drone]

Hi @Alex-Kushleyev
Happy to test it.
My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

Thanks a lot for all the support!

[Alex Kushleyev]

@JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

voxl2-eis.md · dev · voxl / VOXL SDK / Services / voxl-camera-server · GitLab

GitLab.com

GitLab (gitlab.com)

Instead of right side up

"RPY_parent_to_child":    [0, 90, 90]

your extrinsic file should be

"RPY_parent_to_child":    [0, 90, -90]

In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

{
    "parent": "imu_apps",
    "child":  "<camera_name>",
    "T_child_wrt_parent": [0.0, 0.0, 0.0],
    "RPY_parent_to_child":    [0, 90, -90]
}

(I am assuming VOXL2 is mounted in normal orientation, right side up)

Please try.

Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

Alex

[Alex Kushleyev]

Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/

[SKA]

Hi @Alex-Kushleyev! Where can I find the IMX214 drivers for EIS? Thanks!

[Alex Kushleyev]

Hello @SKA,

I just posted the IMX214 sensormodules here : https://storage.googleapis.com/modalai_public/temp/imx214_test_bins/imx214_eis_drivers_20250815.zip

These support the new resolutions 4196x3120 and 2096x1560 needed for MISP / EIS.

IMX214 (M0025-2)

4196x3120 - calibration file resolution 4196x3120 or 2098x1560
2096x1560 - calibration file resolution 2096x1560 or 2098x1560

(calibration file is optional, EIS will run without one)

I also just updated the voxl camera server (dev branch) to enable MISP support for these resolutions and also a new feature to allow specifying the bayer pattern (in case you use the flipped sensormodule).

Here is an example snipped of voxl-camera-server.conf. The parameter bayer_type is optional and hidden by default. you can use rggb or bggr for IMX214. You can change this parameter if you see that your colors are flipped (red and blue).

                        "type": "imx214",
                        "name": "hires0",
                        "bayer_type": "rggb"

The performance of EIS with IMX214 has not been tuned - specifically the behavior of the ROI controller. We can come back to that if there is interest.. Mostly focusing on IMX412 with the fisheye lens for now.

Please try and make sure you can at least make it work and we can discuss performance tuning.

Alex

[SKA]

@Alex-Kushleyev The drivers worked, and the EIS as expected isn't tuned well. The MISP brightness looks alright outdoors, but indoors its very low. Increasing exposure_max_us helps but adds blur. Not sure if that's to be expected with this sensor, since it isn't as good in low-light conditions as the IMX412. Let me know if there's anything I can tune in the conf file to help with the brightness, and EIS output. Thanks!

[Alex Kushleyev]

Hi @SKA,

The down side of using IMX214 vs IMX412 is that IMX214 is a much older sensor and it has several attributes that make it less desirable for using with EIS:

• lower pixel sensitivity and lower maximum gain range, requiring increasing exposure to achieve desired brightness, but this creates motion blur. The image brightness can be increased via digital gain, but will add some pixel noise
• smaller package / lens -- smaller lens does not allow as much light to enter the sensor
• smaller FOV (larger FOV with fisheye lens allows for more compensation margin)
• lower readout speed (about 2x slower). This increases the readout time, which is related to rolling shutter effect.

If the image brightness is the most concern to you right now, I can add a digital gain feature to boost the image brightness, but we are going to see some more pixel noise. I wanted to do that anyway, so may be a good chance to try it. After boosting the digital gain, ideally you would want to run some de-noising algorithm, but we don't have that yet in the MISP pipeline.

Are you working with an existing system that has IMX214 already integrated and want to see if you can enable EIS on that set up?

Alex

[Federico Wyrwal]

I have the starling 2 drone with voxl2 and sdk 1.5. I have the imx412 sensor mounted. The EIS requires a config file that unfortunately does not match the available resolutions for my hires sensor: 4056x3040. Is there a way around this?
Another valid configuration exists for 1024x768

[Alex Kushleyev]

@Federico-Wyrwal , please check the instructions for setting up EIS, specifically

https://docs.modalai.com/camera-video/electronic-image-stabilization/#minimal-software-setup

The latest drivers can be downloaded from here

Please let me know if you have any other questions.

Alex