EIS merge

JP Drone

Hi @Alex-Kushleyev
Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
Would it be possible to include a replacement lens with my current order?
Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

Alex Kushleyev

@JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

Here is an example of EIS running with bad and good lenses (just a short test):

bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

Alex

Alex Kushleyev

The EIS branch is ready for initial tested. Please see the doc file:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

This release also supports eis on 3 additional MISP channels (can be different resolutions).

We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

Please try it out, i would be excited to get initial feedback.

Alex

JP Drone

Hi @Alex-Kushleyev
Happy to test it.
My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

Thanks a lot for all the support!

Alex Kushleyev

@JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/voxl2-eis.md?ref_type=heads#extrinsic-calibration

Instead of right side up

"RPY_parent_to_child":    [0, 90, 90]

your extrinsic file should be

"RPY_parent_to_child":    [0, 90, -90]

In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

{
    "parent": "imu_apps",
    "child":  "<camera_name>",
    "T_child_wrt_parent": [0.0, 0.0, 0.0],
    "RPY_parent_to_child":    [0, 90, -90]
}

(I am assuming VOXL2 is mounted in normal orientation, right side up)

Please try.

Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

Alex

Alex Kushleyev

Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/