EIS merge

Alex Kushleyev

I am working on merging the current EIS features to our main branch, this should be done sometime next week.

Meanwhile, it would help if you briefly describe your use case for EIS, so I can make sure it is supported in the release. Are you using IMX412 camera?

The Rolling Shutter correction won't yet be supported, but I wanted to release what we have so far, which works pretty under low vibration conditions.

If you don't have the build environment set up for building voxl-camera-server, i can provide the deb package for you to test what is on EIS branch right now.

Alex

h3robotics

@Alex-Kushleyev thank you so much. I can wait till the changes are merged and available to update directly on the voxl2.

Yes, I am using the IMX412 camera.

I can start off with whatever you have got for now and hope for a good video feed on my drone without any gimbal. As new changes come along, I will update.

I have never setup a build environment for voxl repos. I am used to building PX4 etc. with some of my custom changes. I would love to learn how to setup the build environment for the voxl repos. So if you can point me in the right direction, that would be great! I see in the docs, under "voxl SDK > custom voxl applications", there are some instructions.

Thank you!

h3robotics

Hi @Alex-Kushleyev ,

Any update on the EIS feature merge?

Thanks.

Alex Kushleyev

Hi @h3robotics, Thanks for your patience.

Actually as of right now, the EIS branch has been updated to have all the latest features from dev.

One big improvement from prior EIS version is that now for each camera we support up to 4 MISP outputs of different resolution, fps, encoder params and each stream can have EIS on or off.

Please hang tight, i just need to tie up some loose ends and add documentation. I will post an update most likely tomorrow. I will also share a voxl-camera-server.conf with the usage examples.

Meanwhile please make sure you can build the eis branch from here, https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis , using voxl-cross:4.0 docker image, which is required.

Alex

Alex Kushleyev

Hi All,

Sorry for the delay again. We were chasing down a vibration issue which was apparently causing significant rolling shutter artifacts on the front camera in my tests. Apparently some of the lenses for IMX412 cameras have loose internal components, causing the vibration which is impossible to correct. On my test platform (Starling 2 Max), I had this issue and no frame modifications improved anything. The last thing to try was to switch out the lens and the issue was completely gone.

We will investigate the reason for the loose lenses and the solution. In my case, shaking the lens would result in audible rattle inside the lens. If anyone has "shaky" lens, we will send a replacement.

I will post some sample videos of what a bad lens might look like.

The good news is that there is actually no major vibration issue on the Starling 2 Max and I can move forward with the EIS sw release.. stay tuned for another day or so!

Alex

JP Drone

Hi @Alex-Kushleyev
Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
Would it be possible to include a replacement lens with my current order?
Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

Alex Kushleyev

@JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

Here is an example of EIS running with bad and good lenses (just a short test):

bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

Alex

Alex Kushleyev

The EIS branch is ready for initial tested. Please see the doc file:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

This release also supports eis on 3 additional MISP channels (can be different resolutions).

We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

Please try it out, i would be excited to get initial feedback.

Alex

JP Drone

Hi @Alex-Kushleyev
Happy to test it.
My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

Thanks a lot for all the support!

Alex Kushleyev

@JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/voxl2-eis.md?ref_type=heads#extrinsic-calibration

Instead of right side up

"RPY_parent_to_child":    [0, 90, 90]

your extrinsic file should be

"RPY_parent_to_child":    [0, 90, -90]

In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

{
    "parent": "imu_apps",
    "child":  "<camera_name>",
    "T_child_wrt_parent": [0.0, 0.0, 0.0],
    "RPY_parent_to_child":    [0, 90, -90]
}

(I am assuming VOXL2 is mounted in normal orientation, right side up)

Please try.

Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

Alex

Alex Kushleyev

Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/