EIS merge

david.moro

We have been following the development on https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis?ref_type=heads. @Alex-Kushleyev , do you have an ETA of when this will be released as part of an SDK update? Thanks!

Alex Kushleyev

Hi @david-moro ,

We have been working on a number of EIS-related features and I am happy to report that I can share instructions for testing it very soon. I am going to push for releasing some beta docs for testing sometime this week. It may be some time before this feature will be officially released in an SDK, but I will merge everything from voxl-camera-server dev branch into the eis branch.

So far, the following features have been implemented:

tested on IMX412 and IMX214 cameras (IMX412 is strongly recommended due to wider FOV and better lens and imager quality, much better low light performance)
Input resolution up to full frame 4xxx by 3xxx
output resolution up to 4K (stabilized)
digital zoom (stabilized) + voxl-portal integration
stabilized roi selection + voxl-portal click and drag
support landscape or portrait camera orientation
multi camera support (simultaneous EIS running on multiple cameras)
multiple stream per camera (stabilized, unstabilized, different encoding bitrates)

No rolling shutter correction (yet)

There is going to be a lot of things to document, so I can start with basic use cases so you could test them out. The only thing from hardware you will need is a VOXL2 or VOXL2 mini and IMX412 camera (M0107 or M0161). Onboard VOXL2 IMU is used.

Thanks for your patience

Alex

david.moro

@Alex-Kushleyev Fantastic! Looking forward to it.

Alex Kushleyev

@david-moro , sorry for the delay. I just posted an initial draft for setting up EIS on VOXL2:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

I will be making some demo videos in the next few days.

Please go through the instructions and make sure you can get camera server and voxl-portal to run (you may need to restart the voxl-portal service after updating the package). By default, you should see a merged view in voxl-portal which consists of stabilized (left) and full frame view (with ROI drawn) on the right. Using the latest voxl-portal, you can interact with the ROI on both sides of the image (see the readme).

Once you set it up and play with zoom, you should update the default zoom to be higher than 1.0 so that you have some stabilization margin (in voxl camera server config file).

I just wanted to get this out, and i'll make a few demo videos very soon, but for now hopefully the instructions work and i will let you explore the features a little bit

You will need to build the voxl-camera-server and voxl-portal packages from the appropriate branches. Hopefully you can do that by yourself. If not, i can post the test debs.

You dont need to do any intrinsic or extrinsic calibration for the initial test (see readme for details)

Please let me know if you run into any issues.

I am going to add a lot more details shortly.

Alex

david.moro

@Alex-Kushleyev Thank you very much! I think we can take it from here

David

Alex Kushleyev

Hi @david-moro ,

I made several videos that show the current functionality of the ROI and EIS features.

ROI features: https://www.youtube.com/watch?v=FXv4855WjNc
EIS: https://www.youtube.com/watch?v=fi2BO_U5f-c
EIS outdoor test : https://www.youtube.com/watch?v=oHDsiTD8Ky4

Please note that the last video was recorded with different EIS settings than currently are used as defaults. However, the same processing pipeline was used.

We will support up to 4K60 stabilized output and concurrent EIS on multiple cameras (already tested). There will be a lot more work done in the coming weeks to improve the video quality when testing on actual flying vehicles.

Alex