EIS merge

david.moro

We have been following the development on https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis?ref_type=heads. @Alex-Kushleyev , do you have an ETA of when this will be released as part of an SDK update? Thanks!

Alex Kushleyev

Hi @david-moro ,

We have been working on a number of EIS-related features and I am happy to report that I can share instructions for testing it very soon. I am going to push for releasing some beta docs for testing sometime this week. It may be some time before this feature will be officially released in an SDK, but I will merge everything from voxl-camera-server dev branch into the eis branch.

So far, the following features have been implemented:

tested on IMX412 and IMX214 cameras (IMX412 is strongly recommended due to wider FOV and better lens and imager quality, much better low light performance)
Input resolution up to full frame 4xxx by 3xxx
output resolution up to 4K (stabilized)
digital zoom (stabilized) + voxl-portal integration
stabilized roi selection + voxl-portal click and drag
support landscape or portrait camera orientation
multi camera support (simultaneous EIS running on multiple cameras)
multiple stream per camera (stabilized, unstabilized, different encoding bitrates)

No rolling shutter correction (yet)

There is going to be a lot of things to document, so I can start with basic use cases so you could test them out. The only thing from hardware you will need is a VOXL2 or VOXL2 mini and IMX412 camera (M0107 or M0161). Onboard VOXL2 IMU is used.

Thanks for your patience

Alex

david.moro

@Alex-Kushleyev Fantastic! Looking forward to it.

Alex Kushleyev

@david-moro , sorry for the delay. I just posted an initial draft for setting up EIS on VOXL2:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

I will be making some demo videos in the next few days.

Please go through the instructions and make sure you can get camera server and voxl-portal to run (you may need to restart the voxl-portal service after updating the package). By default, you should see a merged view in voxl-portal which consists of stabilized (left) and full frame view (with ROI drawn) on the right. Using the latest voxl-portal, you can interact with the ROI on both sides of the image (see the readme).

Once you set it up and play with zoom, you should update the default zoom to be higher than 1.0 so that you have some stabilization margin (in voxl camera server config file).

I just wanted to get this out, and i'll make a few demo videos very soon, but for now hopefully the instructions work and i will let you explore the features a little bit

You will need to build the voxl-camera-server and voxl-portal packages from the appropriate branches. Hopefully you can do that by yourself. If not, i can post the test debs.

You dont need to do any intrinsic or extrinsic calibration for the initial test (see readme for details)

Please let me know if you run into any issues.

I am going to add a lot more details shortly.

Alex

david.moro

@Alex-Kushleyev Thank you very much! I think we can take it from here

David

Alex Kushleyev

Hi @david-moro ,

I made several videos that show the current functionality of the ROI and EIS features.

ROI features: https://www.youtube.com/watch?v=FXv4855WjNc
EIS: https://www.youtube.com/watch?v=fi2BO_U5f-c
EIS outdoor test : https://www.youtube.com/watch?v=oHDsiTD8Ky4

Please note that the last video was recorded with different EIS settings than currently are used as defaults. However, the same processing pipeline was used.

We will support up to 4K60 stabilized output and concurrent EIS on multiple cameras (already tested). There will be a lot more work done in the coming weeks to improve the video quality when testing on actual flying vehicles.

Alex

m.costa

@Alex-Kushleyev Looking forward to trying this out. A couple questions.

Which version of Qualcomm's EIS is this using? 1.0, 2.6, or 3.0?
What's the ETA for getting RSC implemented?

Alex Kushleyev

@m-costa , this is ModalAI EIS 1.0 . The computations are done within voxl-camera-server by processing the raw bayer data directly (cpu and gpu).

I actually have some updates and new videos will share shortly.

Also started testing on a flying platform (Starling 2 Max), will share initial results dual cam EIS (front facing and down facing).

I am curious what platforms folks would be using for testing EIS? One of ModalAI drones or something custom? If using a standard drone, i can document how to set up EIS for that type of drone, since there are a few things that need to be set up / verified in order for EIS to work properly.

Regarding Rolling Shutter Corrections, we have started implementing, will probably have some initial results in 1-2 weeks.

Alex

m.costa

@Alex-Kushleyev So this is a custom implementation, not using any of Qualcomm's built in tools? If so how are you planning on implementing the RSC? Reading through some of the QC documentation, it seems like the supported QC EIS versions (1.0, 2.6) for the RB5 only correct rolling shutter for linear motion on a per frame basis. That doesn't help much on a high vibration platform where the intra-frame motion is non-linear.

Are you planning on implementing X and Y rolling shutter corrections on a per line basis? That would be sweet.

I'm looking to use this on a custom quadcopter that has pretty poor vibration isolation. Eager to see your results so far with the EIS.

Alex Kushleyev

@m-costa , yes it is a completely custom implementation.

For rolling shutter compensation, we are working on per-line compensation (not just X, Y, but full 3D rotation because the vibration can occur along all three axes x, y, and z). However in order for this approach to work well in environment with lots of high frequency vibrations, an IMU would need to be mounted directly on the camera (very rigid mechanical coupling, so that IMU picks up the exact vibrations that the camera is experiencing).

Unfortunately we do not currently offer any cameras with a built-in IMU. However, VOXL2 / VOXL2 mini both have an additional SPI port which can be used for an IMU. The current approach is to use an external IMU in addition to the IMU on VOXL2 and compare the EIS results. Voxl-imu-server already has support for an external IMU over SPI.

Also i should note that it is unlikely that we will be testing on RB5 (because it is EOL) - I am not sure whether you were implying that you are using an RB5.

Finally, you should also investigate your camera mount. Vibration reduction is a whole separate subject, but you should make sure that the camera mount is very rigid and it is rigidly attached to the frame. Sometimes a little flex in the mount can amplify the existing vibrations, so in this case the solution would be to actually make the mount more rigid instead of isolating it with a soft material. Actually using dual IMU approach can reveal if the camera is vibrating more than the main board (voxl2) by comparing the IMU data.

Speaking of the camera - what camera are you using on your quadcopter?

Finally, some new videos. No RSC yet.

dual camera EIS on Starling 2 Max (hand carried) : https://www.youtube.com/watch?v=-BA_nU4kjQs
down-facing camera EIS on Starling 2 Max (flight) : https://www.youtube.com/watch?v=-B5xKmBBCAc

Notes about videos

the ROI tracking still needs some work (it has not been tuned for the down-facing use case)
down-facing camera had almost no vibration picked up by the camera. Some rolling shutter artifacts can be observed in a few cases.
the front camera was also tested, but due to vibration + rolling shutter, the video did not turn out great, so I did not post it. Camera mount needs a little bracing to remove flex and will be tested again. Once I fix the camera mount bracing issue, i can post the before and after videos.
in the dual cam video, you can see some rolling shutter artifacts which appear as slanted vertical features and a little motion due to shaking. This type of distortion is the "easy" one to fix using RSC.

Alex