EIS merge

[david.moro]

We have been following the development on https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis?ref_type=heads. @Alex-Kushleyev , do you have an ETA of when this will be released as part of an SDK update? Thanks!

[Alex Kushleyev]

Hi @david-moro ,

We have been working on a number of EIS-related features and I am happy to report that I can share instructions for testing it very soon. I am going to push for releasing some beta docs for testing sometime this week. It may be some time before this feature will be officially released in an SDK, but I will merge everything from voxl-camera-server dev branch into the eis branch.

So far, the following features have been implemented:

• tested on IMX412 and IMX214 cameras (IMX412 is strongly recommended due to wider FOV and better lens and imager quality, much better low light performance)
• Input resolution up to full frame 4xxx by 3xxx
• output resolution up to 4K (stabilized)
• digital zoom (stabilized) + voxl-portal integration
• stabilized roi selection + voxl-portal click and drag
• support landscape or portrait camera orientation
• multi camera support (simultaneous EIS running on multiple cameras)
• multiple stream per camera (stabilized, unstabilized, different encoding bitrates)

• No rolling shutter correction (yet)

There is going to be a lot of things to document, so I can start with basic use cases so you could test them out. The only thing from hardware you will need is a VOXL2 or VOXL2 mini and IMX412 camera (M0107 or M0161). Onboard VOXL2 IMU is used.

Thanks for your patience

Alex

[david.moro]

@Alex-Kushleyev Fantastic! Looking forward to it.

[Alex Kushleyev]

@david-moro , sorry for the delay. I just posted an initial draft for setting up EIS on VOXL2:

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

I will be making some demo videos in the next few days.

Please go through the instructions and make sure you can get camera server and voxl-portal to run (you may need to restart the voxl-portal service after updating the package). By default, you should see a merged view in voxl-portal which consists of stabilized (left) and full frame view (with ROI drawn) on the right. Using the latest voxl-portal, you can interact with the ROI on both sides of the image (see the readme).

Once you set it up and play with zoom, you should update the default zoom to be higher than 1.0 so that you have some stabilization margin (in voxl camera server config file).

I just wanted to get this out, and i'll make a few demo videos very soon, but for now hopefully the instructions work and i will let you explore the features a little bit

You will need to build the voxl-camera-server and voxl-portal packages from the appropriate branches. Hopefully you can do that by yourself. If not, i can post the test debs.

You dont need to do any intrinsic or extrinsic calibration for the initial test (see readme for details)

Please let me know if you run into any issues.

I am going to add a lot more details shortly.

Alex

[david.moro]

@Alex-Kushleyev Thank you very much! I think we can take it from here

David

[Alex Kushleyev]

Hi @david-moro ,

I made several videos that show the current functionality of the ROI and EIS features.

• ROI features: https://www.youtube.com/watch?v=FXv4855WjNc
• EIS: https://www.youtube.com/watch?v=fi2BO_U5f-c
• EIS outdoor test : https://www.youtube.com/watch?v=oHDsiTD8Ky4

Please note that the last video was recorded with different EIS settings than currently are used as defaults. However, the same processing pipeline was used.

We will support up to 4K60 stabilized output and concurrent EIS on multiple cameras (already tested). There will be a lot more work done in the coming weeks to improve the video quality when testing on actual flying vehicles.

Alex

[m.costa]

@Alex-Kushleyev Looking forward to trying this out. A couple questions.

1. Which version of Qualcomm's EIS is this using? 1.0, 2.6, or 3.0?
2. What's the ETA for getting RSC implemented?

[Alex Kushleyev]

@m-costa , this is ModalAI EIS 1.0 . The computations are done within voxl-camera-server by processing the raw bayer data directly (cpu and gpu).

I actually have some updates and new videos will share shortly.

Also started testing on a flying platform (Starling 2 Max), will share initial results dual cam EIS (front facing and down facing).

I am curious what platforms folks would be using for testing EIS? One of ModalAI drones or something custom? If using a standard drone, i can document how to set up EIS for that type of drone, since there are a few things that need to be set up / verified in order for EIS to work properly.

Regarding Rolling Shutter Corrections, we have started implementing, will probably have some initial results in 1-2 weeks.

Alex

[m.costa]

@Alex-Kushleyev So this is a custom implementation, not using any of Qualcomm's built in tools? If so how are you planning on implementing the RSC? Reading through some of the QC documentation, it seems like the supported QC EIS versions (1.0, 2.6) for the RB5 only correct rolling shutter for linear motion on a per frame basis. That doesn't help much on a high vibration platform where the intra-frame motion is non-linear.

Are you planning on implementing X and Y rolling shutter corrections on a per line basis? That would be sweet.

I'm looking to use this on a custom quadcopter that has pretty poor vibration isolation. Eager to see your results so far with the EIS.

[Alex Kushleyev]

@m-costa , yes it is a completely custom implementation.

For rolling shutter compensation, we are working on per-line compensation (not just X, Y, but full 3D rotation because the vibration can occur along all three axes x, y, and z). However in order for this approach to work well in environment with lots of high frequency vibrations, an IMU would need to be mounted directly on the camera (very rigid mechanical coupling, so that IMU picks up the exact vibrations that the camera is experiencing).

Unfortunately we do not currently offer any cameras with a built-in IMU. However, VOXL2 / VOXL2 mini both have an additional SPI port which can be used for an IMU. The current approach is to use an external IMU in addition to the IMU on VOXL2 and compare the EIS results. Voxl-imu-server already has support for an external IMU over SPI.

Also i should note that it is unlikely that we will be testing on RB5-based platform (because it is EOL in terms of our support) - I am not sure whether you were implying that you are using an RB5.

Finally, you should also investigate your camera mount. Vibration reduction is a whole separate subject, but you should make sure that the camera mount is very rigid and it is rigidly attached to the frame. Sometimes a little flex in the mount can amplify the existing vibrations, so in this case the solution would be to actually make the mount more rigid instead of isolating it with a soft material. Actually using dual IMU approach can reveal if the camera is vibrating more than the main board (voxl2) by comparing the IMU data.

Speaking of the camera - what camera are you using on your quadcopter?

Finally, some new videos. No RSC yet.

• dual camera EIS on Starling 2 Max (hand carried) : https://www.youtube.com/watch?v=-BA_nU4kjQs
• down-facing camera EIS on Starling 2 Max (flight) : https://www.youtube.com/watch?v=-B5xKmBBCAc

Notes about videos

• the ROI tracking still needs some work (it has not been tuned for the down-facing use case)
• down-facing camera had almost no vibration picked up by the camera. Some rolling shutter artifacts can be observed in a few cases.
• the front camera was also tested, but due to vibration + rolling shutter, the video did not turn out great, so I did not post it. Camera mount needs a little bracing to remove flex and will be tested again. Once I fix the camera mount bracing issue, i can post the before and after videos.
• in the dual cam video, you can see some rolling shutter artifacts which appear as slanted vertical features and a little motion due to shaking. This type of distortion is the "easy" one to fix using RSC.

Alex

[m.costa]

@Alex-Kushleyev Sounds like we are pursuing the same sort of hardware setup. We have been using the SPI port connected to a 42688 IMU mounted directly on the rear of an OV64B sensor.
If you aren't targeting VOXL2/VOXL2 mini (both RB5 based), what hardware are you targeting?

[Alex Kushleyev]

@m-costa , the target platform is indeed voxl2 / voxl2 mini. Just to clarify, RB5 and Voxl2 both use the same SOC, which is qrb5165. RB5 is the name of the development board from Qualcomm. One of our EOL dev kits was using RB5 boards.

Is your ov64B part of Hadron module or standalone? We do have support for Hadron and I was planning to test EIS with that camera as well.

Alex

[m.costa]

@Alex-Kushleyev Ahh I see. I was just using RB5 as an abbreviation for qrb5165.
It is a standalone module, but since the hadron just passes through the OV64B MIPI/I2C its quite similar.

Qualcomm advises using the SSC QUPs to interface with IMUs for EIS (I'm guessing for lower latency). Have you seen any differences in latency when comparing the SSC connected 42688 onboard the voxl2 vs the one connected to the generic QUP? Additionally, are you utilizing the 42688 interrupts to synchronize the data with the frames from the camera?

[Alex Kushleyev]

@m-costa ,

SSC can certainly achieve lower latency in terms of uncertainty of the time overhead for doing an SPI FIFO read from the IMU sensor. This is true because the SSC is typically doing a lot less than the CPU, but the SSC is still not a real time processor (compared to a "bare metal" microcontroller implementation). However, i believe the imu server on CPU is running at high priority and also we recently (6 months ago) made an update in the Linux kernel to increase the priority of the SPI interrupts. : here is the change that enables RT SPI message queue priority, if that is of interest: https://gitlab.com/voxl-public/system-image-build/meta-voxl2-bsp/-/commit/b5c3189536a80e8ea55846e7705dadbc5a597ff8

There is some logic to estimate the timestamps of each sample in voxl-imu-server with a fudge factor which as been verified against running VIO, which estimates the time offset between IMU and global shutter camera. Here is a link to the icm42688 driver and the note about time sync https://gitlab.com/voxl-public/voxl-sdk/services/qrb5165-imu-server/-/blob/master/server/src/icm42688.c#L68 .

There is also a timestamp filter which helps with removing timestampt jitter coming from the individual FIFO reads. Since the IMU sensor is running at a constant rate (in FIFO mode), there will not be any short term fluctuations in the sample read times on the sensor itself. The timestamp filter + fudge factor should do a good job quickly latching on to the time sync.

I have also recently done an overlay of camera frame with a rolling shutter artifact and overlaid IMU data at 4khz (well, this is actually a yaw estimate change since the start of the frame show as the vertical line running down the middle). The vertical line shows deflection when the IMU / camera start rotating. It is not perfect because in reality the rotation is 3D but only yaw is shown.

Please ignore the light "waves" here as they are from LED lighting and very low exposure. I reduced the exposure to 1 or 2 ms to avoid motion blur.

I do realize that the IMU sync fudge factor was tuned at 1khz (per note in the link above) and I changed the frequency to 4Khz, so I will definitely double check this, but it looks OK based on the image. It is good enough for initial testing, at least.

The plan for now is to use the SPI IMU connected to the regular QUP, accessed from CPU. Once the rolling shutter compensation pipeline is up and running, the sync can be tested and I am planning to have a way to run the pipeline offline using logged RAW images and IMU data to replicate the exact real-time processing pipeline (so that different IMU time offsets can be tested for dev purposes)

Alex