Running QVIO on a hires camera

Rowan Dempster

Hi Modal team,

Has anyone tried to run QVIO on one of the MPA pipes produced by a hires camera (either IMX214 or 412 I suppose)? Wondering if that's feasible, or if the rolling shutter on those sensors makes feature tracking quite impossible under dynamic movements. Happy to try it out myself if the answer is "unsure"!

Thanks,
Rowan (Cleo Robotics)

Alex Kushleyev

@Rowan-Dempster ,

We have not tried this recently, but it should work. Here are some tips:

Use IMX412 camera (M0161 or similar) because it has great image quality and the fastest readout speed of all of our cameras (IMX214 is not recommended for this, it is an old and "slow" camera sensor)
- faster readout = less rolling shutter skew
use the latest camera drivers, which max out the camera operating speed in all modes : https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250919/imx412_fpv_eis_20250919_drivers.zip
The readout times are documented here for all modes : https://docs.modalai.com/camera-video/low-latency-video-streaming/#imx412-operating-modes
for example 1996x1520 (2x2 binned) mode has about 5.5ms readout time, which is pretty short
QVIO (mvVISLAM.h) has a parameter "readout time", which suggests that it supports rolling shutter. I have not tried it myself, but i heard that it does work.

mvVISLAM_Initialize(...float32_t readoutTime ..)
@param readoutTime
      Frame readout time (seconds). n times row readout time. Set to 0 for 
      global shutter camera.  Frame readout time should be (close to) but 
      smaller than the rolling shutter camera frame period.

Here is where this param is currently set to 0 in voxl-qvio-server : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/main.cpp?ref_type=heads#L371

in order to correctly use the readout time, you have to ensure that the camera pipeline indeed selects the correct camera mode (for which there is the corresponding readout time) : https://docs.modalai.com/camera-video/low-latency-video-streaming/#how-to-confirm-which-camera-resolution-was-selected-by-the-pipeline
- also, readout time is printed out by voxl-camera-server when you run it in -d mode (readout time in nanoseconds here):

VERBOSE: Received metadata for frame 86 from camera imx412
VERBOSE: 	Timestamp: 69237313613
VERBOSE: 	Gain: 1575
VERBOSE: 	Exposure: 22769384
VERBOSE: 	Readout Time: 16328284

keep the exposure low to avoid motion blur (IMX412 has quite a high analog gain, up to 22x and 16x digital gain). If you want to prioritize gain vs exposure, would need to tweak the auto exposure params in camera server (when you get to that point, i can help you)..
it would be interesting to compare performance against QVIO with AR0144 - that would probably require collecting images from AR0144 and IMX412 (side by side) + IMU data and running QVIO offline with each camera.

Good luck if you try it! let me know if you have any other questions. Please keep in mind that QVIO is based on a closed-source library from Qualcomm and our support of QVIO is limited.

Alex