Camera Intrinsics, VIO and more..

[Dinesh Varun Kandiyappan]

Dear Team,

1. I am looking to obtain the following intrinsic calibration values for the Starling 2, but I was unable to find them on the drone. It would be great if these calibration parameters are available: RGB HiRes IMX412, PMD ToF intrinsics.
2. I did find intrinsics for both the forward and downward-facing tracking cameras, as well as extrinsics for all onboard cameras, can I safely assume these values are accurate?
3. I also noticed that the ToF sensor performs poorly outdoors( which I understand is a known limitation), are there any recommended approaches for obtaining metric depth for outdoor SLAM, beyond using monocular depth estimation models? I came across documentation mentioning stereo modules, but I don't see them listed on the products page. Are there plans to make them available, and if so, what depth range can we expect from them?
4. I also noticed that both qVIO and Open-VINS are available onboard. Before I run empirical tests to compare them, do you have any recommendations regarding their usage in terms of accuracy and suitability for different conditions such as indoor vs. outdoor environments?
5. I noticed that the PX4 airframe XY positions appear to be swapped across all the drones. is this intentional?
6. Finally, we are trying to run a deep learning model on the VOXL2, what would be the recommended approach for reading and feeding inputs to the model at high frequency with low latency, while also sending control commands to PX4?

Thank you !

[Dinesh Varun Kandiyappan]

@alex-kushleyev

[Alex Kushleyev]

Hi @dinesh-varun-kandiyappan , please see responses below:

1. the high resolution cameras (IMX412) are not calibrated during production. PMD tof does have intrinsics avaiable and they are downloaded from the sensor directly when the voxl-camera-server strarts. We did make a feature branch to print out those intrinsics for the TOF sensor - we can revisit that (and mainline it). You can use our tool voxl-calibrate-camera to perform intrinsics calibration of the fisheye IMX412 camera. We also have some improvements to make the process more efficient for high resolution images - if you need that, we can share the update (on todo list to mainline). https://docs.modalai.com/calibrate-cameras/

2. all the extrinsics are obtained from the 3D CAD models (not calibrated). they should be pretty accurate, but we do not have any data / analysis on the accuracy

3. Yes, TOF sensor is an indoor sensor because outdoors the IR light will interfere with the sensor. We used to have drones that used DFS (Depth From Stereo, which used an algorithm from Qualcomm), but that stereo camera setup (using old ov7251 cameras) is EOL and we are not actively developing / testing DFS. However, you could potentially put together two AR0144 tracking cameras in a stereo configration and try DFS.

4. QVIO only works using a single camera, but Open-Vins supports multiple cameras and that provides more reliable VIO solution. We are not developing QVIO, but Open-Vins is under active development and testing, so you should probably start with that. Also, QVIO is closed source (and not being actively developed), so it has no potential for updates / improvements.

5. Can you please clarify? the PX4 params should be set up correctly for each drone after you run voxl-configure-mpa

6. Can you please clarify? Is the goal for your model to control the vehicle's flight? (if so, what type of controls?)

Alex

[Dinesh Varun Kandiyappan]

@alex-kushleyev Thanks for the response. I missed this since I didn't receive any notification. It would be nice if it gets notified.

1. I used voxl-calibrate-camera with the -f flag since the lens looked like a fisheye, but it always errored out. Without the fisheye parameter, it worked fine, with a reprojection error of 0.43. I would love to get your update on more efficient processing. May I know which feature branch prints the intrinsics for the ToF camera?

2. Okay, thanks for confirming.

3. I'm building a monocular depth model that predicts relative disparity, which I then scale to metric using VIO features + ToF as anchors. The hard part is reprojecting tracking-front features into the hires frame, since the depth model runs on hires (RGB works much better for these models). Hires has a narrower FOV, so most features fall outside once reprojected.

4. I tried features from both qVIO and OpenVINS. qVIO consistently gives far more. Measured over 250 frames from the same log:

   • qVIO: ~64 features published, ~26 land in the hires FOV
   • OpenVINS: ~17 published, ~2 land in the hires FOV

   Looking at starling2/estimator_config.yaml, I'm guessing this comes from num_pts: 50, max_slam: 30, max_msckf_in_update: 10. Is that right, and is raising them the correct way to get more features published? The transport caps at 64 (VIO_MAX_REPORTED_FEATURES), so I assume that's the ceiling.

5. CA_ROTOR*_PX and CA_ROTOR*_PY look swapped relative to the physical frame. Measuring motor-center to motor-center: front-to-back (X, forward) ≈ 0.066 m, side-to-side (Y) ≈ 0.088 m, so forward is the shorter dimension. But by default the params have it the other way round on all drones we have:

   CA_ROTOR0_PX [87,258] : 0.0850
   CA_ROTOR0_PY [88,259] : 0.0625
   

6. The end goal is an RL policy subscribing to odometry and image topics, outputting PX4 offboard commands.

A few other things I've run into:

7. I want to add the hires stream to OpenVINS as an extra camera, so I don't have to reproject from tracking front. I hope rolling shutter won't affect it much. Also, does it strictly need misp ion?

8. I tried some transformer-based models, but tflite doesn't support several ops, technically it does, but not for the specific shapes I need. Is there a newer voxl-tflite-server with broader op/shape coverage? I also spoke to Qualcomm, and it sounds like the QRB5165 is old enough that anything beyond CNNs is difficult.

9. We are in a situation to run the following at once: custom object detection model, depth estimation, an RL control policy consuming the depth output, GPS, plus streaming hires to the ground station over gstreamer at high rate. Still building, but I'm doubtful all of it fits comfortably on VOXL2. I'm surprised that thermal throttling started with just the single depth model in tflite-server, while GPU utilisation was only ~50%. We're looking at heatsinks and hoping propeller airflow plus a heatsink together are enough and would work in sync.

10. Worst case, is it possible to pair and run two VOXL2s on a Starling 2? Any guidance would be appreciated, and hopefully it doesn't add much payload (11 g plus connectors for the VOXL Mini, I guess). I also read about VOXL3, and it looks very promising. Any idea when it's releasing, and is there a pre-release programme for academic research labs?