ModalAI Forum

Hi ModalAI team,

We are working on high-speed mapping. We are currently looking at matching timestamps, but there seems to be a lot of different implementations in the source code. Can you confirm which clock these timestamps are in, and if any of them are synced?

voxl-camera-server, timestamp for image capture at exposure start, or at image published to pipe?
voxl-tflite-server, we use the image metadata to overwrite the ai_detection_t timestamp.
voxl-mavlink-server, we need to match odometry to image exposure time, this is our biggest confusion atm.

And in general, which function should we be using to get the current time on the voxl?
apps_time_monotonic_ns()?

Best regards,
Peter

@Alex-Kushleyev

Also, I came across this comment in your source code: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blame/master/src/model_helper/yolov8_model_helper.cpp#L167

Is this an old comment, or does YOLOv8 with GPU delegate still not work in SDK 1.6?
Potentially, we would want to use YOLOv11 also for the improved mAP.

Peter

@Alex-Kushleyev

Thank you for the thorough reply.
We are a student organization (~40 people) working on a custom drone swarm based on your ecosystem.
We would be happy to meet online with you and your team to talk more about our efforts, if that works for you.

We are currently working on integrating our code and making custom services on the VOXL.
The plan is to run the YOLOv8 on the GPU.
Considering the EIS runs on the GPU, I don't think compute power will be much of an issue.

It didn't hit me that the local distortion for the AprilTags would be so little; that's a great point.
I think the optimal solution for us is to have a dual IMX412 setup with 4K and then a 2x2 binned 1920x1080 -> 1280x720.
We will be operating in a well-lit area, so I think we can lower the exposure time significantly to reduce the motion blur.

Peter

@Alex-Kushleyev

Alright, the AR0144 is definitely out of question then.

The quality of the video with EIS is very good and should definitely be good enough for us.
It seems like the rolling shutter skew at 12ms is not a big issue for object detection, but I'm curious how
it will perform when trying to analyze very small AprilTags from distance. I estimate that the tags will cover about 2.5% of the image height.

Do you have any data about the computation power spent for dual 4K streams + 1080p EIS? I'm worried that will leave little room for a YOLOv8 tflite model
in addition to other services and code that has to run.

Also if there are any documentation or sources where I can read more about the 2x2 binning that would be good. It's not entirely clear for me yet
where and when the binning happens, and how it affects readout and if the VOXL does the binning requiring compute power.

Peter

@Alex-Kushleyev

Hello! Sorry for the late reply, we have been moving offices for a week.

We are experimenting at the time and have not landed on a specific requirement yet, so rolling shutter is not a strict requirement.
Our plan was to use an AR0144 for object detection and then an IMX412 (4k resolution) to analyze the object we detected.

We would prefer to test a global shutter setup so that we can compare it to our IMX412, but if that is not readily available then maybe we should try the 2x2 binned mode you mentioned, and possibly we could use two IMX412.
We estimate to be flying at a speed of 4 m/s at a height of 5m, and our image height with the current lens at that height is 1m.
So with a 4.2ms readout time that will be a max skew of ~18 pixels in 1920x1080 mode?
It seems like the 1920x1080 can be reduced to 1280x720 with binning, but that is a factor of 1.5 and not 2x2, how does that binning work?

Also, would electronic image stabilization also help to make the image clearer or is that simply to fight vibrations or unpredictable motion?

Lastly, if I understand this correctly, we could try to:

• Lower exposure time
• Choose 1920x1080 resolution for faster readout time
• Use 2x2 binned more for less rolling shutter distortion

Peter

Hello ModalAI Team

I am looking for a global shutter color camera, preferably 1280x720 resolution or similar. Reading other posts it seems as if the AR0144 has been tested to work, but still needs some development to integrate and function well with the VOXL 2.
Is this only in the situation where it will be used for VIO? I will be using it as an object detection camera, and therefore it should be global shutter preferably for clear images. I can't seem to find any global shutter color cameras in your store. Is it possible to purchase this somehow?

I have tried emailing a few times about this for the past months, but can't seem to have received a response or confirmation that my email was sent.

Best Regards
Peter