Topics created by Rowan Dempster

@Moderator Hi Modal,

Does this mean you can create a custom data file partition that you can flash using fastboot?

Yes that is correct.

We have not explored this before but it has been asked a few times. This could be very helpful for other developers.

I'm certainly hopeful that it will be helpful here at Cleo Robotics! So far with my prototyping it works as expected and cuts flashing time of some large docker images we have here at Cleo down by a noticeable fraction (no file overhead via fastboot like with ADB).

I think the snippet I posted covers the baseline functionality of getting a custom "payload" into the data partition. However if there is more I can elaborate on in terms of the toolchain / what's in the payload, and if that elaboration will be helpful to other VOXL2 developers, I would be happy to elaborate 🙂 Just let me know!

Other similar discussion points I tackled recently that I'm happy to talk about lessons of:

Flashing the system image and VOXL/CLEO SDK through a Windows Machine (journeys in USB device drivers) Building Flutter applications for uniform flashing process across all operating systems Building release bundles (i.e. a collection of partition binaries) in CI

Hi @Alex-Kushleyev,

Resurrecting this old thread, we now have the IMX412 on a drone and we are now ready to give to VIO on the IMX412 our full attention and lots of testing effort. Where I'm at now is I have a prototype working and QVIO does run on the IMX412 camera and outputs estimates that seem reasonable, but I'm 100% sure it's not configured as good as it could be cause I made so many assumptions that I would like your input on:

Which camera data / pipe to use
Ideally, we would like the IMX412 VIO to perform close to (or of course better than!) a ar0144 tracking camera, in terms of the quality of the image for feature tracking, low CPU usage, low latency frames, etc etc etc. In this spirit, I've been looking into how to get the MISP normalized pipes coming from the IMX412 and also how to get the camera server producing ION data to get the same CPU usage gains we saw in https://forum.modalai.com/topic/4893/minimizing-voxl-camera-server-cpu-usage-in-sdk1-6.
I saw that in the pipe setup, the normalized code for IMX412 was commented out
4f2e7cff-3212-46c7-9929-988f0ce413e1-image.png
After commenting it back in I was able to see in the portal a decent looking normalized stream. I also see the ION pipe pop up for that norm stream but I haven't tried that ION pipe on the QVIO server yet (I'm confident it would work though, just waiting for https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/commit/d18521776e3e88f396d85aa657769c47f29e9c9f to get tagged!).
Do you see any issue with using the MISP norm pipe for IMX412 VIO, or is that actually what you would recommend?

Which resolution to use
I know you talked about some resolution advice above, but I'm a little bit confused on the specifics on where to put those numbers. You had suggested 1996x1520 for a 5.5ms readout time. Do these numbers go into the Preview Width config fields? Here is the entire diff of the config settings I have been using for my testing:
b75275d6-9c35-48ac-a95c-a59b439e7f54-image.png
The other values I have a question about in that diff is The MISP width / height fields, I chose 998x760 which is half of the Preview Width resolution you suggested. I did this because I wasn't sure of any compute bottle necks that would pop up if I fed a 1996x1520 image into QVIO. Do you think 998x760 is good or maybe I should pick a like 0.75 downsample so something like 1497x1140 for the MISP width/height.

Camera Driver files to use and how to version control and deploy those
Could you confirm that the binary files in https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250919/imx412_fpv_eis_20250919_drivers.zip are still the latest and the recommended binaries to use? Could you also advise on how to version control these files and deploy them to the voxl2 when the camera server .deb is deployed? I want to keep all files related to bringing up the camera in the voxl-camera-server debian if possible. I see some binaries files being stored in this path:
03191a03-455e-4e18-bd50-dd19b9d5c028-image.png

So if I understand the process correctly, those files will end up in /usr/share/modalai/chi-cdk/imx412-fps-misp. Is that where the voxl-configure-cameras C looks for them? Also, do I have to do anything with the com.qti.sensor.imx412_fpv.so file in the zip link that you sent, or do I just ignore that file?

My end desired behavior is that when I install the voxl camera server .deb, I don't have to worry about also copying binary files over to the voxl, or moving any files around on the voxl, or having to remember to run voxl-configure-cameras C. So maybe the path forward there is to have all files deployed into the right places by the .deb install and then in the postinst script auto run voxl-configure-cameras C? What do you think?

Aspect ratio concerns and their affect on field of view and camera calibration
Is the aspect ratio you suggested (1996x1520) the actual aspect ratio of the sensor? Or does the sensor support multiple aspect ratios, or is there something more complex going on I don't understand here? I just want to make sure that we're using as much FoV as the sensor supports. That should be the goal for VIO feature detection and for streaming right, maximize field of view?

Also, how does changing the aspect ratio / resolution affect the camera calibration? We're using kalibr which asks for a focal length bootstrap, which we've been giving it 470 for the ar0144 camera, do you know of a way that we can figure out an accurate focal length bootstrap for the images that we end up using for the IMX412?

How to not lose other IMX412 features like 4k recording and EIS streaming etc
This is kinda my biggest concern about the feasibility of this whole thing: Will we still be able to get the other awesome IMX412 features like high quality streaming to the GCS with EIS, as well as high quality 4K recordings even in difficult low light environments, AND at the same time optimize the IMX412 for VIO which demands stuff like fast readout for less skew?

Any advice you can give here on mapping out the tradeoffs? Are there any non-starters like not being able to get 4K recording if we opt to use the MISP norm pipe for VIO? Or are you confident that we can get the best of all 3 worlds 🙂

Exposure time concerns
I agree with your initial point that needing low exposure for low motion blur is important. As I mentioned in the intro to this message, I have a prototype working and I am now at a place where I can indeed tune the gain vs exposure / auto exposure params. Could you help me with that? I assume that this tradeoff also applies to the ar0144 camera, any lessons I can take from there?

QVIO readoutTime param
Great find and thanks for pointing that out!! To confirm the specific numbers here, if I use the 1996x1520 preview width/height which has a documented read out time of 5.5ms (I should confirm this using the -d mode), then I should put 0.0055 for that parameter?

As always, thank you for your help in camera related matters, we would be no where close to where we are now with robotic perception without your guidance!

To clarify, our current platforms are shipped with voxl-open-vins-server as the default VINS solution; however, QVIO is still available (SDK 1.6.2).

Hi @Rowan-Dempster ,

We just merged the changes to dev : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/merge_requests/39.

A new build of the libmodal-pipe library will be added after tonight's nightly build here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ .

Any SDK releases after this will include this support.

Alex

@Alex-Kushleyev The IMU is being configured with an 8K ODR so by increasing IMU_GYRO_RATEMAX you are reading the FIFO more often but reading less samples each time. So you are mainly increasing the overhead of context switching. Can you characterize how much the load increases just by increasing IMU_GYRO_RATEMAX and not doing any of the extra UART IO? Unfortunately we don't have a lot of control over the low level implementation of the IO drivers that are in the Qualcomm code. And there is no DMA that could help lower IO overhead. One idea would be to lower the ODR to 1K so that you are only reading one sample from the FIFO at each interrupt.

@Rowan-Dempster ,

Here is where the raw data from the TOF sensor comes into voxl-camera-server : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/hal3_camera_mgr.cpp?ref_type=heads#L1242

The data is in the MIPI12 packed format. The TOF library expects RAW16 format, so the data is converted from MIPI12 to RAW16 and fed into the TOF processing library. It is possible that dumping the RAW data is what RRF recording would be doing, so you could add that functionality to camera server to write the raw data to file. I could also help with publishing the RAW12 data as is to a mpa channel, so you can log it outside of the camera server (for example using voxl-record-raw-image utility. You could clarify with PMD if that would work (in other words, you would need to know the log format of the RRF recording)

We have not tried enabling the raw data logging. However, if you know what function needs to be called within PMD library (maybe you just enable logging, and it does everything for you), you should be able to add it somewhere in the TOF init code, maybe here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/tof_interface.cpp?ref_type=heads#L982

You can find the API header files for Royale / Spectre in /usr/include/royale either on VOXL2 directly on in /opt/sysroots/qrb5165_1/usr/include/royale the voxl-cross:V4.4 docker.

Hopefully that helps!

Alex

@Alex-Kushleyev Gotcha, thanks for the info Alex and your help resolving this issue!

@Rowan-Dempster No problem! Kind of primitive but effective 🙂

(top is Royale 4, bottom is 5)

@Rowan-Dempster ,

I myself tried to do this some time ago and ran into issues, which i could not resolve. I am sure also someone else at Modal also tried it before me.

If I remember correctly, my issue was that it could not get camera module in the second instance of the camera server, on this line, and i did not pursue this further (did not spend too much time on it though).

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/hal3_helpers.cpp?ref_type=heads#L79

So in your case you are able to get the camera module?

To help debug, you should run logcat while you are trying this and see what messages might show up there.

And, finally, just wanted to mention that we don't know if what you are doing is possible, but if you want to keep pursuing this direction, I can help if i know the answer to your questions, but may not be able to spend time working on it right now.

EDIT: in order to see if separate instances of camera server are possible, you could try to see if you can run multiple instances of gstreamer plugin that uses qmmfsrc. I found a doc that should be helpful in testing this, take a look : https://thundercomm.s3-ap-northeast-1.amazonaws.com/shop/doc/1596593567074634/Thundercomm EB5_Multimedia SDK User Guide_V1.1_ie89e.pdf

Specifically, the following command should work, although i did not test it:

gst-launch-1.0 -e qtiqmmfsrc name=camsrc ! video/x-raw\(memory:GBM\),format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! omxh264enc control-rate=max-bitrate target-bitrate=6000000 interval-intraframes=29 periodicity-idr=1 ! queue ! filesink location="/data/vid.h264"

Also, i dont see how you sent the camera id in this example, but it should definitely be possible 🙂

See if you can run two independent instances of gstreamer using different cameras. I think it should work, but not sure.

Alex

@Rowan-Dempster I think it is safest to just not use data that reads 0,0,0 as the sensor cannot measure that close anyways

About 95C is when the temperature control loop will kick in and start reducing the maximum core frequencies (gradually). You can monitor the cpu usage and current core frequencies using voxl-inspect-cpu.

Here are the maximum core frequencies for all cores:

cpu0 1804.8 cpu1 1804.8 cpu2 1804.8 cpu3 1804.8 cpu4 2419.2 cpu5 2419.2 cpu6 2419.2 cpu7 2841.6

If you set the cpu governor mode to perf (voxl-set-cpu-mode perf), it will pin all the cores to max frequency and they will stay there unless they are being throttled due to temperature, which you can check using voxl-inspect-cpu.

@Sarika-Sharma We have a variety of parameters sets available in the voxl-px4-params repo https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-px4-params/ However, they are mostly for either indoor or outdoor use. Older versions of PX4, like what our fork is based off of, had a hard time with external vision based navigation coexisting with GPS based navigation. Newer versions of PX4, supposedly, are much better at this but we have not spent the time to prepare a parameter set for that yet. You are free to experiment with our mainline PX4 build here: http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/voxl-mainline-px4_1.17.0-1.0.2-202602240859_arm64.deb. However, if you decide to use the mainline build, be aware that it is considered "experimental" at this point and has very little testing on voxl2 so use extreme caution.

@Rowan-Dempster No, it isn't

@Rowan-Dempster They've been released and are a part of the SDK but we just don't install them by default to save space.

The .debs are onboard already and should install with an apt install

In that context "blackisted" just means don't install by default.

@thomas I assume we don't want to be dealing with 4k raw arrays on the CPU for performance reasons. So yeah as early as possible in the image proc pipeline, and how early we can push it is what I'm asking for advice on. We'll also reach out to @Alex-Kushleyev thanks!!