ModalAI Forum

@Vinny Hi Vinny, no worries we are going to change the electronics so that the other SPI device talks over SPI bus 11.

Setup

• Board: VOXL2 (M0054)
• System image: 1.7.8-M0054-14.1a-perf, kernel 4.19.125
• libqrb5165-io 0.5.0
• Bus 3 controller: qcom,spi-geni, bound to the spi_geni platform driver, exposing a single chip select (spi3.0 → /dev/spidev3.0)

We are adding an ams TMF8829 time-of-flight ranging sensor to an existing VOXL2 design. For layout reasons it is wired to SPI3, the same bus as the onboard ICM-42688 IMU that voxl-imu-server services. The IMU runs at SPI_MODE_3 on the controller's native chip select; the TMF8829 runs at SPI_MODE_0 and is selected through a TLMM GPIO (117), with its enable line on GPIO 118.

The problem

SPI3 exposes exactly one native chip select, and it is wired to the IMU. The GENI controller asserts that native CS on every transfer it performs, so any transfer we issue for the TMF8829 also selects the IMU. Since the TMF8829 wire protocol begins each transaction with a write opcode (0x02), the IMU interprets TMF traffic as register writes addressed to itself. That is not merely a corrupted read — it can reconfigure the IMU mid-flight and take down a flight-critical data stream.

The standard Linux remedy is the SPI_NO_CS mode flag, suppressing the controller's chip select so our GPIO alone selects the target. The kernel rejects it on this platform. Issuing the ioctl directly on the descriptor returned by voxl_spi_get_fd(3):

spi_bus3: SPI_IOC_WR_MODE32 0x40 failed: Invalid argument

0x40 is SPI_MODE_0 | SPI_NO_CS. Both the legacy 8-bit SPI_IOC_WR_MODE and the 32-bit SPI_IOC_WR_MODE32 return EINVAL, which points to SPI_NO_CS not being advertised in the controller driver's mode_bits. Separately, voxl_spi_init() in libqrb5165-io validates its mode argument strictly against SPI_MODE_0..3 and rejects any extra flag bits, so the flag cannot be requested through the library either — but we bypassed that with direct ioctls on the raw fd and the kernel is the hard stop.

Our integration treats this as a safety gate: at startup it writes SPI_MODE_0 | SPI_NO_CS, reads it back, and if the readback fails it disables the TMF entirely and leaves IMU publication untouched rather than risk corrupting the IMU. That gate is what fires today, so the second sensor never comes up.

For completeness, the surrounding work is done and validated on hardware. We have a bus-3 arbiter that serializes access with a mutex, switches mode and clock per transaction, and restores SPI_MODE_3 with native CS on every success and error path. Matched 10-second A/B captures with the TMF path compiled in but gated off show no IMU regression: 1024 Hz in both conditions, 0.977 ms mean sample interval with no interval above 2 ms, and zero FIFO errors, logger drops, or mode-restore failures. The TMF8829 itself is known good and streams reliably at 15–18 MHz when it has a bus to itself. The only missing capability is per-device chip-select control on a shared bus 3.

Path 1 (preferred): declare bus 3 with cs-gpios and a second spidev child node

Rather than work around chip select in userspace, we would rather let the kernel own it, which is how Linux is designed to share an SPI bus. Concretely, on the SPI3 node: raise the chip-select count to 2, add cs-gpios with the native CS in slot 0 and &tlmm 117 in slot 1, and add a second spidev child at reg = <1>. The TMF8829 then appears as its own /dev/spidev3.1 with its own spi-max-frequency and mode.

With that in place the kernel handles chip-select assertion, clock, and mode switching per device. On our side we delete the manual GPIO chip-select toggling and the entire mode-restore arbiter, and the IMU is never exposed to TMF traffic at all — a substantially safer arrangement than anything we can build in userspace.

Questions on this path:

1. Does the spi_geni driver in the 1.7.8 kernel cooperate with the SPI core's generic cs-gpios handling, or does its own set_cs implementation bypass GPIO chip selects?
2. Is there a supported way for us to apply a devicetree overlay or a patched DTB to a VOXL2 system image, or does this have to come from ModalAI as an image change?
3. Are there known constraints on SPI3 specifically — reserved chip selects, pinmux conflicts on TLMM 117, or other consumers of that controller we should know about?

Path 2 (fallback): add SPI_NO_CS to the controller driver's supported mode bits

If the devicetree route isn't viable, the smaller change is to advertise SPI_NO_CS in the GENI SPI controller's mode_bits and honor it in the driver's chip-select path. Our existing arbiter code already works unmodified against that interface, so this would unblock us with no changes on our side, and it would also need voxl_spi_init() in libqrb5165-io to stop rejecting mode values with flag bits set (or we continue issuing raw ioctls on the fd from voxl_spi_get_fd()).

We consider this less idiomatic than Path 1 — chip select stays a userspace responsibility, with per-transaction GPIO toggling and mode restoration that has to be correct on every error path — and it still requires a custom kernel build. But it is a much narrower diff if that matters for your release process.

Question: is SPI_NO_CS omitted from mode_bits deliberately, for a hardware reason on the GENI controller, or simply because nothing has needed it yet?

What we're asking

Guidance on which of these two you'd support, and whether either can be delivered through an official system image rather than a fork we have to maintain. If neither is practical in the near term, our fallback is a hardware change moving the TMF8829 to its own SPI bus, and we'd appreciate confirmation that a shared bus 3 is simply not supported on this platform so we can commit to that redesign.

Happy to share the arbiter source, the A/B log captures, or a minimal reproducer that only issues the failing ioctl if that's useful.

Hi @Alex-Kushleyev ,

Just posting here to get your eyes on https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/merge_requests/72 and potentially open a dialogue about how Cleo collaborate more with ModalAI on voxl-camera-server code changes.

@Alex-Kushleyev Hi Alex, just to give you an update of where I am at: I have successfully logged the raw10 bayer and misp norm pipes of the IMX412 using voxl-logger on the https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-logger/-/tree/extend-cam-logging?ref_type=heads branch. I have also successfully replayed the misp norm ION pipe alongside the IMU pipe and run voxl-qvio-server offline on that data.

I noted that the output of voxl-qvio-server is not deterministic across replays, which is expected since there is no deterministic mechanism to ensure that voxl-qvio-server initializes using the same camera frame, and any slight jitter on which frame it initializes on of course will change the trajectory of the algorithm. However, I believe the replays are deterministic enough to draw conclusions about performance, so I am not too concerned.

This week my plan is to move on to the next step you outlined: replaying the raw10 bayer pipe and running the offline misp implementation. Please let me know if any of the details you outlined in your previous posts have changed or you have any new advice since your original posts 3 weeks ago

Thank you,
Rowan

@Alex-Kushleyev Hi Alex,

Thanks for looking into getting the perf optimizations branch into dev!

Cool good to know that intrinsics can scale like that. The calibration I did was actually at the 998x760 MISP output res. If I then decide to run QVIO on the full 1996x1520 image, then I'll scale the focal length and principal points by N=2. I don't think I'll ever run QVIO on the full 4k res.

I started to look into logging raw10 bayer, but I don't see any way "to use the ion buffers to log the raw10 images" in voxl-logger (as you suggested). Could you clarify if that is a feature you are currently working on, or is ION logging currently supported in voxl-logger and I just missed it?

@Alex-Kushleyev Hi Alex,

I thought before doing anything else I should get the Cleo fork of the voxl-camera-server up to the latest master (v2.2.21). We are currently using v2.2.17 from a couple months ago PLUS the commit on the perf-optimizations branch.

However, when I did the upgrade I noticed a slight uptick in CPU util, probably because the perf-optimizations commit never made it into the master branch? So my next action was try to rebase perf-optimizations off of v2.2.21, but looks like that's a non-trivial rebase because of changes between v2.2.17 and 2.2.21.

Here's the branch where I tried to rebase and just accepted all incoming, unfortunately it does not compile: https://gitlab.com/rowan.dempster/voxl-camera-server/-/merge_requests/1

Could you advise on how to get the perf improvements on top of the latest 2.2.21 release? Or are you planning on merging those perf improvements into the mainline at some point soon and I should just wait for that?

Thank you,
Rowan

@Alex-Kushleyev Hi Alex,

Thanks for the suggestions on how to progress from simpler replay setups to the most advanced one that we are shooting for - I agree that I should validate the replay pipeline step by step on a simple toy data set before shooting for whole thing or collecting a whole dataset suite.

I like your idea of having two IMX412s mounted side by side to allow me to A/B Test the effect of what I'm going to call "physical camera settings" (raw frame res, exposure, gain). Once I'm happy with the state of the data replay pipeline and have the"misp settings" changeable in that pipeline, then I will make an assembly for the mounting of two IMX412s to improve the parameter sweep coverage to the physical camera settings. Of course those physical camera settings parameter sweeps will be more tedious because I can't change the parameter between replays - I will need to change the parameter and then re-record a new dataset and try to get signals on which settings are better / worse that way.

Regarding my question about product feasibility: Thank you for shedding light on the main tradeoff being the unbinned vs 2x2 binned mode, and how that binning mode impacts VIO via the readout time vs. the impacts recording via lower quality. We have decided to still proceed with the analysis to see if using the unbinned image and a higher readout time parameter is viable for QVIO, and if not then at that point we will make the product design regarding the tradeoff for customers.

One more question regarding saving the camera intrinsics:

• If I am going to be playing around with the resolution that ends up being fed to QVIO, won't these intrinsics also change? Will I need to re-run kalibr for each resolution I want to try in order to get an intrinsic file for that specific resolution, or is there some way to adjust the intrinsics numbers manually to make it work if I know the res that the file was generated for and I know the new res. This is also probably a question AI knows the answer to

Thanks for pointing out that we should be doing a temperature vs. bias calibration on the IMU data. This is something that has been in the back of my mind for years but I never spent the time to really look into it and how to quantify the difference. Being able to quantify the accuracy of the qvio output on a dataset that is replayed with and without the temperature compensation would really help me make the argument for why we should start doing this as part of our calibration routines for our product.

@Alex-Kushleyev Hi Alex,

I am super excited by your idea of running parameter sweeps on top of the voxl-replay + offline misp + offline qvio infrastructure you have set up. Kudos to ModalAI for setting up such infrastructure. I myself am a huge believer in setting up the software infra for offline replay + parameter sweeps, it’s how we do all EKF2 development work at Cleo now and it’s paid large dividends once we invested in it.

Anyways, the thing(s) to nail down before going down the path of collecting datasets for offline sweeps is any settings / hardware setup that would have an impact on VIO performance but isn't something that I can change in the offline misp infrastructure. So my question before jumping into this is, what settings impact the raw10 bayer frames?

• Is it the preview mode width / height that set the resolution and readout time of the raw10 bayer frames? Should I stick with 1996x1520 for the data sets?
• Exposure settings would change the raw10 bayer frames right? So that wouldn’t be something I can tune with offline misp?
• Any other settings that would affect the raw10 bayer frames that I’m forgetting that we should nail down before I start collecting a collection of datasets?

I do have a couple other questions before I get started on the optimization part of this project, just to validate that it’s even feasible for our product:

• Will we still be able to get high quality 4K recordings from the IMX412 after optimizing it for VIO, or is there a tradeoff / infeasibility there?
• Will we still be able to stream a high quality image from the IMX412 to the GCS with Electronic Image Stabilization after optimizing the IMX412 for VIO? Or do you know of any incompatibility between the changes we’re talking about for VIO and EIS quality streaming?

I'm definitely excited for this fun project, especially since it might also open up the door for optimizing VIO performance on the other tracking cameras once I have the replay infrastructure set up.

Best,
Rowan

Hi @Alex-Kushleyev,

Resurrecting this old thread, we now have the IMX412 on a drone and we are now ready to give to VIO on the IMX412 our full attention and lots of testing effort. Where I'm at now is I have a prototype working and QVIO does run on the IMX412 camera and outputs estimates that seem reasonable, but I'm 100% sure it's not configured as good as it could be cause I made so many assumptions that I would like your input on:

Which camera data / pipe to use
Ideally, we would like the IMX412 VIO to perform close to (or of course better than!) a ar0144 tracking camera, in terms of the quality of the image for feature tracking, low CPU usage, low latency frames, etc etc etc. In this spirit, I've been looking into how to get the MISP normalized pipes coming from the IMX412 and also how to get the camera server producing ION data to get the same CPU usage gains we saw in https://forum.modalai.com/topic/4893/minimizing-voxl-camera-server-cpu-usage-in-sdk1-6.
I saw that in the pipe setup, the normalized code for IMX412 was commented out

After commenting it back in I was able to see in the portal a decent looking normalized stream. I also see the ION pipe pop up for that norm stream but I haven't tried that ION pipe on the QVIO server yet (I'm confident it would work though, just waiting for https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/commit/d18521776e3e88f396d85aa657769c47f29e9c9f to get tagged!).
Do you see any issue with using the MISP norm pipe for IMX412 VIO, or is that actually what you would recommend?

Which resolution to use
I know you talked about some resolution advice above, but I'm a little bit confused on the specifics on where to put those numbers. You had suggested 1996x1520 for a 5.5ms readout time. Do these numbers go into the Preview Width config fields? Here is the entire diff of the config settings I have been using for my testing:

The other values I have a question about in that diff is The MISP width / height fields, I chose 998x760 which is half of the Preview Width resolution you suggested. I did this because I wasn't sure of any compute bottle necks that would pop up if I fed a 1996x1520 image into QVIO. Do you think 998x760 is good or maybe I should pick a like 0.75 downsample so something like 1497x1140 for the MISP width/height.

Camera Driver files to use and how to version control and deploy those
Could you confirm that the binary files in https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250919/imx412_fpv_eis_20250919_drivers.zip are still the latest and the recommended binaries to use? Could you also advise on how to version control these files and deploy them to the voxl2 when the camera server .deb is deployed? I want to keep all files related to bringing up the camera in the voxl-camera-server debian if possible. I see some binaries files being stored in this path:

So if I understand the process correctly, those files will end up in /usr/share/modalai/chi-cdk/imx412-fps-misp. Is that where the voxl-configure-cameras C looks for them? Also, do I have to do anything with the com.qti.sensor.imx412_fpv.so file in the zip link that you sent, or do I just ignore that file?

My end desired behavior is that when I install the voxl camera server .deb, I don't have to worry about also copying binary files over to the voxl, or moving any files around on the voxl, or having to remember to run voxl-configure-cameras C. So maybe the path forward there is to have all files deployed into the right places by the .deb install and then in the postinst script auto run voxl-configure-cameras C? What do you think?

Aspect ratio concerns and their affect on field of view and camera calibration
Is the aspect ratio you suggested (1996x1520) the actual aspect ratio of the sensor? Or does the sensor support multiple aspect ratios, or is there something more complex going on I don't understand here? I just want to make sure that we're using as much FoV as the sensor supports. That should be the goal for VIO feature detection and for streaming right, maximize field of view?

Also, how does changing the aspect ratio / resolution affect the camera calibration? We're using kalibr which asks for a focal length bootstrap, which we've been giving it 470 for the ar0144 camera, do you know of a way that we can figure out an accurate focal length bootstrap for the images that we end up using for the IMX412?

How to not lose other IMX412 features like 4k recording and EIS streaming etc
This is kinda my biggest concern about the feasibility of this whole thing: Will we still be able to get the other awesome IMX412 features like high quality streaming to the GCS with EIS, as well as high quality 4K recordings even in difficult low light environments, AND at the same time optimize the IMX412 for VIO which demands stuff like fast readout for less skew?

Any advice you can give here on mapping out the tradeoffs? Are there any non-starters like not being able to get 4K recording if we opt to use the MISP norm pipe for VIO? Or are you confident that we can get the best of all 3 worlds

Exposure time concerns
I agree with your initial point that needing low exposure for low motion blur is important. As I mentioned in the intro to this message, I have a prototype working and I am now at a place where I can indeed tune the gain vs exposure / auto exposure params. Could you help me with that? I assume that this tradeoff also applies to the ar0144 camera, any lessons I can take from there?

QVIO readoutTime param
Great find and thanks for pointing that out!! To confirm the specific numbers here, if I use the 1996x1520 preview width/height which has a documented read out time of 5.5ms (I should confirm this using the -d mode), then I should put 0.0055 for that parameter?

As always, thank you for your help in camera related matters, we would be no where close to where we are now with robotic perception without your guidance!

Hi @Alex-Kushleyev ,

Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!

All the best,
Rowan

Hi @Alex-Kushleyev,

I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow voxl-qvio-server to use ION pipes (using DEN_ION_BUF and DEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.

In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.

We will be eagerly awaiting the release of a libmodal-pipe gitlab tag with these changes in it!

Cheers,
Rowan

Hi @Alex-Kushleyev,

I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.

Yes absolutely, I can get that done today.

Thanks for your patience!

Thank you for taking this on!

Rowan

@Alex-Kushleyev Hi Alex, Happy New Years

Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.

Thank you,
Rowan

@Moderator Hi Modal,

Does this mean you can create a custom data file partition that you can flash using fastboot?

Yes that is correct.

We have not explored this before but it has been asked a few times. This could be very helpful for other developers.

I'm certainly hopeful that it will be helpful here at Cleo Robotics! So far with my prototyping it works as expected and cuts flashing time of some large docker images we have here at Cleo down by a noticeable fraction (no file overhead via fastboot like with ADB).

I think the snippet I posted covers the baseline functionality of getting a custom "payload" into the data partition. However if there is more I can elaborate on in terms of the toolchain / what's in the payload, and if that elaboration will be helpful to other VOXL2 developers, I would be happy to elaborate Just let me know!

Other similar discussion points I tackled recently that I'm happy to talk about lessons of:

• Flashing the system image and VOXL/CLEO SDK through a Windows Machine (journeys in USB device drivers)
• Building Flutter applications for uniform flashing process across all operating systems
• Building release bundles (i.e. a collection of partition binaries) in CI

For anyone reading this in the future this works:

docker run --rm --privileged \
    -v "$SYSTEM_IMAGE_DIR":/system-image:ro \
    -v "$CLEO_IMAGE_DIR":/cleo-image \
    -v "$CLEO_PROVISION_DIR":/cleo-provision:ro \
    ubuntu:18.04 bash -c "
    set -e
    
    # Install tools
    apt-get update -qq
    apt-get install -y -qq android-tools-fsutils e2fsprogs >/dev/null
    
    # Extract stock data partition contents
    echo 'Extracting stock m0054-data-fs.ext4...'
    mkdir -p /tmp/data_root
    
    # Convert sparse to raw and mount
    simg2img /system-image/m0054-data-fs.ext4 /tmp/stock_raw.ext4
    mkdir -p /mnt/stock
    mount -o loop,ro /tmp/stock_raw.ext4 /mnt/stock
    
    # Copy all stock files
    cp -a /mnt/stock/* /tmp/data_root/
    umount /mnt/stock
    rm /tmp/stock_raw.ext4
    
    echo 'Stock contents copied.'
    
    # Add cleo-provision directory
    cp -a /cleo-provision /tmp/data_root/cleo-provision
    
    echo 'Added cleo-provision directory'
    
    # Create the ext4 image using same parameters as BitBake recipe:
    #   -s            : sparse output
    #   -l SIZE       : filesystem size in bytes
    #   -a /data      : android mount point
    #   -b 4096       : block size
    make_ext4fs -s -l $USERDATA_SIZE_EXT4 -a /data -b 4096 /cleo-image/m0054-data-fs.ext4 /tmp/data_root
    
    echo ''
    echo 'Created sparse ext4 image successfully'
"

You'll need to be on an arm64 architecture.

I found https://gitlab.com/voxl-public/system-image-build/meta-voxl2/-/blob/qrb5165-ubun1.0-14.1a/recipes-products/image/qti-ubuntu-robotics-image.bbappend#L148, which is in bitbake language (not familiar)?

Is

do_makeuserdata() {
    make_ext4fs -s -l ${USERDATA_SIZE_EXT4} ${IMAGE_EXT4_SELINUX_OPTIONS} \
        -a /data -b 4096 ${DEPLOY_DIR_IMAGE}/${OVERLAYIMAGE_TARGET} ${IMAGE_ROOTFS}/data
}

something I could do just on normal ubuntu in CI? I'm thinking to build a cleo-specific userdata in CI with all of our custom software.

What are ${IMAGE_EXT4_SELINUX_OPTIONS}, is there some secret sauce there?

Hi Modal,

I'm looking into adding some data in the m0054-data-fs.ext file system. I'm having some trouble getting the voxl2 to work correctly when I try to add the data and then create a new Android sparse image that I flash instead of m0054-data-fs.ext. Would it be possible to provide me with the tools used to create that m0054-data-fs.ext file system at ModalAI so that I can use the exact same tools to create the sparse android image after adding my files to it?

Thank you,
Rowan

@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.

@Alex-Kushleyev

sure, that would be helpful.

i will check with the team today to make sure we are all good, so perhaps you should wait for that.

Will do!

@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.