# Minimizing voxl-camera-server CPU usage in SDK1.6 Source: https://forum.modalai.com/topic/4893/minimizing-voxl-camera-server-cpu-usage-in-sdk1-6 Category: VOXL SDK and Software (https://forum.modalai.com/category/47/voxl-sdk-and-software) Tags: camera Posted: 2025-11-26 17:48:18 UTC by Rowan Dempster Replies: 32 · Views: 13474 ## Rowan Dempster · 2025-11-26 17:48:18 UTC Hi Modal, As we at Cleo update voxl-camera-server to SDK 1.6 (from SDK 1.0 with lots of backported changes), I've done some preliminary CPU profiling and have some questions on how to keep voxl-camera-server CPU usage down in SDK 1.6. Two things stand out to me: 1. Using the new `tracking_misp_norm` pipe for our tracking cameras uses ~25% more of a CPU core than using the `tracking` pipe. 25% is total change across our 3 tracking cameras. So the usage goes from ~85% of a core to ~110% of a core (exact numbers change based on core allocation). To illustrate this you can run `voxl-inspect-cam top bottom back hires_small_color tof_depth`, record the CPU usage of voxl-camera-server, and then run `voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm hires_small_color tof_depth` and record the CPU usage again and compare. Is there a way to prevent this from happening, or can you explain why the `tracking_misp_norm` pipes use more CPU? 2. Adding additional clients to the camera pipe topics causes voxl-camera-server to use more CPU. To illustrate this I only run voxl-camera-server (no other services), then inspect our baseline pipes (e.g. `voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm hires_small_color tof_depth`) and then in a new terminal(s) I open new clients again using `voxl-inspect-cam`. For example, when I open 4 new terminals (on top of the baseline terminal) and run `voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm` in each, then the voxl-camera-server CPU usage increases ~70% (from ~110% of a core, to ~180% of a core). This behavior on its own doesn't quite make sense to me, why would additional clients change the CPU load of the server, and by that much? The images should only be computed once, not per client? Furthermore, what's extra strange is that which tracking pipe is used also matters. If the baseline terminal is instead running `voxl-inspect-cam top bottom back hires_small_color tof_depth` and the 4 new client terminals are running `voxl-inspect-cam top bottom back`, then the CPU usage increase only by ~7% (from ~85% to ~92%). The different behavior between type of tracking pipe doesn't make sense to me, could you explain it? Overall, at Cleo we are looking to use the `tracking_misp_norm` pipes going forward, and be able to have multiple clients consuming those pipes with having to worry about increasing CPU usage of the voxl-camera-server. If you could comment on whether these asks are possible, or explain why not (either at this time, or never), that would help us a great deal in our process of taking advantage of Modal's great work in robotic perception! ## Reply by Alex Kushleyev (ModalAI staff) · 2025-11-26 21:30:08 UTC Hi @Rowan-Dempster, We have been looking at some optimizations to help reduce the overall cpu usage of the camera server (not yet in the SDK). Let me test your exact use case and see what can be done. Just FYI, we recently added support for sharing ION buffers directly from camera server, which means the camera clients get the images using zero-copy approach. this allows to save the cpu cycles wasted on sending the image bytes over the pipe, especially when there are multiple clients. If you would like to learn more how to use the ION buffers, I can post some examples soon. One the client side, the API for receiving and ION buffer vs regular buffer is almost the same. One thing that will be different is that the shared ION buffer has to be released by all clients before it can be re-used by the camera server (which makes sense). Even without the ION buffer sharing there is room to reduce the cpu usage, so I will follow up after testing this a bit. Regarding your question whether sending the image to multiple clients should not cause significant extra cpu usage -- yes you are correct, ideally it should not. However, the reason why it is happening here is related to how we set up ION buffer cache policy and currently when the CPU accesses the buffers for the `misp_norm` images (coming from the gpu), the cpu reads are not cached and the read access is expensive. Reading the same buffer multiple times results in repeated CPU-RAM access (for the data that would normally be already fully cached after the first send to the client pipe). However, in some other cases (when the buffer is not used by the cpu, but is shared as ION buffer and client sends the buffer directly to GPU), this approach results in even lower CPU usage. So i think we need to resolve the buffer cache policy based on the use case.. More details will come soon.. Alex ## Reply by Rowan Dempster · 2025-11-26 22:33:04 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex, I appreciate you looking into our use case for us (as always!). Please let me know if I can help by providing more details regarding the structure of our clients that are consuming the camera pipes. If there is a detail Cleo can't share publicly on the forum I can always email you as well or hop on a quick call to elaborate. Understood about the `misp` pipes, that expensive read access would explain both point #1 and the strange part of point #2 in my original post. We at Cleo will be monitoring this closely, since CPU usage regressions is pretty much the only gating item for us upgrading our robotic perception stack to SDK1.6. The `misp_norm` pipes are a great benefit to that perception stack and we'd of course love to take advantage of them as soon as possible. Thus we are definitely open to trying out zero-copy client approaches for keeping CPU usage down, or any other optimizations you could share examples of for us to try out on our use case. ## Reply by Rowan Dempster · 2025-12-04 20:24:10 UTC (in reply to Rowan Dempster) @Alex-Kushleyev Hi Alex, just following up on any update regarding the shared ION buffers or other methods to work around the CPU hit taken by having many clients to `misp` based image pipes. Happy to try out any methods/suggestions with our specific clients! Rowan ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-05 17:23:26 UTC (in reply to Rowan Dempster) Hi @Rowan-Dempster , I started a new branch where I will be working on some performance optimizations in the camera server. https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/perf-optimizations In my initial testing, setting cpu to perf and when running one or two instances of the following: ``` voxl-inspect-cam tracking_front_misp_norm tracking_down_misp_norm ``` i was seeing: ``` 1 instance (2 inspected streams) : 42% CPU (of one core) 2 instances (4 inspected streams): 58% CPU (of one core) ``` with the changes i just committed, i am seeing: ``` 1 instance: 31% cpu 2 instances : 36% cpu ``` If you would like you can test camera server from this branch and see if you can reproduce the results. notes: - the internal buffers were switched from uncached to cached and proper buffer management was added to ensure that data written by GPU is properly accessed by CPU - with these changes, if you use the `_encoded` stream from the tracking camera, it will work, but in dmesg you will see messages related to `qbuf cache ops failed` -- this is still under investigation and will be fixed soon. Meanwhile, I will work on an a simple example that shows the usage of ION buffers, I will try to share it a bit later today. Alex ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-05 21:59:48 UTC (in reply to Alex Kushleyev) Hi @Rowan-Dempster , Please take a look at this example (you can build and run it too) : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/add-new-image-tools/tools/voxl-image-repub.cpp This app can accept a regular image (RAW8 or YUV) and either re-publish it unchanged or crop and publish the result. Sometimes this is useful for quickly cropping an image that is fed into a different application that expects a smaller image or different aspect ratio. The app shows how to subscribe and handle ion buffer streams. Usage: ``` voxl2:/$ voxl-image-repub ERROR: Pipe name not specified Re-publish cropped camera frames (RAW8 or YUV) Options are: -x, --crop-offset-x crop offset in horizontal dimension -y, --crop-offset-y crop offset in vertical dimension -w, --crop-size-x crop size in horizontal dimension (width) -h, --crop-size-y crop size in vertical dimension (height) -o, --output-name output pipe name -u, --usage print this help message The cropped image will be centered if the crop offsets are not provided. typical usage: /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 --crop-offset-x 128 --crop-offset-y 128 ``` example re-publishing ion buffer image as regular image (which you can view in `voxl-portal` ) : ``` voxl-image-repub tracking_front_misp_norm_ion -o test ``` (you can see which ion pipes are available by running `voxl-list-pipes | grep _ion`) Please note that without the previous fix that i posted above, the client process that receives and uncached ION buffer will incur extra CPU load while accessing this buffer. For example, the same `voxl-image-repub` client uses **1.7% cpu** while republishing the normalized image (cached ion buffer), while using **7.3% cpu** republishing an image from an uncached ION buffer. (cpu usage % using one of the smaller cores). Please try and let me know if you have any questions. I know this cached / uncached buffering may be a bit confusing, but i will document this a bit more to help explain it a little better. Alex ## Reply by Rowan Dempster · 2025-12-08 16:11:11 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex, just acknowledging your messages, thanks for looking into this and providing us with a path forward. I will have time over the next 2-3 days to test the perf-optimizations branch and familiarize myself the ION based clients. I will update you then. ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-08 21:23:28 UTC (in reply to Rowan Dempster) @Rowan-Dempster , sounds good, let me know how the testing goes. Alex ## Reply by Rowan Dempster · 2025-12-08 21:37:19 UTC (in reply to Rowan Dempster) @Alex-Kushleyev Actually had some time today to do the profiling and explore the ION pipes. I see the same improvements that you quoted: **On 2.2.17** Normal pipes: 45% MISP pipes: 66% MISP pipes + 4x tracking pipes: 160% **On perf improvements** Normal pipes: 45% MISP pipes: 52% MISP pipes + 4x tracking pipes: 71% **On perf improvements + ION pipes** MISP pipes: 48% MISP pipes + 4x tracking pipes: 52% I started looking into using the ION pipe in the QVIO server code but saw this in the voxl-mpa-tools build script: ``` qrb5165) check_docker "4.3" mkdir -p build32 cd build32 cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} \ -DBUILD_TOOLS=OFF \ -DEN_ION_BUF=OFF ../ make -j$(nproc) cd ../ mkdir -p build64 cd build64 cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_64} \ -DBUILD_TOOLS=ON \ -DEN_ION_BUF=ON ../ make -j$(nproc) cd ../ ;; qrb5165-2) check_docker "4.3" mkdir -p build cd build cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_2_64} \ -DBUILD_TOOLS=ON \ -DEN_ION_BUF=OFF ../ make -j$(nproc) cd ../ ;; ``` Since we're using the qrb5165 (not the qrb5165-2) we have to use the 32bit toolchain to build QVIO (right?): ``` qrb5165) check_docker "4.4" mkdir -p build cd build cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} ../ make -j$(nproc) cd ../ ;; ``` Does this mean that on the qrb5165 (using the 32 bit toolchain), the QVIO server cannot be a client to the ION pipes? Thank you, Rowan ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-08 22:08:09 UTC (in reply to Rowan Dempster) @Rowan-Dempster , Yes, QVIO only runs as a 32-bit app due to the nature of the library from Qualcomm. I tried to build the `voxl-image-repub` application for 32 bit and got the following error: ``` /usr/bin/arm-linux-gnueabi-ld: CMakeFiles/voxl-image-repub.dir/voxl-image-repub.cpp.o: in function `main': voxl-image-repub.cpp:(.text.startup+0x228): undefined reference to `pipe_client_set_ion_buf_helper_cb' ``` So it seems like the 32-bit version of libmodal-pipe does not support sending ION buffers. I just checked with the team - even though we have not tested the ION buffer sharing in 32-bit environment, it should work. You could try to build `libmodal-pipe` library and enable ION support : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/build.sh?ref_type=heads#L76 . Then you would need to install that new library into your docker container where you are building your app, as well as deploy to VOXL2. BTW in order to build the tools in `voxl-mpa-tools` i needed to disable `-Werror` and comment out a few targets like `voxl-convert-image` and `voxl-inspect-cam-ascii` due to lack of 32-bit version of opencv. So.. if you really wanted the QVIO app to use the shared ION buffers, you would have to go that route.. Alex ## Reply by Rowan Dempster · 2025-12-09 15:42:48 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex I'm running into compilation errors when building libmodal-pipe after turning the EN_ION_BUF flag ON (builds fine with the flag OFF) ``` Found voxl-cross version: 4.4 -- --------------------------------------------------------- -- Using voxl-cross 32-bit toolchain for QRB5165 -- C Compiler : /usr/bin/arm-linux-gnueabi-gcc-7 -- C++ Compiler : /usr/bin/arm-linux-gnueabi-g++-7 -- Sysroot : /opt/sysroots/qrb5165_1 -- C flags : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4 -- CXX flags : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4 -- EXE Link Flags : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 -L/usr/lib -- SO Link Flags : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc_eh.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libssp_nonshared.a /opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi/lib/libc_nonshared.a -L/usr/lib -- The C compiler identification is GNU 7.3.0 -- The CXX compiler identification is GNU 7.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/bin/arm-linux-gnueabi-gcc-7 - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/bin/arm-linux-gnueabi-g++-7 - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- INFO: building with ion buf support -- INFO: building with mavlink support INFO: Skipping Building Python Bindings -- INFO: Skipping examples and tools -- Configuring done (1.8s) -- Generating done (0.0s) -- Build files have been written to: /home/root/build32 [ 10%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o [ 30%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o [ 30%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o [ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o [ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o [ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o [ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o [ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o [ 90%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o /home/root/library/src/server.c: In function 'pipe_server_write_ion_buffer': /home/root/library/src/server.c:1757:102: error: format '%ld' expects argument of type 'long int', but argument 5 has type 'int64_t {aka long long int}' [-Werror=format=] printf("server preparing to send ion buffer id %d (fd %d) to clients, n_clients: %d, time: %ld\n", ~~^ %lld ion_buf->buffer_id, fd, c[ch].n_clients, _time_monotonic_ns()); ~~~~~~~~~~~~~~~~~~~~ /home/root/library/src/client.c: In function '_stop_helper_and_remove_pipe': /home/root/library/src/client.c:1353:56: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=] if(en_debug) printf("ch: %d, shutdown socket %ld ns\n", ch, _time_monotonic_ns()); ~~^ ~~~~~~~~~~~~~~~~~~~~ %lld /home/root/library/src/client.c: In function 'pipe_client_close': /home/root/library/src/client.c:1475:43: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=] printf("ch: %d, shutdown socket %ld ms\n", ch, _time_monotonic_ns()); ~~^ ~~~~~~~~~~~~~~~~~~~~ %lld cc1: all warnings being treated as errors make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:93: library/CMakeFiles/modal_pipe.dir/src/client.c.o] Error 1 make[2]: *** Waiting for unfinished jobs.... cc1: all warnings being treated as errors make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:149: library/CMakeFiles/modal_pipe.dir/src/server.c.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2 make: *** [Makefile:136: all] Error 2 ``` Seems like I'm going down an uncharted path here. I'm okay with spending my time charting it out since getting QVIO to work with ION pipes will save us critical CPU util. Does ModalAI have plans to switch the official SDK QVIO server release (for qrb5165) to using ION pipes (I saw ModalAI switched to using the MISP pipes for QVIO a few months ago, taking a CPU hit)? ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-10 04:05:26 UTC (in reply to Rowan Dempster) @Rowan-Dempster, the errors above seem like print formatting issues, but i understand that there could be some others. If you want to try to resolve those and see if there are any more significant errors, I can help. We do not plan to update QVIO to use ion pipes, mainly because QVIO only supports a single camera input, so the savings from going to ION buffers for a single camera is going to be very small. The majority of savings came from the buffer caching policy, which I pointed out in the previous email. Another reason is that we are now focusing on Open Vins, which does support multiple cameras. We do plan to fix the buffer caching policy so to remove this extra cpu usage for the appropriate MISP outputs. The work is ongoing in the branch i mentioned above. Are you running multiple instances of QVIO, one for each camera? Alex ## Reply by Rowan Dempster · 2025-12-10 15:44:26 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Sounds good, I proceeded with resolving the basic typing issues related to 32 bit monotonic time. Next issue is a linker error that looks a bit more tricky: ``` -- Build files have been written to: /home/root/build32 [ 10%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o [ 50%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o [ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o [ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o [ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o [ 90%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o [100%] Linking CXX shared library libmodal_pipe.so /opt/sysroots/qrb5165_1/usr/lib/libgbm.so: file not recognized: file format not recognized collect2: error: ld returned 1 exit status make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:228: library/libmodal_pipe.so] Error 1 make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2 make: *** [Makefile:136: all] Error 2 ``` ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-10 19:56:33 UTC (in reply to Rowan Dempster) @Rowan-Dempster , yeah that is going to be a problem.. I don't think we have a 32 bit version of `libgbm.so` for VOXL2. This library is used for allocating the ION buffers. The next thing to try would be to remove the buffer allocation part from the 32-bit build. Actually receiving and reading the buffer does not involve libgbm. the client just gets a FD, which needs to be mmap'ed and used. (this would remove ability to allocate new ION buffers, which you actually don't need on the client side). I just commented out the following from the library CMakeLists: ``` #list(APPEND LIBS_TO_LINK gbm) #list(APPEND all_src_files src/buffers/gbm.cpp) ``` and here are the errors: ``` /usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_alloc_bufs': buffers.cpp:(.text+0x10c): undefined reference to `allocate_one_buffer(mpa_ion_buf_t*, int, int, unsigned int, unsigned int)' /usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x184): undefined reference to `init_buffer_allocator()' /usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_delete_bufs': buffers.cpp:(.text+0x24c): undefined reference to `delete_one_buffer(mpa_ion_buf_t*)' /usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x2a0): undefined reference to `shutdown_buffer_allocator()' ``` You could try replacing those functions with a fatal print statement "not implemented". Maybe that would work? Alex ## Reply by Rowan Dempster · 2025-12-11 19:33:43 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex, I compiled libmodal-pipe with ION client-only support in 32 bit builds. I then changed the voxl-mpa-tools toolchain to 32 bit and set the 32 bit toolchain to actually build the tool executables and removed the ones that didn't compile. After deploying those two modified packages to the VOXL2 and running the same repub command that I ran before, I am getting a runtime error while parsing the incoming ION buffers on the client side ``` voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146 ``` Tracking it down, it looks like the call to `pipe_client_bytes_in_pipe` in the `_ion_buf_cb` is what's printing the error. If I remove that line (so not returning if `pipe_client_bytes_in_pipe` returns `0`) it prints some garbage (incorrect) metadata and then segfaults: ``` voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion CH0 Got input ion frame size: 10330x-2843, format: STEREO_NV12, stride: 50834, size: 396 bytes Segmentation fault: Fault thread: voxl-image-repu(tid: 7900) Fault address: 0x4 Address not mapped. Segmentation fault ``` Note that I didn't recompile voxl-camera-server on-top of the modified libmodal-pipe, don't think that would change anything. Let me know if you want me to send you my diffs of libmodal-pipe (w.r.t. `master`) and voxl-mpa-tools (w.r.t. `add-new-image-tools`). P.S. I also tried `voxl-inspect-ion-stream top_misp_norm_ion` (that tool is also 32 bit now) and it's printing garbage too ``` | Pipe Name | fd | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps |t | top_misp_norm_ion | 6 | 455 | 3237 |-29847 | 52.43 |-24576 |1448040524 |1955547.7 | 30.0 | 0.6 ``` ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-11 19:45:55 UTC (in reply to Rowan Dempster) @Rowan-Dempster , I think you are close.. sure, if you want to share the diff or make a fork of the repos, I will try it out. Alex ## Reply by Rowan Dempster · 2025-12-11 19:58:03 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Here are the diffs: - libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs - voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs Thanks for your help! ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-12 15:04:44 UTC (in reply to Rowan Dempster) Hi @Rowan-Dempster, thank you. I will take a look today. Alex ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-12 16:43:12 UTC (in reply to Alex Kushleyev) Hi @Rowan-Dempster , I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server: https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-test The issue in `libmodal-pipe` was with a struct size difference for 32 / 64 bit systems. Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent. `voxl-image-repub` is working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version of `voxl-inspect-ion-stream` works as expected. Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross). Alex ## Reply by Rowan Dempster · 2025-12-15 17:18:38 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex, Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions). Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed: ``` ================================================================= thread is locked to cores: 4 5 6 7 connected to mavlink pipe Connected to cpu-monitor Starting Camera: back (id #0) gbm_create_device(156): Info: backend name is: msm_drm MISP Initializing for camera back Detected 1 platform(s) Detected 1 GPU device(s) Estimated imu dt = 0.000977s WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: back ERROR: Failed to start camera: back Starting Camera: top (id #1) MISP Initializing for camera top WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: top ERROR: Failed to start camera: top Starting Camera: tof (id #2) Starting Camera: hires (id #3) WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize encoder for camera: hires ERROR: Failed to start camera: hires Starting Camera: bottom (id #4) MISP Initializing for camera bottom WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: bottom ERROR: Failed to start camera: bottom ``` What are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process? Thank you, Rowan ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-15 17:24:57 UTC (in reply to Rowan Dempster) @Rowan-Dempster , nice! I am glad it worked. i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that? For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require `libgbm.so`.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed. Alex ## Reply by Rowan Dempster · 2025-12-15 17:31:54 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days. ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-15 17:40:40 UTC (in reply to Rowan Dempster) @Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers. i will check with the team today to make sure we are all good, so perhaps you should wait for that. Alex ## Reply by Rowan Dempster · 2025-12-15 17:41:38 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev >sure, that would be helpful. > i will check with the team today to make sure we are all good, so perhaps you should wait for that. Will do! ## Reply by Alex Kushleyev (ModalAI staff) · 2025-12-17 14:24:08 UTC (in reply to Rowan Dempster) @Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into `libmodal-pipe`. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here. Thanks again for your help! Alex ## Reply by Rowan Dempster · 2025-12-17 15:59:00 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago. ## Reply by Rowan Dempster · 2026-01-05 14:58:04 UTC (in reply to Rowan Dempster) @Alex-Kushleyev Hi Alex, Happy New Years :) Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo. Thank you, Rowan ## Reply by Alex Kushleyev (ModalAI staff) · 2026-01-08 15:00:14 UTC (in reply to Rowan Dempster) Hi @Rowan-Dempster, Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay! Happy New Year! Alex ## Reply by Alex Kushleyev (ModalAI staff) · 2026-01-21 21:58:57 UTC (in reply to Alex Kushleyev) Hi @Rowan-Dempster , I updated both projects to be compatible with 32 and 64 bit builds. https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-test Now you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support. In order to support this on the 32-bit client application side, you need to have the following defines: - `-DEN_ION_BUF` - `-DEN_32BIT_ION_BUF` as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22 Also, you can build just the 32-bit `voxl-mpa-tools` using `./build.sh arm32-only` I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev. Thanks for your patience! Alex ## Reply by Rowan Dempster · 2026-01-23 15:23:53 UTC (in reply to Alex Kushleyev) Hi @Alex-Kushleyev, > I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev. Yes absolutely, I can get that done today. > Thanks for your patience! Thank you for taking this on! Rowan ## Reply by Rowan Dempster · 2026-01-23 17:00:20 UTC (in reply to Rowan Dempster) Hi @Alex-Kushleyev, I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow `voxl-qvio-server` to use ION pipes (using `DEN_ION_BUF` and `DEN_32BIT_ION_BUF`). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more. In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work. We will be eagerly awaiting the release of a `libmodal-pipe` gitlab tag with these changes in it! Cheers, Rowan ## Reply by Rowan Dempster · 2026-02-04 15:44:46 UTC (in reply to Rowan Dempster) Hi @Alex-Kushleyev , Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP! All the best, Rowan ## Reply by Alex Kushleyev (ModalAI staff) · 2026-02-06 21:43:29 UTC (in reply to Rowan Dempster) Hi @Rowan-Dempster , We just merged the changes to dev : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/merge_requests/39. A new build of the `libmodal-pipe` library will be added after tonight's nightly build here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ . Any SDK releases after this will include this support. Alex