Minimizing voxl-camera-server CPU usage in SDK1.6
-
@Rowan-Dempster, the errors above seem like print formatting issues, but i understand that there could be some others. If you want to try to resolve those and see if there are any more significant errors, I can help.
We do not plan to update QVIO to use ion pipes, mainly because QVIO only supports a single camera input, so the savings from going to ION buffers for a single camera is going to be very small. The majority of savings came from the buffer caching policy, which I pointed out in the previous email. Another reason is that we are now focusing on Open Vins, which does support multiple cameras.
We do plan to fix the buffer caching policy so to remove this extra cpu usage for the appropriate MISP outputs. The work is ongoing in the branch i mentioned above.
Are you running multiple instances of QVIO, one for each camera?
Alex
-
@Alex-Kushleyev Sounds good, I proceeded with resolving the basic typing issues related to 32 bit monotonic time. Next issue is a linker error that looks a bit more tricky:
-- Build files have been written to: /home/root/build32 [ 10%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o [ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o [ 50%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o [ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o [ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o [ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o [ 90%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o [100%] Linking CXX shared library libmodal_pipe.so /opt/sysroots/qrb5165_1/usr/lib/libgbm.so: file not recognized: file format not recognized collect2: error: ld returned 1 exit status make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:228: library/libmodal_pipe.so] Error 1 make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2 make: *** [Makefile:136: all] Error 2 -
@Rowan-Dempster , yeah that is going to be a problem.. I don't think we have a 32 bit version of
libgbm.sofor VOXL2. This library is used for allocating the ION buffers.The next thing to try would be to remove the buffer allocation part from the 32-bit build. Actually receiving and reading the buffer does not involve libgbm. the client just gets a FD, which needs to be mmap'ed and used. (this would remove ability to allocate new ION buffers, which you actually don't need on the client side).
I just commented out the following from the library CMakeLists:
#list(APPEND LIBS_TO_LINK gbm) #list(APPEND all_src_files src/buffers/gbm.cpp)and here are the errors:
/usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_alloc_bufs': buffers.cpp:(.text+0x10c): undefined reference to `allocate_one_buffer(mpa_ion_buf_t*, int, int, unsigned int, unsigned int)' /usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x184): undefined reference to `init_buffer_allocator()' /usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_delete_bufs': buffers.cpp:(.text+0x24c): undefined reference to `delete_one_buffer(mpa_ion_buf_t*)' /usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x2a0): undefined reference to `shutdown_buffer_allocator()'You could try replacing those functions with a fatal print statement "not implemented". Maybe that would work?
Alex
-
@Alex-Kushleyev Hi Alex, I compiled libmodal-pipe with ION client-only support in 32 bit builds. I then changed the voxl-mpa-tools toolchain to 32 bit and set the 32 bit toolchain to actually build the tool executables and removed the ones that didn't compile. After deploying those two modified packages to the VOXL2 and running the same repub command that I ran before, I am getting a runtime error while parsing the incoming ION buffers on the client side
voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146Tracking it down, it looks like the call to
pipe_client_bytes_in_pipein the_ion_buf_cbis what's printing the error. If I remove that line (so not returning ifpipe_client_bytes_in_pipereturns0) it prints some garbage (incorrect) metadata and then segfaults:voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion CH0 Got input ion frame size: 10330x-2843, format: STEREO_NV12, stride: 50834, size: 396 bytes Segmentation fault: Fault thread: voxl-image-repu(tid: 7900) Fault address: 0x4 Address not mapped. Segmentation faultNote that I didn't recompile voxl-camera-server on-top of the modified libmodal-pipe, don't think that would change anything.
Let me know if you want me to send you my diffs of libmodal-pipe (w.r.t.
master) and voxl-mpa-tools (w.r.t.add-new-image-tools).P.S. I also tried
voxl-inspect-ion-stream top_misp_norm_ion(that tool is also 32 bit now) and it's printing garbage too| Pipe Name | fd | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps |t | top_misp_norm_ion | 6 | 455 | 3237 |-29847 | 52.43 |-24576 |1448040524 |1955547.7 | 30.0 | 0.6 -
I think you are close.. sure, if you want to share the diff or make a fork of the repos, I will try it out.
Alex
-
@Alex-Kushleyev Here are the diffs:
- libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs
- voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs
Thanks for your help!
-
Hi @Rowan-Dempster, thank you. I will take a look today.
Alex
-
Hi @Rowan-Dempster ,
I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server:
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testThe issue in
libmodal-pipewas with a struct size difference for 32 / 64 bit systems.Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent.
voxl-image-repubis working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version ofvoxl-inspect-ion-streamworks as expected.Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross).
Alex
-
@Alex-Kushleyev Hi Alex,
Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).
Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:
================================================================= thread is locked to cores: 4 5 6 7 connected to mavlink pipe Connected to cpu-monitor Starting Camera: back (id #0) gbm_create_device(156): Info: backend name is: msm_drm MISP Initializing for camera back Detected 1 platform(s) Detected 1 GPU device(s) Estimated imu dt = 0.000977s WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: back ERROR: Failed to start camera: back Starting Camera: top (id #1) MISP Initializing for camera top WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: top ERROR: Failed to start camera: top Starting Camera: tof (id #2) Starting Camera: hires (id #3) WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize encoder for camera: hires ERROR: Failed to start camera: hires Starting Camera: bottom (id #4) MISP Initializing for camera bottom WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: bottom ERROR: Failed to start camera: bottomWhat are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?
Thank you,
Rowan -
@Rowan-Dempster , nice! I am glad it worked.
i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?
For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require
libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.Alex
-
@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.
-
@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Alex
-
sure, that would be helpful.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Will do!
