Minimizing voxl-camera-server CPU usage in SDK1.6

[Alex Kushleyev]

@Rowan-Dempster , sounds good, let me know how the testing goes.

Alex

[Rowan Dempster]

@Alex-Kushleyev Actually had some time today to do the profiling and explore the ION pipes.

I see the same improvements that you quoted:

On 2.2.17
Normal pipes: 45%
MISP pipes: 66%
MISP pipes + 4x tracking pipes: 160%

On perf improvements
Normal pipes: 45%
MISP pipes: 52%
MISP pipes + 4x tracking pipes: 71%

On perf improvements + ION pipes
MISP pipes: 48%
MISP pipes + 4x tracking pipes: 52%

I started looking into using the ION pipe in the QVIO server code but saw this in the voxl-mpa-tools build script:

	qrb5165)
		check_docker "4.3"
		mkdir -p build32
		cd build32
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} \
				-DBUILD_TOOLS=OFF \
				-DEN_ION_BUF=OFF ../
		make -j$(nproc)
		cd ../
		mkdir -p build64
		cd build64
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_64} \
				-DBUILD_TOOLS=ON \
				-DEN_ION_BUF=ON ../
		make -j$(nproc)
		cd ../
		;;

	qrb5165-2)
		check_docker "4.3"
		mkdir -p build
		cd build
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_2_64} \
				-DBUILD_TOOLS=ON \
				-DEN_ION_BUF=OFF ../
		make -j$(nproc)
		cd ../
		;;

Since we're using the qrb5165 (not the qrb5165-2) we have to use the 32bit toolchain to build QVIO (right?):

	qrb5165)
		check_docker "4.4"
		mkdir -p build
		cd build
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} ../
		make -j$(nproc)
		cd ../
		;;

Does this mean that on the qrb5165 (using the 32 bit toolchain), the QVIO server cannot be a client to the ION pipes?

Thank you,
Rowan

[Alex Kushleyev]

@Rowan-Dempster ,

Yes, QVIO only runs as a 32-bit app due to the nature of the library from Qualcomm.

I tried to build the voxl-image-repub application for 32 bit and got the following error:

/usr/bin/arm-linux-gnueabi-ld: CMakeFiles/voxl-image-repub.dir/voxl-image-repub.cpp.o: in function `main':
voxl-image-repub.cpp:(.text.startup+0x228): undefined reference to `pipe_client_set_ion_buf_helper_cb'

So it seems like the 32-bit version of libmodal-pipe does not support sending ION buffers.

I just checked with the team - even though we have not tested the ION buffer sharing in 32-bit environment, it should work. You could try to build libmodal-pipe library and enable ION support : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/build.sh?ref_type=heads#L76 .

Then you would need to install that new library into your docker container where you are building your app, as well as deploy to VOXL2.

BTW in order to build the tools in voxl-mpa-tools i needed to disable -Werror and comment out a few targets like voxl-convert-image and voxl-inspect-cam-ascii due to lack of 32-bit version of opencv.

So.. if you really wanted the QVIO app to use the shared ION buffers, you would have to go that route..

Alex

[Rowan Dempster]

@Alex-Kushleyev Hi Alex I'm running into compilation errors when building libmodal-pipe after turning the EN_ION_BUF flag ON (builds fine with the flag OFF)

Found voxl-cross version: 4.4
-- ---------------------------------------------------------
-- Using voxl-cross 32-bit toolchain for QRB5165
-- C Compiler     : /usr/bin/arm-linux-gnueabi-gcc-7
-- C++ Compiler   : /usr/bin/arm-linux-gnueabi-g++-7
-- Sysroot        : /opt/sysroots/qrb5165_1
-- C flags        : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4
-- CXX flags      : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4
-- EXE Link Flags : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 -L/usr/lib
-- SO Link Flags  : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc_eh.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libssp_nonshared.a /opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi/lib/libc_nonshared.a -L/usr/lib
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/arm-linux-gnueabi-gcc-7 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/arm-linux-gnueabi-g++-7 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- INFO: building with ion buf support
-- INFO: building with mavlink support
INFO: Skipping Building Python Bindings
-- INFO: Skipping examples and tools
-- Configuring done (1.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/root/build32
[ 10%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o
[ 30%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o
[ 30%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o
[ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o
[ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o
[ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o
[ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o
[ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o
[ 90%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o
/home/root/library/src/server.c: In function 'pipe_server_write_ion_buffer':
/home/root/library/src/server.c:1757:102: error: format '%ld' expects argument of type 'long int', but argument 5 has type 'int64_t {aka long long int}' [-Werror=format=]
         printf("server preparing to send ion buffer id %d (fd %d) to clients, n_clients: %d, time: %ld\n",
                                                                                                    ~~^
                                                                                                    %lld
                ion_buf->buffer_id, fd, c[ch].n_clients, _time_monotonic_ns());
                                                         ~~~~~~~~~~~~~~~~~~~~                          
/home/root/library/src/client.c: In function '_stop_helper_and_remove_pipe':
/home/root/library/src/client.c:1353:56: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         if(en_debug) printf("ch: %d, shutdown socket %ld ns\n", ch, _time_monotonic_ns());
                                                      ~~^            ~~~~~~~~~~~~~~~~~~~~
                                                      %lld
/home/root/library/src/client.c: In function 'pipe_client_close':
/home/root/library/src/client.c:1475:43: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         printf("ch: %d, shutdown socket %ld ms\n", ch, _time_monotonic_ns());
                                         ~~^            ~~~~~~~~~~~~~~~~~~~~
                                         %lld
cc1: all warnings being treated as errors
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:93: library/CMakeFiles/modal_pipe.dir/src/client.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cc1: all warnings being treated as errors
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:149: library/CMakeFiles/modal_pipe.dir/src/server.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Seems like I'm going down an uncharted path here. I'm okay with spending my time charting it out since getting QVIO to work with ION pipes will save us critical CPU util. Does ModalAI have plans to switch the official SDK QVIO server release (for qrb5165) to using ION pipes (I saw ModalAI switched to using the MISP pipes for QVIO a few months ago, taking a CPU hit)?

[Alex Kushleyev]

@Rowan-Dempster, the errors above seem like print formatting issues, but i understand that there could be some others. If you want to try to resolve those and see if there are any more significant errors, I can help.

We do not plan to update QVIO to use ion pipes, mainly because QVIO only supports a single camera input, so the savings from going to ION buffers for a single camera is going to be very small. The majority of savings came from the buffer caching policy, which I pointed out in the previous email. Another reason is that we are now focusing on Open Vins, which does support multiple cameras.

We do plan to fix the buffer caching policy so to remove this extra cpu usage for the appropriate MISP outputs. The work is ongoing in the branch i mentioned above.

Are you running multiple instances of QVIO, one for each camera?

Alex

[Rowan Dempster]

@Alex-Kushleyev Sounds good, I proceeded with resolving the basic typing issues related to 32 bit monotonic time. Next issue is a linker error that looks a bit more tricky:

-- Build files have been written to: /home/root/build32
[ 10%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o
[ 50%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o
[ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o
[ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o
[ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o
[ 90%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o
[100%] Linking CXX shared library libmodal_pipe.so
/opt/sysroots/qrb5165_1/usr/lib/libgbm.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:228: library/libmodal_pipe.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

[Alex Kushleyev]

@Rowan-Dempster , yeah that is going to be a problem.. I don't think we have a 32 bit version of libgbm.so for VOXL2. This library is used for allocating the ION buffers.

The next thing to try would be to remove the buffer allocation part from the 32-bit build. Actually receiving and reading the buffer does not involve libgbm. the client just gets a FD, which needs to be mmap'ed and used. (this would remove ability to allocate new ION buffers, which you actually don't need on the client side).

I just commented out the following from the library CMakeLists:

#list(APPEND LIBS_TO_LINK gbm)
#list(APPEND all_src_files src/buffers/gbm.cpp)

and here are the errors:

/usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_alloc_bufs':
buffers.cpp:(.text+0x10c): undefined reference to `allocate_one_buffer(mpa_ion_buf_t*, int, int, unsigned int, unsigned int)'
/usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x184): undefined reference to `init_buffer_allocator()'
/usr/bin/aarch64-linux-gnu-ld: CMakeFiles/modal_pipe.dir/src/buffers.cpp.o: in function `mpa_ion_buf_pool_delete_bufs':
buffers.cpp:(.text+0x24c): undefined reference to `delete_one_buffer(mpa_ion_buf_t*)'
/usr/bin/aarch64-linux-gnu-ld: buffers.cpp:(.text+0x2a0): undefined reference to `shutdown_buffer_allocator()'

You could try replacing those functions with a fatal print statement "not implemented". Maybe that would work?

Alex

[Rowan Dempster]

@Alex-Kushleyev Hi Alex, I compiled libmodal-pipe with ION client-only support in 32 bit builds. I then changed the voxl-mpa-tools toolchain to 32 bit and set the 32 bit toolchain to actually build the tool executables and removed the ones that didn't compile. After deploying those two modified packages to the VOXL2 and running the same repub command that I ran before, I am getting a runtime error while parsing the incoming ION buffers on the client side

voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test                                                                                                                                                 
Subscribed to ION input pipe: top_misp_norm_ion                                                                                                                                                                       
invalid metadata, magic number=85, expected 1229934146                                                                                                                                                                
invalid metadata, magic number=85, expected 1229934146                                                                                                                                                                
invalid metadata, magic number=85, expected 1229934146

Tracking it down, it looks like the call to pipe_client_bytes_in_pipe in the _ion_buf_cb is what's printing the error. If I remove that line (so not returning if pipe_client_bytes_in_pipe returns 0) it prints some garbage (incorrect) metadata and then segfaults:

voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test
Subscribed to ION input pipe: top_misp_norm_ion
CH0 Got input ion frame size: 10330x-2843, format: STEREO_NV12, stride: 50834, size: 396 bytes

Segmentation fault:
Fault thread: voxl-image-repu(tid: 7900)
Fault address: 0x4
Address not mapped.
Segmentation fault

Note that I didn't recompile voxl-camera-server on-top of the modified libmodal-pipe, don't think that would change anything.

Let me know if you want me to send you my diffs of libmodal-pipe (w.r.t. master) and voxl-mpa-tools (w.r.t. add-new-image-tools).

P.S. I also tried voxl-inspect-ion-stream top_misp_norm_ion (that tool is also 32 bit now) and it's printing garbage too

|         Pipe Name |  fd  |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  |t
| top_misp_norm_ion |    6 |     455 | 3237 |-29847 | 52.43 |-24576 |1448040524 |1955547.7  | 30.0 |    0.6

[Alex Kushleyev]

@Rowan-Dempster ,

I think you are close.. sure, if you want to share the diff or make a fork of the repos, I will try it out.

Alex

[Rowan Dempster]

@Alex-Kushleyev Here are the diffs:

• libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs
• voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs

Thanks for your help!

[Alex Kushleyev]

Hi @Rowan-Dempster, thank you. I will take a look today.

Alex

[Alex Kushleyev]

Hi @Rowan-Dempster ,

I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server:

https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-test

The issue in libmodal-pipe was with a struct size difference for 32 / 64 bit systems.

Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent.

voxl-image-repub is working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version of voxl-inspect-ion-stream works as expected.

Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross).

Alex

[Rowan Dempster]

@Alex-Kushleyev Hi Alex,

Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).

Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:

=================================================================
thread is locked to cores: 4 5 6 7
connected to mavlink pipe
Connected to cpu-monitor
Starting Camera: back (id #0)
gbm_create_device(156): Info: backend name is: msm_drm
MISP Initializing for camera back
 Detected 1 platform(s)
 Detected 1 GPU device(s)
Estimated imu dt = 0.000977s
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: back
ERROR:   Failed to start camera: back
Starting Camera: top (id #1)
MISP Initializing for camera top
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: top
ERROR:   Failed to start camera: top
Starting Camera: tof (id #2)
Starting Camera: hires (id #3)
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize encoder for camera: hires
ERROR:   Failed to start camera: hires
Starting Camera: bottom (id #4)
MISP Initializing for camera bottom
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: bottom
ERROR:   Failed to start camera: bottom

What are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?

Thank you,
Rowan

[Alex Kushleyev]

@Rowan-Dempster , nice! I am glad it worked.

i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?

For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.

Alex

[Rowan Dempster]

@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.

[Alex Kushleyev]

@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.

i will check with the team today to make sure we are all good, so perhaps you should wait for that.

Alex

[Rowan Dempster]

@Alex-Kushleyev

sure, that would be helpful.

i will check with the team today to make sure we are all good, so perhaps you should wait for that.

Will do!

[Alex Kushleyev]

@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.

Thanks again for your help!

Alex

[Rowan Dempster]

@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.

[Rowan Dempster]

@Alex-Kushleyev Hi Alex, Happy New Years

Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.

Thank you,
Rowan