Minimizing voxl-camera-server CPU usage in SDK1.6

Alex Kushleyev

Hi @Rowan-Dempster, thank you. I will take a look today.

Alex

Alex Kushleyev

I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server:

https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-test

The issue in libmodal-pipe was with a struct size difference for 32 / 64 bit systems.

Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent.

voxl-image-repub is working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version of voxl-inspect-ion-stream works as expected.

Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross).

Alex

Rowan Dempster

@Alex-Kushleyev Hi Alex,

Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).

Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:

=================================================================
thread is locked to cores: 4 5 6 7
connected to mavlink pipe
Connected to cpu-monitor
Starting Camera: back (id #0)
gbm_create_device(156): Info: backend name is: msm_drm
MISP Initializing for camera back
 Detected 1 platform(s)
 Detected 1 GPU device(s)
Estimated imu dt = 0.000977s
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: back
ERROR:   Failed to start camera: back
Starting Camera: top (id #1)
MISP Initializing for camera top
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: top
ERROR:   Failed to start camera: top
Starting Camera: tof (id #2)
Starting Camera: hires (id #3)
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize encoder for camera: hires
ERROR:   Failed to start camera: hires
Starting Camera: bottom (id #4)
MISP Initializing for camera bottom
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize MISP encoder for camera: bottom
ERROR:   Failed to start camera: bottom

What are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?

Thank you,
Rowan

Alex Kushleyev

@Rowan-Dempster , nice! I am glad it worked.

i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?

For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.

Alex

Rowan Dempster

@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.

Alex Kushleyev

@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.

i will check with the team today to make sure we are all good, so perhaps you should wait for that.

Alex

Rowan Dempster

@Alex-Kushleyev

sure, that would be helpful.

i will check with the team today to make sure we are all good, so perhaps you should wait for that.

Will do!

Alex Kushleyev

@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.

Thanks again for your help!

Alex

Rowan Dempster

@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.

Rowan Dempster

@Alex-Kushleyev Hi Alex, Happy New Years

Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.

Thank you,
Rowan

Alex Kushleyev

Hi @Rowan-Dempster,

Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay!

Happy New Year!

Alex

Alex Kushleyev

Hi @Rowan-Dempster ,

I updated both projects to be compatible with 32 and 64 bit builds.

https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-test

Now you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support.

In order to support this on the 32-bit client application side, you need to have the following defines:

-DEN_ION_BUF
-DEN_32BIT_ION_BUF

as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22

Also, you can build just the 32-bit voxl-mpa-tools using ./build.sh arm32-only

I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.

Thanks for your patience!

Alex

Rowan Dempster

Hi @Alex-Kushleyev,

I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.

Yes absolutely, I can get that done today.

Thanks for your patience!

Thank you for taking this on!

Rowan

Rowan Dempster

Hi @Alex-Kushleyev,

I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow voxl-qvio-server to use ION pipes (using DEN_ION_BUF and DEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.

In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.

We will be eagerly awaiting the release of a libmodal-pipe gitlab tag with these changes in it!

Cheers,
Rowan

Rowan Dempster

Hi @Alex-Kushleyev ,

Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!

All the best,
Rowan

Alex Kushleyev

Hi @Rowan-Dempster ,

We just merged the changes to dev : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/merge_requests/39.

A new build of the libmodal-pipe library will be added after tonight's nightly build here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ .

Any SDK releases after this will include this support.

Alex

ModalAI Forum

Minimizing voxl-camera-server CPU usage in SDK1.6