Minimizing voxl-camera-server CPU usage in SDK1.6
-
@Alex-Kushleyev Hi Alex,
Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).
Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:
================================================================= thread is locked to cores: 4 5 6 7 connected to mavlink pipe Connected to cpu-monitor Starting Camera: back (id #0) gbm_create_device(156): Info: backend name is: msm_drm MISP Initializing for camera back Detected 1 platform(s) Detected 1 GPU device(s) Estimated imu dt = 0.000977s WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: back ERROR: Failed to start camera: back Starting Camera: top (id #1) MISP Initializing for camera top WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: top ERROR: Failed to start camera: top Starting Camera: tof (id #2) Starting Camera: hires (id #3) WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize encoder for camera: hires ERROR: Failed to start camera: hires Starting Camera: bottom (id #4) MISP Initializing for camera bottom WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: bottom ERROR: Failed to start camera: bottomWhat are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?
Thank you,
Rowan@Rowan-Dempster , nice! I am glad it worked.
i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?
For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require
libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.Alex
-
@Rowan-Dempster , nice! I am glad it worked.
i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?
For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require
libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.Alex
@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.
-
@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.
@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Alex
-
@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Alex
sure, that would be helpful.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Will do!
-
sure, that would be helpful.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Will do!
@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into
libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.Thanks again for your help!
Alex
-
@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into
libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.Thanks again for your help!
Alex
@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.
-
@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.
@Alex-Kushleyev Hi Alex, Happy New Years
Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.
Thank you,
Rowan -
@Alex-Kushleyev Hi Alex, Happy New Years
Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.
Thank you,
RowanHi @Rowan-Dempster,
Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay!
Happy New Year!
Alex
-
Hi @Rowan-Dempster,
Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay!
Happy New Year!
Alex
Hi @Rowan-Dempster ,
I updated both projects to be compatible with 32 and 64 bit builds.
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testNow you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support.
In order to support this on the 32-bit client application side, you need to have the following defines:
-DEN_ION_BUF-DEN_32BIT_ION_BUF
as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22
Also, you can build just the 32-bit
voxl-mpa-toolsusing./build.sh arm32-onlyI did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Thanks for your patience!
Alex
-
Hi @Rowan-Dempster ,
I updated both projects to be compatible with 32 and 64 bit builds.
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testNow you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support.
In order to support this on the 32-bit client application side, you need to have the following defines:
-DEN_ION_BUF-DEN_32BIT_ION_BUF
as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22
Also, you can build just the 32-bit
voxl-mpa-toolsusing./build.sh arm32-onlyI did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Thanks for your patience!
Alex
Hi @Alex-Kushleyev,
I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Yes absolutely, I can get that done today.
Thanks for your patience!
Thank you for taking this on!
Rowan
-
Hi @Alex-Kushleyev,
I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Yes absolutely, I can get that done today.
Thanks for your patience!
Thank you for taking this on!
Rowan
Hi @Alex-Kushleyev,
I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow
voxl-qvio-serverto use ION pipes (usingDEN_ION_BUFandDEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.
We will be eagerly awaiting the release of a
libmodal-pipegitlab tag with these changes in it!Cheers,
Rowan -
Hi @Alex-Kushleyev,
I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow
voxl-qvio-serverto use ION pipes (usingDEN_ION_BUFandDEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.
We will be eagerly awaiting the release of a
libmodal-pipegitlab tag with these changes in it!Cheers,
RowanHi @Alex-Kushleyev ,
Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!
All the best,
Rowan -
Hi @Alex-Kushleyev ,
Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!
All the best,
RowanHi @Rowan-Dempster ,
We just merged the changes to dev : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/merge_requests/39.
A new build of the
libmodal-pipelibrary will be added after tonight's nightly build here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ .Any SDK releases after this will include this support.
Alex
-
R Rowan Dempster referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login