Minimizing voxl-camera-server CPU usage in SDK1.6
-
@Alex-Kushleyev Hi Alex, I compiled libmodal-pipe with ION client-only support in 32 bit builds. I then changed the voxl-mpa-tools toolchain to 32 bit and set the 32 bit toolchain to actually build the tool executables and removed the ones that didn't compile. After deploying those two modified packages to the VOXL2 and running the same repub command that I ran before, I am getting a runtime error while parsing the incoming ION buffers on the client side
voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146 invalid metadata, magic number=85, expected 1229934146Tracking it down, it looks like the call to
pipe_client_bytes_in_pipein the_ion_buf_cbis what's printing the error. If I remove that line (so not returning ifpipe_client_bytes_in_pipereturns0) it prints some garbage (incorrect) metadata and then segfaults:voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test Subscribed to ION input pipe: top_misp_norm_ion CH0 Got input ion frame size: 10330x-2843, format: STEREO_NV12, stride: 50834, size: 396 bytes Segmentation fault: Fault thread: voxl-image-repu(tid: 7900) Fault address: 0x4 Address not mapped. Segmentation faultNote that I didn't recompile voxl-camera-server on-top of the modified libmodal-pipe, don't think that would change anything.
Let me know if you want me to send you my diffs of libmodal-pipe (w.r.t.
master) and voxl-mpa-tools (w.r.t.add-new-image-tools).P.S. I also tried
voxl-inspect-ion-stream top_misp_norm_ion(that tool is also 32 bit now) and it's printing garbage too| Pipe Name | fd | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps |t | top_misp_norm_ion | 6 | 455 | 3237 |-29847 | 52.43 |-24576 |1448040524 |1955547.7 | 30.0 | 0.6I think you are close.. sure, if you want to share the diff or make a fork of the repos, I will try it out.
Alex
-
I think you are close.. sure, if you want to share the diff or make a fork of the repos, I will try it out.
Alex
@Alex-Kushleyev Here are the diffs:
- libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs
- voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs
Thanks for your help!
-
@Alex-Kushleyev Here are the diffs:
- libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs
- voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs
Thanks for your help!
Hi @Rowan-Dempster, thank you. I will take a look today.
Alex
-
Hi @Rowan-Dempster, thank you. I will take a look today.
Alex
Hi @Rowan-Dempster ,
I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server:
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testThe issue in
libmodal-pipewas with a struct size difference for 32 / 64 bit systems.Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent.
voxl-image-repubis working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version ofvoxl-inspect-ion-streamworks as expected.Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross).
Alex
-
Hi @Rowan-Dempster ,
I made a few changes (based on your branches) to libmodal-pipe and the 32-bit tools and now it seems to be working without any run time errors from the test tools or 64bit camera server:
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testThe issue in
libmodal-pipewas with a struct size difference for 32 / 64 bit systems.Please note that when you build libmodal pipe, do not deploy the 64bit version to voxl2 as it will break things. Later we will need to fix this up so the struct padding is platform dependent.
voxl-image-repubis working now . I had to disable the FD caching due to a strange issue (segfault) when inserting into std::map. Did not investigate why that happened. Without duplicating the FD (using the original FD in the callback, everything works). Also the 32-bit version ofvoxl-inspect-ion-streamworks as expected.Feel free to test this out. If you don't find any more issues, we can try to mainline this change after cleanup. I am a bit concerned that std::map insert failed, but it is possible this is because i am using the latest voxl-cross to build the app, but my test board is running on old SDK (and there could have been a c++ runtime libs update sometime in the recent voxl-cross).
Alex
@Alex-Kushleyev Hi Alex,
Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).
Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:
================================================================= thread is locked to cores: 4 5 6 7 connected to mavlink pipe Connected to cpu-monitor Starting Camera: back (id #0) gbm_create_device(156): Info: backend name is: msm_drm MISP Initializing for camera back Detected 1 platform(s) Detected 1 GPU device(s) Estimated imu dt = 0.000977s WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: back ERROR: Failed to start camera: back Starting Camera: top (id #1) MISP Initializing for camera top WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: top ERROR: Failed to start camera: top Starting Camera: tof (id #2) Starting Camera: hires (id #3) WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize encoder for camera: hires ERROR: Failed to start camera: hires Starting Camera: bottom (id #4) MISP Initializing for camera bottom WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: bottom ERROR: Failed to start camera: bottomWhat are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?
Thank you,
Rowan -
@Alex-Kushleyev Hi Alex,
Success! I was able to run QVIO using the ION pipe and saw the resultant reduction in CPU util both on the camera server process as well as the QVIO process (small but noticeable reductions).
Note that I was not able to run the camera server with the 32-bit customized libmodal-pipe was installed. So the approach I took was run voxl-camera-server first with the unmodified libmodal-pipe, then install the 32-bit customized libmodal pipe, and run QVIO. Of course this would not be feasible in production. Here are the errors I'm getting when trying to run voxl-camera-server with the 32-bit customized libmodal-pipe installed:
================================================================= thread is locked to cores: 4 5 6 7 connected to mavlink pipe Connected to cpu-monitor Starting Camera: back (id #0) gbm_create_device(156): Info: backend name is: msm_drm MISP Initializing for camera back Detected 1 platform(s) Detected 1 GPU device(s) Estimated imu dt = 0.000977s WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: back ERROR: Failed to start camera: back Starting Camera: top (id #1) MISP Initializing for camera top WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: top ERROR: Failed to start camera: top Starting Camera: tof (id #2) Starting Camera: hires (id #3) WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize encoder for camera: hires ERROR: Failed to start camera: hires Starting Camera: bottom (id #4) MISP Initializing for camera bottom WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (1200000 bps provided). Using FPS hack. scale = 2.500000 WARNING: Encoder expecting(16) more buffers than module allocated(1) ERROR: OMX Set config failed! ERROR: Failed to initialize MISP encoder for camera: bottom ERROR: Failed to start camera: bottomWhat are the next steps towards mainlining this in a way that works "out of the box"? Anything else I should test or any way I can support that process?
Thank you,
Rowan@Rowan-Dempster , nice! I am glad it worked.
i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?
For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require
libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.Alex
-
@Rowan-Dempster , nice! I am glad it worked.
i think the issue here is that we have changed the struct definition (of the ion buffer type), which should not be changed for the 64-bit library. So until we hopefully mainline this, you can just manually copy the 32-bit libmodal-pipe.so into /usr/lib/ (while the original 64-bit version will remain in /usr/lib64). Did you try that?
For mainlining this, i think we will need to do a #ifdef 32 bit and insert the allignment dummy bytes as well as bypassing the ion allocation functions, which require
libgbm.so.. You could try that change as well, just need to figure out how to detect 64 vs 32 bit in the few source files that we changed.Alex
@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.
-
@Alex-Kushleyev Okay sounds like we're almost there in terms of readiness for mainline then. Would you like me to take that on and put up a gitlab MR from my fork into modalai's libmodal-pipe repo? I can schedule an hour or two to polish that up and get it ready sometime in the next few days.
@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Alex
-
@Rowan-Dempster , sure, that would be helpful. I don't see any issues with merging that in, since if we do it right, it will not affect 64 bit build at all and will only add limited support for 32 bit -- limited, meaning, for clients only, cannot allocate new buffers.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Alex
sure, that would be helpful.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Will do!
-
sure, that would be helpful.
i will check with the team today to make sure we are all good, so perhaps you should wait for that.
Will do!
@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into
libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.Thanks again for your help!
Alex
-
@Rowan-Dempster , we will go ahead and integrate the limited 32-bit ION buffer support into
libmodal-pipe. Did you make any changes to my latest branch from a few days ago? If so, I can take whatever changes you have and do the final integration. If you have not made any changes (and don't have any suggestions), I can just finish it up from here.Thanks again for your help!
Alex
@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.
-
@Alex-Kushleyev Sounds good! I did not make any changes since the revisions you shared 5 days ago.
@Alex-Kushleyev Hi Alex, Happy New Years
Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.
Thank you,
Rowan -
@Alex-Kushleyev Hi Alex, Happy New Years
Is there an ETA on when the 32-bit ION buffer support in libmodal-pipe will be main-lined? For our internal time planning here at Cleo.
Thank you,
RowanHi @Rowan-Dempster,
Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay!
Happy New Year!
Alex
-
Hi @Rowan-Dempster,
Thanks for the reminder. I will try to get this in by early next week. Sorry for the delay!
Happy New Year!
Alex
Hi @Rowan-Dempster ,
I updated both projects to be compatible with 32 and 64 bit builds.
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testNow you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support.
In order to support this on the 32-bit client application side, you need to have the following defines:
-DEN_ION_BUF-DEN_32BIT_ION_BUF
as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22
Also, you can build just the 32-bit
voxl-mpa-toolsusing./build.sh arm32-onlyI did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Thanks for your patience!
Alex
-
Hi @Rowan-Dempster ,
I updated both projects to be compatible with 32 and 64 bit builds.
https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/ion-32bit-testNow you should be able to install the libmodal pipe package without breaking 64 bit build and it will enable the 32-bit ion buffer client support.
In order to support this on the 32-bit client application side, you need to have the following defines:
-DEN_ION_BUF-DEN_32BIT_ION_BUF
as seen here : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/ion-32bit-test/CMakeLists.txt#L22
Also, you can build just the 32-bit
voxl-mpa-toolsusing./build.sh arm32-onlyI did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Thanks for your patience!
Alex
Hi @Alex-Kushleyev,
I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Yes absolutely, I can get that done today.
Thanks for your patience!
Thank you for taking this on!
Rowan
-
Hi @Alex-Kushleyev,
I did a quick test. Can you please test to make sure the updated libmodal pipe works with your 32 bit application and also does not break the main 64-bit functionality? Then i think we can merge it to dev.
Yes absolutely, I can get that done today.
Thanks for your patience!
Thank you for taking this on!
Rowan
Hi @Alex-Kushleyev,
I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow
voxl-qvio-serverto use ION pipes (usingDEN_ION_BUFandDEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.
We will be eagerly awaiting the release of a
libmodal-pipegitlab tag with these changes in it!Cheers,
Rowan -
Hi @Alex-Kushleyev,
I confirmed that the https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/ion-32bit-test branch does indeed allow
voxl-qvio-serverto use ION pipes (usingDEN_ION_BUFandDEN_32BIT_ION_BUF). I have not detected any other breakages of the Cleo SDK or performance degradations, but that's something we will monitor as we test more.In CPU perf mode, average core util dropped from 31% to 22%, this is a huge help, thank you very much for your work.
We will be eagerly awaiting the release of a
libmodal-pipegitlab tag with these changes in it!Cheers,
RowanHi @Alex-Kushleyev ,
Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!
All the best,
Rowan -
Hi @Alex-Kushleyev ,
Has there been a tag created in https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe that contains these changes? Please let me know when that happens, would would like to get these CPU gains moved into production ASAP!
All the best,
RowanHi @Rowan-Dempster ,
We just merged the changes to dev : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/merge_requests/39.
A new build of the
libmodal-pipelibrary will be added after tonight's nightly build here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ .Any SDK releases after this will include this support.
Alex
-
R Rowan Dempster referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login