Voxl2 Docker (Ubuntu 22) with OpenCL/Adreno

eric

Hello, we have a ROS2 humble autonomy stack that we we build in a CI pipeline and deploy to various platforms, including Voxl2.

The pipeline selects an appropriate base image for respective platforms (including nvida jetson, voxl, etc), each of which is a custom docker image based on ubuntu:jammy with unique layers for platform dependencies.

We would like to be able to utilize the Adreno GPU within our voxl2 target via OpenCL. I could use some assistance understanding how I might go about this for VOXL2. I am aware of this VOXL1 example: https://gitlab.com/voxl-public/voxl-docker-images/voxl-docker-opencl. Unfortunately, this example uses a prebuilt package for OpenCL to provide Adreno support.

Is there any documentation that could be shared regarding how this Adreno OpenCL library was built so that we can build an equivalent library in Ubuntu 22 that supports Adreno/VOXL2?

Any guidance would be greatly appreciated.

Thank you,
Eric

eric

For reference, I've tried downloading the adreno-opencl-sdk-2.0 from qualcomm, loaded the libraries into the docker, and run privileged while also adding devices /dev/dri and /dev/kgsl-3d0. I also added a vendor including the path to qualcomm's libOpenCL library. When I query platforms using the C++ api, the script just hangs. If I don't set the vendor, it fails with a -1001 error.

Inside the docker, I can build all the sdk examples just fine. I just can't access Adreno.

Thanks again for any help.

eric

@Eric-Katzfey I know it's been a few years, but I see your name all over the voxl-docker-opencl commit history was wondering if you'd be able to share your thoughts regarding how I might approach this? Specifically, if there are any steps you'd recommend I take to get OpenCL integrated into docker for Voxl2.

Thanks,
Eric

Eric Katzfey

@eric Yes, I did put that together based on some Qualcomm example code for VOXL. Not really sure how to do something similar on VOXL 2 but I'll ask around the office to see if anyone has some ideas on how to get that going.

eric

@Eric-Katzfey Really appreciate it! Thanks so much

Alex Kushleyev

@eric I will try it out, please give me a few days.

Alex

Alex Kushleyev

Quick update, I did some testing and tried searching documentation and could not get it to work. There are posts online asking Qualcomm whether this is possible, but there is no response there.

I know this would be a useful feature and I will try again next week. I want to see how this worked on VOXL1 and perhaps I am missing something.

In my test, I am also just doing a simple device query and it works on the host VOXL2 but not inside docker, tried mapping various .so libraries and devices to the docker container and no luck yet.

Alex

eric

@Alex-Kushleyev That aligns pretty well with my own experience so far. Thanks again for looking into this!

Alex Kushleyev

@eric I was able to get the GPU device query inside ubuntu 22.04 docker working using the following steps. It is possible that we can reduce the number of mapped devices and libraries to the docker container, but i am just going to give you this information right now so you can test. I will try to clean this up a bit later. I only tried the device query for now, but i figured i would let you know that there is progress..

#run docker
docker run -it --rm --privileged --device=/dev/kgsl-3d0 --device=/dev/ion -v /proc:/proc -v /firmware/image:/firmware/image -v /lib/firmware:/lib/firmware -v /sys/class:/sys/class -v /sys/bus:/sys/bus -v /sys/devices:/sys/devices -v /data:/data -v /usr/lib/liblog.so.0:/usr/lib/liblog.so.0 -v /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so -v /usr/lib/libcutils.so.0:/usr/lib/libcutils.so.0 -v /usr/lib/libllvm-qcom.so:/usr/lib/libllvm-qcom.so -v /usr/lib/libion.so.0.0.0:/usr/lib/libion.so.0.0.0 -v /usr/lib/libsync.so.0.0.0:/usr/lib/libsync.so.0.0.0 -v /usr/lib/libgsl.so:/usr/lib/libgsl.so -v /usr/lib/libCB.so:/usr/lib/libCB.so -v /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.5600.4:/usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.5600.4 -v `pwd`:/opt/code -w /opt/code arm64v8/ubuntu:22.04 bash

apt-get update
apt install --no-install-recommends -y pocl-opencl-icd

then run your test app to query the device..

eric

@Alex-Kushleyev

Alex Kushleyev

OK, a little more clean-up, it seems this is the minimal set of libraries /devices needed:

docker run -it --rm --privileged \
	-v /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so \
	-v /usr/lib/libCB.so:/usr/lib/libCB.so \
	-v /usr/lib/libgsl.so:/usr/lib/libgsl.so \
	-v /usr/lib/liblog.so.0:/usr/lib/liblog.so.0 \
	-v /usr/lib/libcutils.so.0:/usr/lib/libcutils.so.0 \
	-v /usr/lib/libsync.so.0.0.0:/usr/lib/libsync.so.0.0.0 \
	-v /usr/lib/libion.so.0.0.0:/usr/lib/libion.so.0.0.0 \
	-v /usr/lib/libllvm-qcom.so:/usr/lib/libllvm-qcom.so \
	-v /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.5600.4:/usr/lib/aarch64-linux-gnu/libglib-2.0.so.0.5600.4 \
	-v `pwd`:/opt/code -w /opt/code \
	arm64v8/ubuntu:22.04 bash

(--privileged mode maps all the needed devices to the docker container)

Then install some more packages (not sure if this can be reduced, not clear exactly what is missing):

apt-get update
apt install --no-install-recommends -y pocl-opencl-icd

Maybe we can figure out what lib is still missing so that pocl-opencl-icd does not have to be installed.. At least the issue was the a missing library, not a mapped device

For testing, I used a device query script from here:

root@733a6d4d5fdb:/opt/code# ./simple_query 
1. Device: QUALCOMM Adreno(TM)
 1.1 Hardware version: OpenCL 2.0 Adreno(TM) 650
 1.2 Software version: OpenCL 2.0 QUALCOMM build: commit # changeid # Date: 11/10/21 Wed Local Branch:  Remote Branch:  Compiler E031.37.12.01
 1.3 OpenCL C version: OpenCL C 2.0 Adreno(TM) 650
 1.4 Parallel compute units: 3

I also verified that a simple matrix multiplication app also worked (not provided here)

@eric , can you please let me know if this works for you?

Alex

eric

@Alex-Kushleyev

OMG IT WORKS!!

I was able to extract all these libraries from the host and directly install them inside the docker, and now the pcol-opencl-icd installation isn't needed.

This is really important for us, since it allows us to build external dependencies that rely on OpenCL in our pipeline directly without bind mounts (outside the host environment).

Really, really appreciate all your help!

FROM arm64v8/ubuntu:22.04

# Install necessary dependencies
RUN apt-get update && \
    apt-get install -y \
    cmake \
    build-essential \
    libglib2.0-0

# Copy Adreno GPU dependencies
# - libcutils0_0-r1_arm64.deb
# - libsync_1.0-r1_arm64.deb
# - qti-libion_0-r1_arm64.deb
# - liblog0_1.0-r1_arm64.deb
# - qti-adreno_1.0-r0_arm64.deb
COPY dep /root/dep

# Create required directory for qti-adreno install
RUN mkdir /usr/include/KHR && dpkg -i /root/dep/*.deb 

# Copy and build test script
COPY ./hellocl /root/hellocl
RUN cd /root/hellocl && mkdir build && cd build && cmake .. && make

CMD ["bash"]

voxl2:~/opencl$ docker run -it --rm --privileged opencl:latest ./root/hellocl/build/hellocl
Platform Information:
Platform Name: QUALCOMM Snapdragon(TM)
Platform Vendor: QUALCOMM
Platform Version: OpenCL 2.0 QUALCOMM build: commit # changeid # Date: 11/10/21 Wed Local Branch:  Remote Branch: 
Platform Profile: FULL_PROFILE
Platform Extensions:  
------------------------------------
Device Information:
Device Name: QUALCOMM Adreno(TM)
Device Vendor: QUALCOMM
Driver Version: OpenCL 2.0 QUALCOMM build: commit # changeid # Date: 11/10/21 Wed Local Branch:  Remote Branch:  Compiler E031.37.12.01
Device Version: OpenCL 2.0 Adreno(TM) 650
Device OpenCL C Version: OpenCL C 2.0 Adreno(TM) 650
Device Max Compute Units: 3
This should be three: 3

Alex Kushleyev

hi @eric ,

Nice! very clean.

Did you use dpkg-repack to create debs of installed packages, such as:

apt-get install dpkg-repack
dpkg-repack qti-adreno

Cool trick!

I will test this out and add to our docs.

Alex

eric

@Alex-Kushleyev Yes, dpkg -S <file path> to figure out which debs installed which libraries (ie, dpkg -S /usr/lib/libOpenCL.so), apt-cache show to see the source (ubuntu ppa vs modalai), then dpkg-repack to repack the modalai debs.

Thanks again!

Alex Kushleyev

@eric , thanks again for your input on this, i have posted a complete tutorial how to enable OpenCL in Docker on VOXL2 : https://docs.modalai.com/voxl-2-opencl-in-docker/

Alex

eric

@Alex-Kushleyev Awesome! Thanks again for all your help with this!