Using realsense camera for tflite-server inference

[Riccardo Franceschini]

Hi need to use a realsense camera (L515 in particular but I tried also with a d455 and is the same) as a uvc device to be used with the voxl-tflite-server, on voxl2 with sdk 1.1.2 but also tried with a 0.9.5 and the behaviour was the same.

The camera is connected and detected as shown by

lsusb
Bus 002 Device 004: ID 8086:0b64 Intel Corp. 
Bus 002 Device 003: ID 2c7c:0800  
Bus 002 Device 002: ID 0424:5744 Standard Microsystems Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 0424:2740 Standard Microsystems Corp. 
Bus 001 Device 004: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
Bus 001 Device 002: ID 0424:2744 Standard Microsystems Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

with video streams from 2 to 9

voxl2:~$ ls /dev/video
video0   video2   video32  video4   video6   video8   
video1   video3   video33  video5   video7   video9```

When I try to run

voxl-uvc-server -v 8086 -p 0b64 -d

specifying the vendor and the product, the camera is detected but when it tries to open the camera the server crash (I have also changed image dimension but nothing changed)

loading config file
Enabling debug messages
=================================================================
width:                            640
height:                           480
fps:                              30
pipe_name:                        uvc
=================================================================
voxl-uvc-server starting
Image resolution 640x480, 30 fps chosen
Vendor ID 0x8086 chosen
Product ID 0x0b64 chosen
UVC initialized
Device 8086:0b64 found
uvc_open failed
UVC exited
voxl-uvc-server ending

I have tried to run the uvc-server with a standar logitech webcam and it works fine.

I tried also to simulate the behaviour of a webcam using v4l2loopback as suggested here but the kernel module is not supported and I am not able to install the package, I have then tried also the start the compilation of the kernel to understand if it was feasible to add it manually but also this process crashed at the sync phase within the container tutorial :

user@54bf0bc04ffb:~$ ./qrb5165-sync.sh 
[INFO] - Syncing CLO repo...
  File "/usr/bin/repo", line 51
    def print(self, *args, **kwargs):
            ^
SyntaxError: invalid syntax

Also the camera works properly when connected the voxl2 and with the realsense packages installed inside a seperate container.

The question is, how can I achieve to connect the camera to the tflite server? any suggestion or different solution is welcome

[Moderator]

@Riccardo-Franceschini Hi Riccardo, we do not have any support for Realsense so you will need to write your own MPA server to connect to tflite-server. You could start with forking the uvc-server or trying to fix the crash you are currently encountering. we are happy to incorporate merge requests

[K.Stute]

Hi @Riccardo-Franceschini ,
sorry to reactivate this old topic but I'm stuck trying to get a real-sense D455 to work over the USB-port of the voxl2. Since apparently the real-sense products are not support by modalai I'd like to know how you got the camera connected and working.
Thanks in advance!

[Riccardo Franceschini]

Hi @K-Stute, sorry for the late reply .

Anyway, I connected the camera over the usb connection on the 5G modem, you need to be sure to have the proper cable (the standard one of the realsense is fine) and that the connection is over a usb 3.0 (only one of the two port of the modem works), otherwise everything works quite out of the box if you run everything on docker container (haven't tried without a container).

[K.Stute]

@Riccardo-Franceschini is there any chance that you could send me your docker image (or maybe the layers that are used to install the realsense "software")? I've been struggling to get the realsense in a ros2 docker working. The only think that worked for me is a docker image based on Alpine.