Qualcomm Flight RB5 5G Camera Setup

doe43john77 · 8 Jun 2023, 17:01

Hi,

I'm attempting to setup the hi-res camera on the Qualcomm Flight RB5 5G (RB5).
So far, I've been following the docs for the Qualcomm Flight RB5 SDK, available here.

Note: I understand that as of July 2022, the RB5 has integrated the VOXL SDK
into its firmware, replacing the old RB5 SDK. When I first began setting up the
RB5 and trying to connect it to QGroundControl (QGC), I performed this firware
update. In the process of troubleshooting the connection process, I was looking
through some systemd error messages which said that I was missing some
packages which happened to belong to the old RB5 SDK. I ended up reinstalling
the old firmware. The whole process of connecting to QGC took 2 weeks, so I'd
like to avoid a firmware update to VOXL if possible.

My first step was to configure the rb5-camera-server service via the adb
shell to run on startup using the rb5-camera-server-configure command. After
running this, I checked to see if the services were running for cameras 0, 1, 2
, 4 and 5. Here is the output:

Cameras 0, 1, 2 and 5 have similar outputs where the say there is an unknown
"lvalue" in the systemd configuration file (which I never touched):

● rb5-camera-server@0.service - "rb5-camera-server-0"
   Loaded: loaded (/usr/bin/rb5-camera-server; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2023-06-08 15:35:49 UTC; 34s ago
 Main PID: 2197 (rb5-camera-serv)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/system-rb5\x2dcamera\x2dserver.slice/rb5-camera-server@0.service
           └─2197 /usr/bin/rb5-camera-server -c 0

Jun 08 15:35:48 m0052 systemd[1]: Starting "rb5-camera-server-0"...
Jun 08 15:35:49 m0052 systemd[1]: Started "rb5-camera-server-0".
Jun 08 15:35:49 m0052 rb5-camera-server[2197]: gbm_create_device(156): Info: backend name is: msm_drm
Jun 08 15:35:49 m0052 systemd[1]: /etc/systemd/system/rb5-camera-server@.service:12: Unknown lvalue 'After' in section 'Service'
Jun 08 15:35:49 m0052 systemd[1]: /etc/systemd/system/rb5-camera-server@.service:13: Unknown lvalue 'Requires' in section 'Service'

Camera 4, on the other hand, gives the following output which follows the
example in the docs:

● rb5-camera-server@4.service - "rb5-camera-server-4"
   Loaded: loaded (/usr/bin/rb5-camera-server; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2023-06-08 15:35:50 UTC; 15min ago
  Process: 2219 ExecStartPre=/bin/sleep 1 (code=exited, status=0/SUCCESS)
 Main PID: 2224 (rb5-camera-serv)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/system-rb5\x2dcamera\x2dserver.slice/rb5-camera-server@4.service
           └─2224 /usr/bin/rb5-camera-server -c 4

Jun 08 15:35:49 m0052 systemd[1]: Starting "rb5-camera-server-4"...
Jun 08 15:35:50 m0052 systemd[1]: Started "rb5-camera-server-4".
Jun 08 15:35:52 m0052 rb5-camera-server[2224]: gbm_create_device(156): Info: backend name is: msm_drm

So, my first question: Why is the camera server service saying that there is
an unknown option in the startup configuration files when I never touched them
to begin with?

Next, I tested the cameras with the rb5-camera-client command. All cameras
began pushing metadata through the pipe successfully. I confirmed this by using
systemctl status rb5-camera-server@X.

Frame Information:
Individual Pixel Value 500: 4 

Metadata Information:
Timestamp:1267899026668 
Frame ID:43
Frame Width:640
Frame Height:480
fps: 30.00125

The frame width and FPS line up with the specs of the secondary cameras. An example of the systemd log is shown below:

● rb5-camera-server@0.service - "rb5-camera-server-0"
   Loaded: loaded (/usr/bin/rb5-camera-server; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2023-06-08 15:35:49 UTC; 28min ago
 Main PID: 2197 (rb5-camera-serv)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/system-rb5\x2dcamera\x2dserver.slice/rb5-camera-server@0.service
           └─2197 /usr/bin/rb5-camera-server -c 0

Jun 08 15:35:48 m0052 systemd[1]: Starting "rb5-camera-server-0"...
Jun 08 15:35:49 m0052 systemd[1]: Started "rb5-camera-server-0".
Jun 08 15:35:49 m0052 rb5-camera-server[2197]: gbm_create_device(156): Info: backend name is: msm_drm
Jun 08 15:35:49 m0052 systemd[1]: /etc/systemd/system/rb5-camera-server@.service:12: Unknown lvalue 'After' in section 'Service'
Jun 08 15:35:49 m0052 systemd[1]: /etc/systemd/system/rb5-camera-server@.service:13: Unknown lvalue 'Requires' in section 'Service'
Jun 08 15:55:26 m0052 rb5-camera-server[2197]: camera ID is set to 0
Jun 08 15:55:26 m0052 rb5-camera-server[2197]: [INFO] Publishing to the following pipe: /dev/rb5-camera-pipe-0
Jun 08 15:55:26 m0052 rb5-camera-server[2197]: [INFO] Started camera ID: 0

Second step was to configure the rb5-streamer service (which controls
camera 3, the hi-res camera) to run on startup using the
rb5-streamer-configure factory_enable command. The service seems to start up
normally based on the systemd log:

● rb5-streamer.service - rb5-streamer
   Loaded: loaded (/etc/systemd/system/rb5-streamer.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2023-06-08 16:09:28 UTC; 14s ago
 Main PID: 10448 (rb5-streamer)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/rb5-streamer.service
           └─10448 /usr/bin/rb5-streamer

Jun 08 16:09:28 m0052 systemd[1]: Started rb5-streamer.
Jun 08 16:09:28 m0052 rb5-streamer[10448]: [INFO] RTSP stream ready at rtsp://VEHICLES_IP:8554/camera3

For references, here is the example from the technical docs:

● rb5-streamer.service - "rb5-streamer"
   Loaded: loaded (/usr/bin/rb5-streamer; indirect; vendor preset: enabled)
   Active: active (running) since Mon 2021-07-26 16:28:00 UTC; 15s ago

Jul 26 16:28:00 qrb5165-rb5 systemd[1]: Started "rb5-streamer".
Jul 26 16:28:00 qrb5165-rb5 rb5-streamer[9522]: gbm_create_device(156)
Jul 26 16:28:00 qrb5165-rb5 rb5-streamer[9522]: stream ready at rtsp://127.0.0.1:8554/test

My output is missing gbm_create_device(156) and the actual IP address. I
should note that the drone is connected to Wi-Fi successfully and has an IP
address that I can ping and SSH into.

For the most part, everything starts up without any hiccups. However, when I
try to open the stream in VLC the stream fails. QGC doesn't even show the video
and I'm stumped. I followed the directions, checked I was putting the right
RTSP addressed and I have no clue how to proceed.

Help would be immensely appreciated.

Thanks!

tom · 8 Jun 2023, 17:05

@doe43john77 I'd recommend loading VOXL SDK 0.9.5 for RB5 and going from there: https://developer.modalai.com/asset/view/124

doe43john77 · 12 Jun 2023, 14:16

@tom Hi Tom. I updated the firmware of the RB5 to the VOXL 0.9.5 SDK. The update went smoothly. I'm trying to connect the drone to QGroundControl but I'm not having luck. I think I've covered all my bases: the host PC and drone are on the same Wi-Fi network, the drone's voxl-px4 service is running (as confirmed by using the voxl-inspect-services command). Suggestions?

tom · 12 Jun 2023, 17:01

@doe43john77 There are a few other services necessary in order to connect to QGC.

voxl-mavlink-server needs to be running
voxl-vision-px4 needs to be running

You also need to modify the config file at /etc/modalai/voxl-vision-px4.conf and change the qgc_ip value to the IP address of your ground station and restart the service with:
systemctl restart voxl-vision-px4

doe43john77 · 12 Jun 2023, 18:58

@tom I have enabled and started running the voxl-mavlink-server service but only been able to enable (but not start) the voxl-vision-px4 service. I've included a more detailed output voxl-vision-px4 service by running systemctl status voxl-vision-px4 -l:

rb5:/$ systemctl status voxl-vision-px4 -l
● voxl-vision-px4.service - voxl-vision-px4
   Loaded: loaded (/usr/bin/voxl-vision-px4; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2023-06-12 18:52:55 UTC; 33s ago
  Process: 2759 ExecStart=/usr/bin/voxl-vision-px4 (code=exited, status=255)
 Main PID: 2759 (code=exited, status=255)

Jun 12 18:52:55 m0052 voxl-vision-px4[2759]:     frame:              tof
Jun 12 18:52:55 m0052 systemd[1]: voxl-vision-px4.service: Main process exited, code=exited, status=255/n/a
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]: voa_input #4
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]:     enabled:            1
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]:     type:               rangefinder
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]:     input_pipe:         rangefinders
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]:     frame:              body
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]: =================================================================
Jun 12 18:52:55 m0052 voxl-vision-px4[2759]: loading extrinsics config file
Jun 12 18:52:55 m0052 systemd[1]: voxl-vision-px4.service: Failed with result 'exit-code'.

Also, I tried modifying the voxl-vision-px4.conf file with the IP address of my host machine with QGC in case the voxl-vision-px4 service needs the IP address for proper startup, but that didn't help.

Additionally, I saw the word "extrinsics" in the systemctl output and I was curious if it was referring to the opencv files we had to backup before the upgrading the SDK (which I did btw).

tom · 12 Jun 2023, 20:34

@doe43john77 In order to get all of the services in a know good spot, you should be able to run voxl-configure-mpa and select RB5-Flight and it'll setup all the services all needed. Then you would have to go back and modify the qgc_ip again in the conf file.

In regard to the opencv files, that's likely unrelated to this issue, but to restore them, you would just throw them into /data/modalai

doe43john77 · 13 Jun 2023, 14:45

@tom Got it! Its connecting to QGC now!

I still don't understand how to start up the cameras. The instructions in the user guide are in terms of the deprecated rb5-streamer package.

tom · 13 Jun 2023, 17:11

@doe43john77 If you ran through voxl-configure-mpa then voxl-camera-server should be enabled and running, you can check this with voxl-inspect-services.

Your options for viewing a camera stream are either using voxl-portal (navigate to the IP address of your drone in your PC's browser and select the camera tab)

or using voxl-streamer in order to stream to programs like QGC or VLC. This can be setup using voxl-configure-streamer

doe43john77 · 14 Jun 2023, 13:22

@tom Got it! I can access the VOXL Portal page and view the camera feed. The hi-res camera is sending 480p video for some reason. Is there a configuration file I need to tweak to push out the 4K resolution?

tom · 14 Jun 2023, 17:17

@doe43john77 You can find some info here: https://docs.modalai.com/voxl-camera-server/#changing-the-resolution-of-an-image-sensor

The default was set low as otherwise you're pumping a ton of data over the network which may not be necessary depending on the application

doe43john77 · 15 Jun 2023, 14:23

@tom Ah. This is a simple config file change. Thank you!