Using Qualcomm's camera subsystem on VOXL2

[AnthoJack]

Hi,

While reading about the capabilities of the qrb5165 I found that Qualcomm provides a camera subsystem with some of its platforms (notably the RB5 robotics platform) that I would like to try on the VOXL2. Sadly it isn't installed by default so I wanted to know if anyone had ever tried doing that before. If so, how well does it work and where can I find the packages to install it on the VOXL2 ?

I know that the VOXL2 image already provides a camera server but I would like to try using the qualcomm's version too to compare performance

[Chad Sweet]

Can you please be more specific about the package you are trying to use? Of course our voxl-camera-server is built on top of the Qualcomm camera framework.

[AnthoJack]

Oh so the qualcomm's camera server is installed already (couldn't find it mentioned anywhere). Then more specifically I'd like to use the qmmfsrc gstreamer element to recover images directly from it instead of going through your camera server but couldn't find how and where to retrieve the plugin

[modaltb]

We aren't supporting gst* stuff necessarily as our focus is via our camera-server as an attempt make general development on top of the QC stack easier over time. That said it is how I tested cameras initially and the framework is there and "works" as far as my use cases went, but it's a heavy lift sometimes going through the QC docs to find what you need.

For example, here's what I used to capture frames to disk to validate a ov7251 sensor:

gst-launch-1.0 -e qtiqmmfsrc camera=0 name=tracker ! video/x-h264,format=NV12,width=640,height=480,framerate=30/1 ! queue ! h264parse ! avdec_h264 ! videoconvert ! pngenc ! multifilesink enable-last-sample=false location="/data/misc/camera/cam0_img_%d_.png"

Now the gst- command line stuff is totally out of my wheel house, I generally copy paste commands here just to get my stuff done, but if you are familiar with gst- stuff then you should be able to find what you need on target.

[AnthoJack]

Great, thank you. I wasn't searching for the right name for the src module ("qmmfsrc" instead of "qtiqmmfsrc"). I'm not exactly knowledgeable about Gstreamer myself, I was just looking for a way to recover and process the images that allowed for a bit more control and configurability to do some tests.

You do seem to support gst stuff though since voxl-streamer uses it

[paynej]

I've been working on another RB5 platform and am migrating to VOXL. I used a command similar to above:

GST_DEBUG=3 gst-launch-1.0 -e qtiqmmfsrc camera=0 name=tracker ! video/x-h264,format=NV12,width=640,height=480,framerate=30/1 ! queue ! h264parse ! avdec_h264 ! videoconvert ! pngenc ! multifilesink enable-last-sample=false location="/data/misc/camera/cam0_img_%d_.png"

I get images, yet they are greyscale instead of color. Any help would be great!

Jeff

[Chad Sweet]

Which camera config are you using? https://docs.modalai.com/voxl2-camera-configs/

Are you sure it's not a monochrome image sensor you are using?

[paynej]

Should be hires and tracking only. I'm trying to use the color hires one. I noticed when using the voxl-streamer, that the tracking one is probably monochrome.
An older /etc/modalai/voxl-camera-server.conf.save shows all 4 cameras, tracking, hires and stereo.

Thanks for your reply. I hope to have this figured out soon.

[paynej]

C11. I'm trying to use the color hires one. I noticed when using the voxl-streamer, that the tracking ones and stereo is monochrome.

So now

GST_DEBUG=3 gst-launch-1.0 -e qtiqmmfsrc camera=3 ! video/x-raw,format=NV12,width=640,height=480,framerate=2/1 ! queue  ! videoconvert ! jpegenc ! multifilesink enable-last-sample=false location="/data/misc/camera/cam0_img_%d_.jpg"

is giving me no image.

All other cameras (0,1,2,4,5) are working correctly with B/W

If I enable voxl-camera-server and voxl-streamer-hires, then I get a RTSP color stream, so the HW seems good.

Thanks for your reply. I hope to have this figured out soon.

[paynej]

I think I have the snapshot hires color working!  I was grabbing the first picture instead of the last.  The first one, the sensor must not be ready.

[paynej]

Back again...

I have my ML program running now with the qtiqmmfscrc pipeline and disabling voxl-camera-server. I'd like to run voxl-camera-server with just the tracking and stereo cameras so voxl-qvio-server will get data. I tried to remove just the hires camera from /etc/modalai/voxl-camera-server.conf but running voxl-camera-server aborts:

rb5:/$ /usr/bin/voxl-camera-server -d 0
No secondary id found for camera: tracking, assuming mono
Secondary id found for camera: stereo_front, assuming stereo
Secondary id found for camera: stereo_rear, assuming stereo
------ voxl-camera-server: Starting camera server
Attempting to open the hal module
SUCCESS: Camera module opened on attempt 0
Aborted

rb5:/$ more /etc/modalai/voxl-camera-server.conf
{
	"version":	0.1,
	"cameras":	[{
			"name":	"tracking",
			"enabled":	true,
			"frame_rate":	30,
			"type":	"ov7251",
			"camera_id":	2,
			"ae_desired_msv":	60,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}, {
			"name":	"stereo_front",
			"enabled":	true,
			"frame_rate":	30,
			"type":	"ov7251",
			"camera_id":	0,
			"camera_id_second":	1,
			"ae_desired_msv":	60,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}, {
			"name":	"stereo_rear",
			"enabled":	true,
			"frame_rate":	30,
			"type":	"ov7251",
			"camera_id":	5,
			"camera_id_second":	4,
			"ae_desired_msv":	60,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}]
}

Any advice? Thank you for your time.

[Chad Sweet]

The Qualcomm HAL3 camera framework only allows one application to connect, which is why we created voxl-camera-server. What you want to do is not possible due to this limitation from Qualcomm

[paynej]

Thank you for the reply. That makes sense.

What are the available outputs of voxl-camera-server? It looks like voxl-qvio-server uses a pipe. Wondering if I have another option to qtiqmmfscrc, rtsp and udp.

Thanks you, I appreciate you taking the time to reply.

[Chad Sweet]

that is documented here: https://docs.modalai.com/voxl-camera-server/#output-of-camera-server

[paynej]

The Pipe API worked great! Thanks.

Can the hires camera output RBG instead of NV12 and can voxl-camera-server support that?