Running inference on a gstream with tflite

Samuel Lehman

Hello, I was wondering if any one knows a way to pipe a video stream from gstream into the tflite server so I can run inference on it. I am currently running the commands:

gst-launch-1.0 v4l2src device=/dev/video2 ! "video/x-raw, width=1920,height=1080" ! videoconvert ! video/x-raw,format=YUY2 ! videoconvert ! x264enc ! rtph264pay ! udpsink host=192.168.0.149 port=8554

on the voxl2 and

gst-launch-1.0 udpsrc port=8554 ! "application/x-rtp, payload=127" ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink

on my host machine which gives me a video stream.

Ideally I would just use the uvc server but I dont believe the uvc server supports the camera I am using. Is there a way to make a pipe out of the stream to then be the source for the tflite server? Or preferably a way to get it working with the uvc server?

For reference I am using a Sony FCB-ER8530/J 20x 4K CMOS Block Camera. Running voxl-uvc-server -l gives me

Got device descriptor for 04b4:00f9 USB3Neo 07780

Found device 04b4:00f9

DEVICE CONFIGURATION (04b4:00f9/USB3Neo 07780) ---
Status: idle
VideoControl:
	bcdUVC: 0x0110
VideoStreaming(1):
	bEndpointAddress: 131
	Formats:
	UncompressedFormat(1)
		  bits per pixel: 12
		  GUID: 4934323000001000800000aa00389b71 (I420)
		  default frame: 1
		  aspect ratio: 16x9
		  interlace flags: 00
		  copy protect: 00
			FrameDescriptor(1)
			  capabilities: 00
			  size: 1920x1080
			  bit rate: -446365696--446365696
			  max frame size: 16588800
			  default interval: 1/29
			  interval[0]: 1/29
			  interval[1]: 1/14
			  interval[2]: 1/7
			  interval[3]: 1/3
			  interval[4]: 1/1
			FrameDescriptor(2)
			  capabilities: 00
			  size: 3840x2160
			  bit rate: 962150400-962150400
			  max frame size: 4147200
			  default interval: 1/29
			  interval[0]: 1/29
			  interval[1]: 1/14
			  interval[2]: 1/7
			  interval[3]: 1/3
			  interval[4]: 1/1

tom

@Samuel-Lehman If this is a UVC camera you may be able to use voxl-uvc-server in order to create an MPA pipe and have tflite server read from that pipe

It may take some slight tweaking to get voxl-uvc-server to support your specific camera