Optimizing video streams

dan64

My team is looking into ways to minimize bandwidth and latency while streaming video from multiple Starling Max drones over 4G+VPN. In one of our tests, we disabled all but one pipe in voxl-camera-server.conf for the hires_front camera, and disabled all for the hires_down:

"en_preview": false,
"en_small_video": true,
"en_large_video": false,
"en_snapshot": false,

According to these docs, streams are not activated until a client connects, but disabling these other pipes seemed to have made a significant improvement in stream quality.
https://docs.modalai.com/voxl-camera-server/#hal3-streams

Can someone please confirm: if all pipes are enabled, and a client connects to a specific pipe (e.g. small video only), are all streams then activated? Any tips or suggestions to optimize bandwidth and reduce onboard overhead are appreciated!

dan64

For additional context, here is our voxl-streamer.conf:

{
  "input-pipe":  "hires_front_small_encoded",
  "bitrate":  1000000,
  "rotation":  0,
  "decimator":  1,
  "port":  8900
}

Alex Kushleyev

@dan64 ,

Can you please specify what resolution and FPS you need for your low latency stream? Also, are you going to simultaneously record a higher resolution stream no VOXL2 (not sent via WIFI).

which encoding are you using h264 or h265 and approximate desired bit rate? Please note for h265 you will need about half the bandwidth of h264, so h265 is strongly recommended if it is supported on the receiving end.

Also, what camera are you using?

We can provide guidance for achieving minimum latency with your specifications.

Alex

paul.ammer

@Alex-Kushleyev

I work with Dan and he submitted this post on my behalf before my forum account was created.

Right now, we're using the default resolution + FPS on the hires_front camera (using the "small_video"), so 1024x768 at 30 FPS. I intend to experiment with lowering the frame rate and resolution next week. We are also using h265.

Alex Kushleyev

@paul-ammer , i am going to write up a document with some tips for getting low latency streaming working. Let me get back to you tomorrow.

I will also answer original questions from Dan.

Can you please clarify which camera you are using?

Alex

paul.ammer

@Alex-Kushleyev We're using the "hires_front" camera on the Starling 2 Max. I don't know exactly what type of camera (hardware-wise) is on the drone, nor how to find that information, but this is the current configuration for that camera in the voxl-camera-server.conf file:

{
			"type":	"imx412",
			"name":	"hires_front",
			"enabled":	true,
			"camera_id":	1,
			"fps":	30,
			"en_preview":	false,
			"preview_width":	640,
			"preview_height":	480,
			"en_raw_preview":	false,
			"en_small_video":	true,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h265",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"en_large_video":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h265",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	38,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	30,
			"en_snapshot":	true,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"ae_mode":	"isp"
		}

Does that answer your question?

Alex Kushleyev

@paul-ammer , yes thank you. The hires_front camera on Starling 2 Max is the IMX412 camera, specifically M0161 module (with IR cut filter) : https://www.modalai.com/products/msu-m0161?variant=48860038365488

Can you also please let me know if you are looking to record a higher resolution video (what resolution) while streaming a smaller resolution via wifi? This is important because it will determine whether we can use a lower resolution mode directly on camera, which reduced delay, as opposed to getting a very large 4K image and then down sampling it just for the smaller image. I will explain what I mean shortly in my detailed write up (sorry, taking a bit longer than expected).

Alex

paul.ammer

@Alex-Kushleyev For our particular use case, there isn't any particular need to record higher resolution video in addition to streaming the smaller resolution over the network (which for us is currently 4G LTE since we are not flying somewhere with WiFi available).