Optimizing video streams

[dan64]

My team is looking into ways to minimize bandwidth and latency while streaming video from multiple Starling Max drones over 4G+VPN. In one of our tests, we disabled all but one pipe in voxl-camera-server.conf for the hires_front camera, and disabled all for the hires_down:

"en_preview": false,
"en_small_video": true,
"en_large_video": false,
"en_snapshot": false,

According to these docs, streams are not activated until a client connects, but disabling these other pipes seemed to have made a significant improvement in stream quality.
https://docs.modalai.com/voxl-camera-server/#hal3-streams

Can someone please confirm: if all pipes are enabled, and a client connects to a specific pipe (e.g. small video only), are all streams then activated? Any tips or suggestions to optimize bandwidth and reduce onboard overhead are appreciated!

[dan64]

For additional context, here is our voxl-streamer.conf:

{
  "input-pipe":  "hires_front_small_encoded",
  "bitrate":  1000000,
  "rotation":  0,
  "decimator":  1,
  "port":  8900
}

[Alex Kushleyev]

@dan64 ,

Can you please specify what resolution and FPS you need for your low latency stream? Also, are you going to simultaneously record a higher resolution stream no VOXL2 (not sent via WIFI).

which encoding are you using h264 or h265 and approximate desired bit rate? Please note for h265 you will need about half the bandwidth of h264, so h265 is strongly recommended if it is supported on the receiving end.

Also, what camera are you using?

We can provide guidance for achieving minimum latency with your specifications.

Alex

[paul.ammer]

@Alex-Kushleyev

I work with Dan and he submitted this post on my behalf before my forum account was created.

Right now, we're using the default resolution + FPS on the hires_front camera (using the "small_video"), so 1024x768 at 30 FPS. I intend to experiment with lowering the frame rate and resolution next week. We are also using h265.

[Alex Kushleyev]

@paul-ammer , i am going to write up a document with some tips for getting low latency streaming working. Let me get back to you tomorrow.

I will also answer original questions from Dan.

Can you please clarify which camera you are using?

Alex

[paul.ammer]

@Alex-Kushleyev We're using the "hires_front" camera on the Starling 2 Max. I don't know exactly what type of camera (hardware-wise) is on the drone, nor how to find that information, but this is the current configuration for that camera in the voxl-camera-server.conf file:

{
			"type":	"imx412",
			"name":	"hires_front",
			"enabled":	true,
			"camera_id":	1,
			"fps":	30,
			"en_preview":	false,
			"preview_width":	640,
			"preview_height":	480,
			"en_raw_preview":	false,
			"en_small_video":	true,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h265",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"en_large_video":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h265",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	38,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	30,
			"en_snapshot":	true,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"ae_mode":	"isp"
		}

Does that answer your question?

[Alex Kushleyev]

@paul-ammer , yes thank you. The hires_front camera on Starling 2 Max is the IMX412 camera, specifically M0161 module (with IR cut filter) : https://www.modalai.com/products/msu-m0161?variant=48860038365488

Can you also please let me know if you are looking to record a higher resolution video (what resolution) while streaming a smaller resolution via wifi? This is important because it will determine whether we can use a lower resolution mode directly on camera, which reduced delay, as opposed to getting a very large 4K image and then down sampling it just for the smaller image. I will explain what I mean shortly in my detailed write up (sorry, taking a bit longer than expected).

Alex

[paul.ammer]

@Alex-Kushleyev For our particular use case, there isn't any particular need to record higher resolution video in addition to streaming the smaller resolution over the network (which for us is currently 4G LTE since we are not flying somewhere with WiFi available).

[Alex Kushleyev]

@paul-ammer , sorry for the delay. Initial version of the document that addresses latency in wifi streaming has been published : https://docs.modalai.com/camera-video/low-latency-video-streaming/

There is a lot of information there, please take a look and let me know where you get stuck.

I suggest starting to experiment without updating the camera drivers (as discussed in the document), so you reduce number of variables at the beginning. You can also use the Qualcomm ISP pipeline (using hires_front_small_encoded stream) and we can discuss what MISP can be used for at another time.

In general, you will find that the majority of the delay is going to be on the receiver side, with some additional delay possible in the server side after the camera pipeline (buffering of the encoded frames before sending them out - I believe gstreamer rtsp server adds some delay. voxl-streamer is based on gstreamer).

Regarding the earlier question of unused streams: yes it is true that if streams like large_video, etc, are enabled but not used, they will not consume significant resources, however, if enabled, the stream specifications (dimensions) will play a role when the camera pipeline selects the camera operating mode, which should satisfy requirements of all streams. The most extreme case is when you enable the snapshot at highest possible resolution (but use the snapshot very rarely), and your video streaming resolution is relatively low - the camera pipeline will have to select the camera mode that provides the largest image size (which will be the slowest in terms of latency) and then will downscale to get the video stream of the desired (small) resolution.

Please let me know what other questions you may have. I will continue updating the document in order to clarify things and based on feedback. If other users have questions, feel free to follow up in this thread.

Alex