streamer default h265
-
@Alex-Kushleyev it's probably just my gstreamer ignorance and it not being as simple to set up an h265 stream as swapping our h264 with h265
@bendraper , you got 90% there!
I just tested your version and was able to reproduce the issue, but the following change fixes it:
from
"profile", G_TYPE_STRING, "baseline",to
"profile", G_TYPE_STRING, "main",There is no
baselineprofile in h265. Please try it - i was able to stream h265 with modified voxl-streamer and play usingffplay.full diff below. We will test more and probably mainline this soon.
Alex
diff --git a/src/pipeline.c b/src/pipeline.c index 53f47b7..5f88315 100644 --- a/src/pipeline.c +++ b/src/pipeline.c @@ -398,9 +398,9 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl // g_object_set(context->h264_parser, "alignment", "nal", NULL); // Configure the OMX encoder - g_object_set(context->omx_encoder, "control-rate", 1, + g_object_set(context->omx_h265_encoder, "control-rate", 1, "interval-intraframes", 30, NULL); - g_object_set(context->omx_encoder, "target-bitrate", + g_object_set(context->omx_h265_encoder, "target-bitrate", context->output_stream_bitrate, NULL); // Configure the RTP input queue @@ -443,10 +443,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl gst_caps_unref(filtercaps); // Configure the caps filter to reflect the output of the OMX encoder - filtercaps = gst_caps_new_simple("video/x-h264", + filtercaps = gst_caps_new_simple("video/x-h265", "width", G_TYPE_INT, context->output_stream_width, "height", G_TYPE_INT, context->output_stream_height, - "profile", G_TYPE_STRING, "baseline", + "profile", G_TYPE_STRING, "main", NULL); } if ( ! filtercaps) { @@ -482,10 +482,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl context->video_rotate, context->video_rotate_filter, context->encoder_queue, - context->omx_encoder, + context->omx_h265_encoder, context->rtp_filter, context->rtp_queue, - context->rtp_payload, + context->rtp_h265_payload, NULL); } @@ -536,10 +536,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl last_element = context->video_rotate_filter; if ( ! gst_element_link_many(last_element, context->encoder_queue, - context->omx_encoder, + context->omx_h265_encoder, context->rtp_filter, context->rtp_queue, - context->rtp_payload, + context->rtp_h265_payload, NULL)) { M_ERROR("Couldn't finish pipeline linking part 4\n"); return NULL; -
@bendraper , you got 90% there!
I just tested your version and was able to reproduce the issue, but the following change fixes it:
from
"profile", G_TYPE_STRING, "baseline",to
"profile", G_TYPE_STRING, "main",There is no
baselineprofile in h265. Please try it - i was able to stream h265 with modified voxl-streamer and play usingffplay.full diff below. We will test more and probably mainline this soon.
Alex
diff --git a/src/pipeline.c b/src/pipeline.c index 53f47b7..5f88315 100644 --- a/src/pipeline.c +++ b/src/pipeline.c @@ -398,9 +398,9 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl // g_object_set(context->h264_parser, "alignment", "nal", NULL); // Configure the OMX encoder - g_object_set(context->omx_encoder, "control-rate", 1, + g_object_set(context->omx_h265_encoder, "control-rate", 1, "interval-intraframes", 30, NULL); - g_object_set(context->omx_encoder, "target-bitrate", + g_object_set(context->omx_h265_encoder, "target-bitrate", context->output_stream_bitrate, NULL); // Configure the RTP input queue @@ -443,10 +443,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl gst_caps_unref(filtercaps); // Configure the caps filter to reflect the output of the OMX encoder - filtercaps = gst_caps_new_simple("video/x-h264", + filtercaps = gst_caps_new_simple("video/x-h265", "width", G_TYPE_INT, context->output_stream_width, "height", G_TYPE_INT, context->output_stream_height, - "profile", G_TYPE_STRING, "baseline", + "profile", G_TYPE_STRING, "main", NULL); } if ( ! filtercaps) { @@ -482,10 +482,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl context->video_rotate, context->video_rotate_filter, context->encoder_queue, - context->omx_encoder, + context->omx_h265_encoder, context->rtp_filter, context->rtp_queue, - context->rtp_payload, + context->rtp_h265_payload, NULL); } @@ -536,10 +536,10 @@ GstElement *create_custom_element(GstRTSPMediaFactory *factory, const GstRTSPUrl last_element = context->video_rotate_filter; if ( ! gst_element_link_many(last_element, context->encoder_queue, - context->omx_encoder, + context->omx_h265_encoder, context->rtp_filter, context->rtp_queue, - context->rtp_payload, + context->rtp_h265_payload, NULL)) { M_ERROR("Couldn't finish pipeline linking part 4\n"); return NULL; -
@Alex-Kushleyev Incredible, it works! Thanks Alex
@bendraper , great! Thank you for doing most of the work
I will add an option to specify output encoder type (for the unencoded input case) so h264/5 can be selected via command line argument.
Alex
-
@bendraper , great! Thank you for doing most of the work
I will add an option to specify output encoder type (for the unencoded input case) so h264/5 can be selected via command line argument.
Alex
@Alex-Kushleyev Awesome. On a different note, do you know why ffmpeg has no issue with the aspect ratio of the stream but in QGroundControl, the stream seems to be vertically squished? I would guess ffmpeg correctly makes some assumptions about the aspect ratio that voxl-streamer doesn't outwardly declare and maybe QGC doesn't make these assumptions?
-
@Alex-Kushleyev Awesome. On a different note, do you know why ffmpeg has no issue with the aspect ratio of the stream but in QGroundControl, the stream seems to be vertically squished? I would guess ffmpeg correctly makes some assumptions about the aspect ratio that voxl-streamer doesn't outwardly declare and maybe QGC doesn't make these assumptions?
@bendraper , i believe that when the encoder starts producing frames, the first frame is a header frame that has the stream description. This condition is also triggered if client disconnects (encoding stops) and then reconnects, causing the stream to start again. I am not sure what the whole pipeline looks like for getting video to QGC, but if the header is not sent over or is not parsed, this can lead to incorrect stream dimension? However i would assume that is the header is lost, the video would be completely garbled, not just aspect ratio, which probably means that QGC is in fact decoding correctly but simply rendering the image using a fixed aspect ratio…
-
@bendraper , i believe that when the encoder starts producing frames, the first frame is a header frame that has the stream description. This condition is also triggered if client disconnects (encoding stops) and then reconnects, causing the stream to start again. I am not sure what the whole pipeline looks like for getting video to QGC, but if the header is not sent over or is not parsed, this can lead to incorrect stream dimension? However i would assume that is the header is lost, the video would be completely garbled, not just aspect ratio, which probably means that QGC is in fact decoding correctly but simply rendering the image using a fixed aspect ratio…
I just pushed two updates to a branch of voxl-streamer:
- https://gitlab.com/voxl-public/voxl-sdk/services/voxl-streamer/-/commits/enable-h265-encode
- add option to use h265 encoder for non-encoded input frames (use either command line option
-e h265or--encoder h265or invoxl-streamer.conffile"encoder": "h264" - switch the h264 profile from
baselinetomainwhich allows usingCABACcoding, which has better compression properties than default CAVLC at some expense of HW encoder usage (negligible) and potential additional cpu usage on decoding side. However, this will definitely be better for low bandwidth streaming. h265 uses CABAC coding as the only option.
It seems like it's working well, but not much testing has been done so far..
Alex
-
I just pushed two updates to a branch of voxl-streamer:
- https://gitlab.com/voxl-public/voxl-sdk/services/voxl-streamer/-/commits/enable-h265-encode
- add option to use h265 encoder for non-encoded input frames (use either command line option
-e h265or--encoder h265or invoxl-streamer.conffile"encoder": "h264" - switch the h264 profile from
baselinetomainwhich allows usingCABACcoding, which has better compression properties than default CAVLC at some expense of HW encoder usage (negligible) and potential additional cpu usage on decoding side. However, this will definitely be better for low bandwidth streaming. h265 uses CABAC coding as the only option.
It seems like it's working well, but not much testing has been done so far..
Alex
-
@Alex-Kushleyev This is great, much appreciated! Also regarding that squished stream in QGC, seems to only be a linux thing so probably not much to do about it right now.
@bendraper You can also set the aspect ratio of the video in QGroundControls application settings.
-
@bendraper You can also set the aspect ratio of the video in QGroundControls application settings.
-
@Eric-Katzfey Yea I think that functionality may be broken though. If I try to do the inverse aspect ratio, it just kinda shrinks the stream as opposed to stretching it back out
@bendraper said in streamer default h265:
@Eric-Katzfey Yea I think that functionality may be broken though. If I try to do the inverse aspect ratio, it just kinda shrinks the stream as opposed to stretching it back out
It's also seemingly fixed in the daily builds of QGC... Oh well
-
I'm looking at the camera server documentation and this is saying that H265 essentially requires twice the bandwidth that H264 does. Isn't that backwards? Isn't H265 more efficient at the cost of latency? @Alex-Kushleyev
This is not a requirement but a current limitation of voxl-camera-server configuration of the encoder. Yes, h265 is generally 2x more efficient and having the minimum bitrate 2x compared to h264 does not make sense. This is a limitation due to some unknown mis-configuration of the encoder by the camera server. We will try to fix this, but it is not high priority, so not sure when this will happen.
The gstreamer (voxl-streamer) does not have this limitation, please double check it. I have tested voxl-streamer with bitrate down to 0.25 mbps in h265 mode and video looks surprisingly good. Such configuration can be good for streaming over very low bandwidth link.
By the way, voxl-camera-server supports dynamic bitrate updates by sending a command to camera server (within the mentioned limits), for example:
voxl-send-command hires_front_large_encoded set_large_venc_mbps 4.0Alex
-
This is not a requirement but a current limitation of voxl-camera-server configuration of the encoder. Yes, h265 is generally 2x more efficient and having the minimum bitrate 2x compared to h264 does not make sense. This is a limitation due to some unknown mis-configuration of the encoder by the camera server. We will try to fix this, but it is not high priority, so not sure when this will happen.
The gstreamer (voxl-streamer) does not have this limitation, please double check it. I have tested voxl-streamer with bitrate down to 0.25 mbps in h265 mode and video looks surprisingly good. Such configuration can be good for streaming over very low bandwidth link.
By the way, voxl-camera-server supports dynamic bitrate updates by sending a command to camera server (within the mentioned limits), for example:
voxl-send-command hires_front_large_encoded set_large_venc_mbps 4.0Alex
@Alex-Kushleyev Sounds good thanks for clarifying. Also, it looks like you can use a hybrid CQP and CBR mode per the documentation, but when I use "cbr+cqp" in the camera server conf file, it does not seem to like it. Is this feature not available yet?
-
@Alex-Kushleyev Sounds good thanks for clarifying. Also, it looks like you can use a hybrid CQP and CBR mode per the documentation, but when I use "cbr+cqp" in the camera server conf file, it does not seem to like it. Is this feature not available yet?
@bendraper , ah, yes the feature is available and I can see how the naming is misleading.
In order to achieve the CBR + CQP functionality, you do the following:
- select
cbrmode just like normal CBR - specify maximum bit rate, lets say
1.5 mbit, which is the lowest we can currently do with h264 - now if you want to achieve lower bitrate when there is not much motion (instead of always forcing the desired bit rate), set
_venc_Qminto a larger number (than the default 15 or so).
Basically by default
_venc_Qminis very low, which means the encoder is allowed to increase the image quality (decrease quantization level) if there is not much motion, so that you meet the target constant bit rate.By increasing this Qmin, you will not allow the encoder to keep increasing the quality under low motion condition, so it will cap the maximum quality level (Qmin) and reduce bandwidth. You can start with values of Qmin at 20, 25, 30 and see how low your bandwidth will get when not moving. When you start moving the camera, the bitrate should jump up to the CBR value, if needed. Please note that the quantization factor Qmin is not linear, i don't have the exact formula.
In general what happens in the encoder, if you want to maintain some CBR, the Quantization level is dynamically changed (within allowed range) depending on how much image is changing, so lots of motion in image will result in lowering overall quality (increasing quantization) in order to still be able to fit within the allowed CBR bitrate budget.
Please try it out and let me know how that works for you..
Alex
- select
-
@bendraper , ah, yes the feature is available and I can see how the naming is misleading.
In order to achieve the CBR + CQP functionality, you do the following:
- select
cbrmode just like normal CBR - specify maximum bit rate, lets say
1.5 mbit, which is the lowest we can currently do with h264 - now if you want to achieve lower bitrate when there is not much motion (instead of always forcing the desired bit rate), set
_venc_Qminto a larger number (than the default 15 or so).
Basically by default
_venc_Qminis very low, which means the encoder is allowed to increase the image quality (decrease quantization level) if there is not much motion, so that you meet the target constant bit rate.By increasing this Qmin, you will not allow the encoder to keep increasing the quality under low motion condition, so it will cap the maximum quality level (Qmin) and reduce bandwidth. You can start with values of Qmin at 20, 25, 30 and see how low your bandwidth will get when not moving. When you start moving the camera, the bitrate should jump up to the CBR value, if needed. Please note that the quantization factor Qmin is not linear, i don't have the exact formula.
In general what happens in the encoder, if you want to maintain some CBR, the Quantization level is dynamically changed (within allowed range) depending on how much image is changing, so lots of motion in image will result in lowering overall quality (increasing quantization) in order to still be able to fit within the allowed CBR bitrate budget.
Please try it out and let me know how that works for you..
Alex
I have updated the docs to clarify the setting for CBR + CQP mode:
Camera Server
ModalAI technical documentation for VOXL and VOXL 2 Companion Computers for PX4 and ArduPilot Obstacle Avoidance and GPS-denied navigation, assembled in the USA
ModalAI Technical Docs (docs.modalai.com)
CBR + CQP This is a hybrid mode that uses Constatnt Quantization when there is little motion and Constant Bitrate to limit the maximum bitrate In order to use this mode, select cbr as the bitrate control, however also adjust the Qmin parameter, as described below - select
-
I have updated the docs to clarify the setting for CBR + CQP mode:
Camera Server
ModalAI technical documentation for VOXL and VOXL 2 Companion Computers for PX4 and ArduPilot Obstacle Avoidance and GPS-denied navigation, assembled in the USA
ModalAI Technical Docs (docs.modalai.com)
CBR + CQP This is a hybrid mode that uses Constatnt Quantization when there is little motion and Constant Bitrate to limit the maximum bitrate In order to use this mode, select cbr as the bitrate control, however also adjust the Qmin parameter, as described below@bendraper, in case you need to set lower CBR bitrates than previously stated, we have a workaround for the 1.5 and 3.0 mbps limits. Camera server in dev branch supports lower bitrates and we have tested h265 as low as 0.25Mbps works pretty well.
The workaround tells the encoder that fps is higher than actual, allowing us to achieve effective lowering of the bitrate. You can just set your bitrate in camera server config to the desired bitrate and if bitrate is below the “limit”, the encoder fps setting is adjusted accordingly to achieve the low bitrate. This does not have any adverse effect on output quality and the fps value is not actually sent with the encoded stream.
If you want to adjust the bitrate dynamically below that “limit”, you may want to hardcode the value of
m_fpsHackScaleto a certain value like 12. Value of 12 would allow you to set h265 to 3.0/12 = 0.25Mbps or higher.Let me know if you need any more details. We will work towards improving this workaround in the future.
Alex
-
@bendraper, in case you need to set lower CBR bitrates than previously stated, we have a workaround for the 1.5 and 3.0 mbps limits. Camera server in dev branch supports lower bitrates and we have tested h265 as low as 0.25Mbps works pretty well.
The workaround tells the encoder that fps is higher than actual, allowing us to achieve effective lowering of the bitrate. You can just set your bitrate in camera server config to the desired bitrate and if bitrate is below the “limit”, the encoder fps setting is adjusted accordingly to achieve the low bitrate. This does not have any adverse effect on output quality and the fps value is not actually sent with the encoded stream.
If you want to adjust the bitrate dynamically below that “limit”, you may want to hardcode the value of
m_fpsHackScaleto a certain value like 12. Value of 12 would allow you to set h265 to 3.0/12 = 0.25Mbps or higher.Let me know if you need any more details. We will work towards improving this workaround in the future.
Alex
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login