Periodic Intra Video Refresh

[Anthony Kang]

Hello ModalAI team,

I am currently working on optimizing my video pipeline for a low-latency, bandwidth-constrained radio link. To eliminate the massive bandwidth spikes caused by periodic IDR frames, I am implementing Gradual Decoder Refresh (Periodic Intra Refresh) by modifying voxl-camera-server.

Here is my current hardware setup:

• Compute: VOXL2 (QRB5165)

• Image Sensor: IMX412

• Target Stream: Small video stream (H.265 / CBR) while adjusting bitrate dynamically via int OMXVideoEncoder::SetTargetBitrate(uint32_t bps)

I am planning editing src/venc/OMX.cpp and setting the following parameter: OMX_IndexParamVideoIntraRefresh. Before starting implementation, I had questions about the approach and tuning parameters.

def struct OMX_VIDEO_PARAM_INTRAREFRESHTYPE {
 OMX_U32 nSize;
 OMX_VERSIONTYPE nVersion;
 OMX_U32 nPortIndex;
 OMX_VIDEO_INTRAREFRESHTYPE eRefreshMode;
 OMX_U32 nAirMBs;
 OMX_U32 nAirRef;
 OMX_U32 nCirMBs;
} OMX_VIDEO_PARAM_INTRAREFRESHTYPE; 

• Cyclical vs. Adaptive vs. Hybrid: According to "The OpenMAX Integration Layer Specification", the QRB5165 supports three different modes for periodic intra refresh. Cyclical keeps a steady bitrate, adaptive can handle rapid motion, and hybrid combines these two approaches. Ideally, the hybrid seems like the best use case for drone FPV but adds a layer of implementation complexity. Which mode would you recommend?

• Param Tuning: What is the suggested approach to tuning nAirMBs, nAirRef, and nCirMBs experimentally? Do you already have a set of recommended values?

• Should nCirMBs scale linearly with the target bitrate when the dynamic bitrate path adjusts mbps at runtime? And for H.265, is the MB count interpreted at 16×16 MB granularity or translated internally to 32×32/64×64 CTUs?

• Configuration Conflicts: Are there any known conflicts between setting OMX_IndexParamVideoIntraRefresh and the other default QMMF parameters (like the dynamic bitrate updates) currently set in OMX.cpp?

• Firmware/HAL quirks: Are there any specific "gotchas" or undocumented requirements for the QCOM Media HAL on the VOXL2 when using periodic intra refresh?

Any advice on tuning these parameters for the QRB5165 would be highly appreciated.

Thank you!

[Alex Kushleyev]

@Anthony-Kang , we are looking into this. However, initial attempts to enable Intra Refresh on VOXL2 via the OMX api did not succeed either for h264 or h265. Also h265 does not work with fixed Macro Blocks (Coding Tree Units instead), so it seems the OMX structures for this param would not work with h265.

Alex