Streaming H.265/HEVC from hires camera on VOXL

SmittyHalibut · 4 Feb 2022, 18:35

I'm trying to stream H.265/HEVC from the hires camera over the network. Eventually, I want 4k H.265, but right now I'm having trouble doing ANY H.265.

Following the instructions here:

What works: I can stream using the preview profile in voxl-camera-server using the nv21 format. I've tried up to 1080p (haven't bothered trying higher yet) and it streams pretty well (some blips) to VLC on my laptop. VLC claims it's encoded with H.264 (not H.265).

What doesn't work:

Changing the preview format from nv21 to h265, voxl-camera-server just stops running. When I look at the config file after, the format has changed to raw8.
- I looked around the source code and it looks like preview and video have different formats they accept. h265 is listed as a video format. What I'm seeing is consistent with this.
Disabling preview and enabling video, no matter whether the video format is h264 or h265 (the only two supported formats according to the source code), voxl-camera-server runs, but never sends any data to the MPA sockets.
- I've confirmed this with both voxl-streamer and voxl-inspect-cam, both read from /run/mpa/hires/*. Neither of them see any frames, and voxl-streamer never opens a UDP socket.

Can someone help me understand what I'm missing? It looks like the problem is with voxl-camera-server, but I can't figure out what I'm doing wrong.

Here's /etc/modalai/voxl-camera-server.conf (but keep in mind the changes I mentioned above:

{
	"version":	0.2,
	"port_J2":	{
		"name":	"hires",
		"enabled":	true,
		"type":	"hires",
		"api":	"hal3",
		"frame_rate":	30,
		"override_id":	-1,
		"auto_exposure_mode":	"isp",
		"preview":	{
			"enabled":	true,
			"width":	1920,
			"height":	1080,
			"format":	"nv21"
		},
		"video":	{
			"enabled":	false,
			"width":	1920,
			"height":	1080,
			"format":	"h265"
		},
		"snapshot":	{
			"enabled":	false,
			"width":	1024,
			"height":	768,
			"format":	"jpg"
		}
	},
	"port_J3":	{
		"name":	"empty",
		"enabled":	false
	},
	"port_J4":	{
		"name":	"empty",
		"enabled":	false
	}
}

Here's /etc/modalai/voxl-streamer.conf:

{
    "conf-version":"1.1",
    "configuration": "tracking",
    "hires": {
        "input": {
            "interface": "mpa",
            "mpa-camera": "hires" },
        "output": {
            "stream": {
                "rotation": 0,
                "width": 1920,
                "height": 1080,
                "decimator": 2,
                "bitrate": 50000000 } }
    },
[trimmed a bunch of not-relevant-here configs]
}

Thank you for your help.

-Mark

Chad Sweet · 4 Feb 2022, 23:36

Following through the code, it looks like the video handler simply writes the data to a file. This isn't really a complete answer, but I would start looking at the code here and see if you can find where that data goes.

voxl-streamer is another program you might want to look at. It's gstreamer based and handles a lot of this stuff with gstreamer

SmittyHalibut · 5 Feb 2022, 00:13

@Chad-Sweet I assume that’s where it writes to the FIFO socket in /run/mpa/hires/.

voxl-streamer Gets its frames from voxl-camera-server via the FIFO mentioned above. Unless there’s a way to get it to read from MIPI directly, but the documentation reads like voxl-streamer only knows how to read from MPA.

The part that’s not working is getting voxl-camera-server to write an H.265 stream to MPA.

Or, alternatively, is voxl-streamer Capable of reencoding from whatever it gets from MPA, into H.265?

Chad Sweet · 6 Feb 2022, 01:38

You should primarily reconfigure voxl-camera-server if you are changing which image sensors are connected to the platform.

voxl-camera-server converts MIPI data to MPA.

voxl-streamer converts MPA data to video.

I believe you should only use voxl-streamer for what you are trying to achieve

Chad Sweet · 6 Feb 2022, 01:41

You should likely undo any changes you have made to voxl-camera-server's config file, or reinstall the SDK

Eric Katzfey · 6 Feb 2022, 17:26

@SmittyHalibut voxl-streamer is only capable of h264 at the moment. It relies on the OMX based encoder and that only supports h264.

SmittyHalibut · 7 Feb 2022, 18:28

@Chad-Sweet I've got to at least modify voxl-camera-server to send 4k video to MPA, right? Is the preview vs video thing a red herring then?

@Eric-Katzfey The Snapdragon 821 page claims it'll do H.265 at 30fps. Is that "it has enough CPU horsepower to do H.265 at 30fps in software", or "It has a hardware encoder, but the OMX driver support just isn't there yet?" Or is that a question for Qualcomm?

Thank you both for your help in understanding how all this fits together.

So, if I'm understanding correctly:

voxl-camera-server only takes data directly from the camera and sends it to MPA fs-sockets. Whatever format the camera presents, is what shows up on MPA. So if the CAMERA did H.265 internally, then I could use the video configed for h265 and that would work? But since the IMX214 I'm using doesn't, I have to use nv21.)
voxl-streamer is what actually does the transcoding from what the camera presents (nv21) to what sends up on the stream (OMX based H.264 in this case).

It looks like my next steps are to see what H.265 encode options Qualcomm was talking about, and figuring out how to add it to voxl-streamer.

If I get this working, y'all take pull-requests? I hate maintaining my own branches of supported-by-someone-else code.

Chad Sweet · 7 Feb 2022, 18:39

yes, we definitely encourage pull requests

You would need to change resolution in the voxl-camera-server config, but that's about it. It will send NV21 which voxl-streamer understands

When you set it to video it is currently just writing that data to a file (here). that video setting is not a well supported feature, it should probably be integrated better with voxl-streamer at some point.

Eric Katzfey · 7 Feb 2022, 18:39

@SmittyHalibut Yes, I believe you have most of that correct. On the 821 part Qualcomm supports, primarily, their own proprietary QMMF framework. This will generate h265 and it will use hardware. voxl-streamer is based on GStreamer and uses the Qualcomm OMX element which only supports h264.

SmittyHalibut · 7 Feb 2022, 19:05

@Chad-Sweet Well, I'll be... Look at that, I see the file now.

Ok, so video writes to disk, and preview streams to MPA. Gotcha.

@Eric-Katzfey Bah. Different frameworks. Ok, I'll poke at it. I've just signed up for QDN (I did say I'm just getting started at this, right? ) so hopefully I'll be able to get some documentation. If you have any pointers on getting started with this: which forums, documentation, people to talk to, etc... I'd appreciate it.

I can copy-search/replace-hack-until-it-works with the best of them, but I am NOT a video expert, so don't hold your breath for any PRs any time soon.

Again, thanks for filling in the blanks in my understanding and pointing me in the right direction.

SmittyHalibut · 7 Feb 2022, 22:52

@Eric-Katzfey said in Streaming H.265/HEVC from hires camera on VOXL:

@SmittyHalibut Yes, I believe you have most of that correct. On the 821 part Qualcomm supports, primarily, their own proprietary QMMF framework. This will generate h265 and it will use hardware. voxl-streamer is based on GStreamer and uses the Qualcomm OMX element which only supports h264.

Clarification on this statement: OMX doesn't support H.265! I get it now. Looks like the latest proposed spec was v1.2.0 and it doesn't specify H.265 or HEVC anywhere. It's not that GStreamer hasn't been updated to use H.265, its that OMX, the underlying interface that GStreamer (and therefore voxl-streamer I understand the problem now.

So the next thing to figure out: Is QMMF available on VOXL? Or am I looking at installing it myself? Or am I looking at a whole different OS? Let's find out!

Again, thanks for all your help.

Chad Sweet · 7 Feb 2022, 22:59

QMMF is installed, but it is not needed here. You can see how to enable h265 encoding here.

Again, this just writes the h265 data to a file, but you should have what you need to enable.

SmittyHalibut · 8 Feb 2022, 19:21

@Chad-Sweet Oh, interesting. Sorry, I missed that detail the first time you said it, my bad.

If voxl-camera-server is writing H.265 to MPA instead of NV21, will voxl-streamer attempt to transcode that to H.264? Or will it just pass it through to RTMP?

Chad Sweet · 9 Feb 2022, 05:34

voxl-camera-server does not output h265 or any compressed video to MPA. It only outputs raw frame data to MPA.

What is shown in the code is an example though of generating h265 data which you can use to build your own application.

SmittyHalibut · 9 Feb 2022, 03:19

@Chad-Sweet voxl-camera-server isn't doing the encoding to H.265? What ends up in /data/misc/camera/modalai_video.h265 then? Just the raw frames? Or did you mean that voxl-camera-server doesn't write H.265 to MPA? (See below.)

On a side note: I enabled both preview and video to see if I could write to MPA and to disk at the same time, and this is what voxl-inspect-cam shows:

|size(bytes)| height | width  |exposure(ms)| gain | frame id |latency(ms)|Framerate(hz)| format
|  1179648  |    768 |   1024 |       8.33 |   81 |   15811  |      54.7 |      0.0    | H265
|   460800  |    480 |    640 |       8.33 |   81 |   15812  |      45.4 |      0.0    | NV21
|  1179648  |    768 |   1024 |       8.33 |   81 |   15813  |      51.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15814  |      44.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15815  |      50.8 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15816  |      47.6 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15817  |      49.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15818  |      49.2 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15819  |      50.3 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15820  |      50.3 |     30.1    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15821  |      52.2 |     30.1    | H265
[...]

Basically, a whole lot of H.265 frames interleaved with a few nv21 frames. This is coming from MPA. The H.265 data is also being written to disk. The fact that the Size in Bytes is exactly the same for every frame, and it's exactly 12 bits per pixel (1179648/(1024*768) = 1.5), suggests that this is not actually getting encoded before transmission.

Chad Sweet · 9 Feb 2022, 05:35

@SmittyHalibut I updated my previous response to be more clear.