# Streaming H.265/HEVC from hires camera on VOXL Source: https://forum.modalai.com/topic/712/streaming-h-265-hevc-from-hires-camera-on-voxl Category: Flight Core and Legacy VOXL (https://forum.modalai.com/category/9/flight-core-and-legacy-voxl) Tags: voxl Posted: 2022-02-04 18:35:57 UTC by SmittyHalibut Replies: 15 · Views: 3320 ## SmittyHalibut · 2022-02-04 18:35:57 UTC I'm trying to stream H.265/HEVC from the `hires` camera over the network. Eventually, I want 4k H.265, but right now I'm having trouble doing ANY H.265. Following the instructions here: * https://docs.modalai.com/voxl-camera-server/ * https://docs.modalai.com/voxl-streamer/ **What works:** I can stream using the `preview` profile in `voxl-camera-server` using the `nv21` format. I've tried up to 1080p (haven't bothered trying higher yet) and it streams pretty well (some blips) to VLC on my laptop. VLC claims it's encoded with H.264 (not H.265). **What doesn't work:** * Changing the `preview` format from `nv21` to `h265`, `voxl-camera-server` just stops running. When I look at the config file after, the format has changed to `raw8`. * I looked around the source code and it looks like `preview` and `video` have different formats they accept. `h265` is listed as a `video` format. What I'm seeing is consistent with this. * Disabling `preview` and enabling `video`, no matter whether the `video` format is `h264` or `h265` (the only two supported formats according to the source code), `voxl-camera-server` runs, but never sends any data to the MPA sockets. * I've confirmed this with both `voxl-streamer` and `voxl-inspect-cam`, both read from `/run/mpa/hires/*`. Neither of them see any frames, and `voxl-streamer` never opens a UDP socket. Can someone help me understand what I'm missing? It looks like the problem is with `voxl-camera-server`, but I can't figure out what I'm doing wrong. Here's `/etc/modalai/voxl-camera-server.conf` (but keep in mind the changes I mentioned above: ``` { "version": 0.2, "port_J2": { "name": "hires", "enabled": true, "type": "hires", "api": "hal3", "frame_rate": 30, "override_id": -1, "auto_exposure_mode": "isp", "preview": { "enabled": true, "width": 1920, "height": 1080, "format": "nv21" }, "video": { "enabled": false, "width": 1920, "height": 1080, "format": "h265" }, "snapshot": { "enabled": false, "width": 1024, "height": 768, "format": "jpg" } }, "port_J3": { "name": "empty", "enabled": false }, "port_J4": { "name": "empty", "enabled": false } } ``` Here's `/etc/modalai/voxl-streamer.conf`: ``` { "conf-version":"1.1", "configuration": "tracking", "hires": { "input": { "interface": "mpa", "mpa-camera": "hires" }, "output": { "stream": { "rotation": 0, "width": 1920, "height": 1080, "decimator": 2, "bitrate": 50000000 } } }, [trimmed a bunch of not-relevant-here configs] } ``` Thank you for your help. -Mark ## Reply by Chad Sweet (ModalAI staff) · 2022-02-04 23:36:42 UTC Following through the code, it looks like the video handler simply writes the data to a file. This isn't really a complete answer, but I would start looking at the code [here](https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/omx_video_encoder/omx_video_encoder.cpp#L741) and see if you can find where that data goes. [voxl-streamer](https://docs.modalai.com/voxl-streamer/) is another program you might want to look at. It's gstreamer based and handles a lot of this stuff with gstreamer ## Reply by SmittyHalibut · 2022-02-05 00:13:22 UTC (in reply to Chad Sweet) @Chad-Sweet I assume that’s where it writes to the FIFO socket in `/run/mpa/hires/`. `voxl-streamer` Gets its frames from `voxl-camera-server` via the FIFO mentioned above. Unless there’s a way to get it to read from MIPI directly, but the documentation reads like `voxl-streamer` only knows how to read from MPA. The part that’s not working is getting `voxl-camera-server` to write an H.265 stream to MPA. Or, alternatively, is `voxl-streamer` Capable of reencoding from whatever it gets from MPA, into H.265? ## Reply by Chad Sweet (ModalAI staff) · 2022-02-06 01:38:40 UTC You should primarily reconfigure voxl-camera-server if you are changing which image sensors are connected to the platform. voxl-camera-server converts MIPI data to MPA. voxl-streamer converts MPA data to video. I believe you should only use voxl-streamer for what you are trying to achieve ## Reply by Chad Sweet (ModalAI staff) · 2022-02-06 01:41:02 UTC You should likely undo any changes you have made to voxl-camera-server's config file, or reinstall the SDK ## Reply by Eric Katzfey (ModalAI staff) · 2022-02-06 17:26:34 UTC (in reply to SmittyHalibut) @SmittyHalibut voxl-streamer is only capable of h264 at the moment. It relies on the OMX based encoder and that only supports h264. ## Reply by SmittyHalibut · 2022-02-07 18:28:13 UTC (in reply to Eric Katzfey) @Chad-Sweet I've got to at least modify `voxl-camera-server` to send 4k video to MPA, right? Is the `preview` vs `video` thing a red herring then? @Eric-Katzfey The [Snapdragon 821 page](https://www.qualcomm.com/products/snapdragon-821-mobile-platform) claims it'll do H.265 at 30fps. Is that "it has enough CPU horsepower to do H.265 at 30fps in software", or "It has a hardware encoder, but the OMX driver support just isn't there yet?" Or is that a question for Qualcomm? Thank you both for your help in understanding how all this fits together. So, if I'm understanding correctly: * `voxl-camera-server` only takes data directly from the camera and sends it to MPA fs-sockets. Whatever format the camera presents, is what shows up on MPA. So if the CAMERA did H.265 internally, then I could use the `video` configed for `h265` and that would work? But since the [IMX214 I'm using](https://www.modalai.com/products/voxl-4k-high-resolution-camera) doesn't, I have to use `nv21`.) * `voxl-streamer` is what actually does the transcoding from what the camera presents (`nv21`) to what sends up on the stream (OMX based H.264 in this case). It looks like my next steps are to see what H.265 encode options Qualcomm was talking about, and figuring out how to add it to `voxl-streamer`. If I get this working, y'all take pull-requests? I hate maintaining my own branches of supported-by-someone-else code. ## Reply by Chad Sweet (ModalAI staff) · 2022-02-07 18:39:07 UTC yes, we definitely encourage pull requests You would need to change resolution in the voxl-camera-server config, but that's about it. It will send NV21 which voxl-streamer understands When you set it to video it is currently just writing that data to a file ([here](https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp#L199)). that video setting is not a well supported feature, it should probably be integrated better with voxl-streamer at some point. ## Reply by Eric Katzfey (ModalAI staff) · 2022-02-07 18:39:25 UTC (in reply to SmittyHalibut) @SmittyHalibut Yes, I believe you have most of that correct. On the 821 part Qualcomm supports, primarily, their own proprietary QMMF framework. This will generate h265 and it will use hardware. voxl-streamer is based on GStreamer and uses the Qualcomm OMX element which only supports h264. ## Reply by SmittyHalibut · 2022-02-07 19:05:39 UTC (in reply to Chad Sweet) @Chad-Sweet Well, I'll be... Look at that, I see the file now. Ok, so `video` writes to disk, and `preview` streams to MPA. Gotcha. @Eric-Katzfey Bah. Different frameworks. Ok, I'll poke at it. I've just signed up for QDN (I did say I'm just getting started at this, right? ;-) ) so hopefully I'll be able to get some documentation. If you have any pointers on getting started with this: which forums, documentation, people to talk to, etc... I'd appreciate it. I can copy-search/replace-hack-until-it-works with the best of them, but I am NOT a video expert, so don't hold your breath for any PRs any time soon. Again, thanks for filling in the blanks in my understanding and pointing me in the right direction. ## Reply by SmittyHalibut · 2022-02-07 22:52:08 UTC (in reply to Eric Katzfey) @Eric-Katzfey said in [Streaming H\.265/HEVC from hires camera on VOXL](/post/3310): > @SmittyHalibut Yes, I believe you have most of that correct. On the 821 part Qualcomm supports, primarily, their own proprietary QMMF framework. This will generate h265 and it will use hardware. voxl-streamer is based on GStreamer and uses the Qualcomm OMX element which only supports h264. Clarification on this statement: _OMX_ doesn't support H.265! I get it now. Looks like the [latest proposed spec was v1.2.0](https://www.khronos.org/registry/OpenMAX-IL/specs/OpenMAX_IL_1_2_0_Specification.pdf) and it doesn't specify H.265 or HEVC anywhere. It's not that GStreamer hasn't been updated to use H.265, its that OMX, the underlying interface that GStreamer (and therefore `voxl-streamer` I understand the problem now. So the next thing to figure out: Is QMMF available on VOXL? Or am I looking at installing it myself? Or am I looking at a whole different OS? Let's find out! Again, thanks for all your help. ## Reply by Chad Sweet (ModalAI staff) · 2022-02-07 22:59:35 UTC QMMF is installed, but it is not needed here. You can see how to enable h265 encoding [here](https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp#L214). Again, this just writes the h265 data to a file, but you should have what you need to enable. ## Reply by SmittyHalibut · 2022-02-08 19:21:49 UTC (in reply to Chad Sweet) @Chad-Sweet Oh, interesting. Sorry, I missed that detail the first time you said it, my bad. If `voxl-camera-server` is writing H.265 to MPA instead of NV21, will `voxl-streamer` attempt to transcode that to H.264? Or will it just pass it through to RTMP? ## Reply by Chad Sweet (ModalAI staff) · 2022-02-08 19:31:12 UTC voxl-camera-server does not output h265 or any compressed video to MPA. It only outputs raw frame data to MPA. What is shown in the code is an example though of generating h265 data which you can use to build your own application. ## Reply by SmittyHalibut · 2022-02-09 03:19:19 UTC @Chad-Sweet `voxl-camera-server` isn't doing the encoding to H.265? What ends up in `/data/misc/camera/modalai_video.h265` then? Just the raw frames? Or did you mean that `voxl-camera-server` doesn't write H.265 to MPA? (See below.) On a side note: I enabled both `preview` and `video` to see if I could write to MPA and to disk at the same time, and this is what `voxl-inspect-cam` shows: ``` |size(bytes)| height | width |exposure(ms)| gain | frame id |latency(ms)|Framerate(hz)| format | 1179648 | 768 | 1024 | 8.33 | 81 | 15811 | 54.7 | 0.0 | H265 | 460800 | 480 | 640 | 8.33 | 81 | 15812 | 45.4 | 0.0 | NV21 | 1179648 | 768 | 1024 | 8.33 | 81 | 15813 | 51.7 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15814 | 44.7 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15815 | 50.8 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15816 | 47.6 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15817 | 49.7 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15818 | 49.2 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15819 | 50.3 | 0.0 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15820 | 50.3 | 30.1 | H265 | 1179648 | 768 | 1024 | 8.33 | 81 | 15821 | 52.2 | 30.1 | H265 [...] ``` Basically, a whole lot of H.265 frames interleaved with a few `nv21` frames. This is coming from MPA. The H.265 data is also being written to disk. The fact that the Size in Bytes is exactly the same for every frame, and it's exactly 12 bits per pixel (1179648/(1024*768) = 1.5), suggests that this is not actually getting encoded before transmission. ## Reply by Chad Sweet (ModalAI staff) · 2022-02-09 05:35:57 UTC (in reply to SmittyHalibut) @SmittyHalibut I updated my previous response to be more clear.