Switching tack: I've configured voxl-camera-server to encode H.265 and write to /data/misc/camera:

Mem: 1396848K used, 2464472K free, 17096K shrd, 16692K buff, 318292K cached
CPU0: 32.2% usr 21.0% sys  0.0% nic 42.8% idle  0.2% io  2.1% irq  1.4% sirq
CPU1: 22.3% usr 29.8% sys  0.0% nic 46.5% idle  0.0% io  0.8% irq  0.4% sirq
CPU2: 53.7% usr  1.2% sys  0.0% nic 44.5% idle  0.0% io  0.2% irq  0.2% sirq
CPU3: 50.6% usr  0.4% sys  0.0% nic 48.7% idle  0.0% io  0.2% irq  0.0% sirq
Load average: 5.62 4.10 3.21 3/582 3801
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 3600     1 root     S     850m 22.4   0 30.1 /usr/bin/voxl-camera-server
 1966     1 root     S     665m 17.5   0 19.6 /system/bin/mm-qcamera-daemon
 2321     1 root     S     3988  0.1   1  0.9 /sbin/leprop-service
 2159     1 system   S     372m  9.8   1  0.7 /sbin/logd
    7     2 root     SW       0  0.0   1  0.4 [rcu_preempt]
 3445     2 root     SW       0  0.0   1  0.3 [kworker/u8:14]

And the CPU is in good shape:

Name   Freq (MHz) Temp (C) Util (%)
-----------------------------------
cpu0        480.0     66.3    25.86
cpu1        480.0     65.6    22.50
cpu2       2150.4     70.1    81.82
cpu3       2150.4     66.9    22.68
Total                 70.1    38.21
10s avg                       38.38
-----------------------------------
GPU           0.0     60.3     0.00
GPU 10s avg                    0.00
-----------------------------------
memory temp:       62.1 C
memory used:  1376/3770 MB
-----------------------------------
Flags
CPU freq scaling mode: auto
GPU freq scaling mode: auto
-----------------------------------

And this is voxl-camera-server encoding H.265, at 3840x2160x30fps, writing to flash.

So either voxl-streamer is doing software encoding very poorly, or something about the network stack is VERY broken. I'm inclined to think the former.

It looks like my VOXL is over-heating and down-clocking.

I've stripped the config down to the following:

• voxl-streamer only. voxl-camera-server is stopped.
• voxl-streamer configured to use video-test at 3840x2160x30fps with a bit rate of 1Mbps (I also tried 5Mbps and 10Mbps; the problems mentioned below were the same.)

   "video-test": {                   
       "input": {                    
           "interface": "test",      
           "frame": {          
               "width": 3840,  
               "height": 2160,        
               "format": "uyvy" } },
       "output": {                  
           "stream": {              
               "rotation": 0,       
               "width": 3840,       
               "height": 2160,
               "rate": 30,    
               "bitrate": 1000000 } } 
  

Observed results with one stream going:

• The video stream is still choppy (and very blocky at 1Mbps, as expected), lots of dropped frames.

• voxl-streamer is consumping about 75% total CPU. thermal-engine is about 5%, and the rest of "other" and about 12.5% idle.

• CPU consumption is 100% on CPU2 and CPU3, and about 50% on CPU0 and CPU1.

  Mem: 1182424K used, 2678896K free, 17100K shrd, 16348K buff, 215972K cached
  CPU0: 45.9% usr 21.1% sys  0.0% nic 29.2% idle  0.0% io  2.5% irq  1.1% sirq
  CPU1: 66.0% usr 12.8% sys  0.0% nic 19.2% idle  0.0% io  1.2% irq  0.5% sirq
  CPU2: 99.4% usr  0.1% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq  0.3% sirq
  CPU3: 98.5% usr  0.5% sys  0.0% nic  0.1% idle  0.0% io  0.7% irq  0.0% sirq
  Load average: 5.98 4.01 2.74 5/591 3516
    PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
   3491  3228 root     S     781m 20.6   0 76.8 voxl-streamer -c video-test
   1995     1 root     S <  2154m 56.8   0  5.6 /usr/bin/thermal-engine
   2378     2 root     SW       0  0.0   0  1.0 [msm_thermal:fre]
    188     2 root     SW       0  0.0   1  0.5 [kgsl_worker_thr]
    249     2 root     SW       0  0.0   2  0.4 [kworker/u8:6]
   2212     1 root     S    18872  0.4   0  0.3 /usr/bin/voxl-cpu-monitor
  

• voxl-cpu-monitor is warning of overheating, and slow clock rates:

  Name   Freq (MHz) Temp (C) Util (%)
  -----------------------------------
  cpu0        307.2     81.6    12.72
  cpu1        307.2     81.6    15.26
  cpu2       1248.0     92.5    58.04
  cpu3       1248.0     93.8    58.04
  Total                 93.8    36.01
  10s avg                       35.95
  -----------------------------------
  GPU           0.0     74.6     0.00
  GPU 10s avg                    0.00
  -----------------------------------
  memory temp:       77.1 C
  memory used:  1156/3770 MB
  -----------------------------------
  Flags
  CPU freq scaling mode: auto
  GPU freq scaling mode: auto
  CPU OVERHEAT WARNING
  -----------------------------------
  

  And this is AFTER putting a heat-sink on the CPU/RAM stack (An unused copper RPi sink I had laying about.)

• Also note the GPU is at 0%. Is the hardware H.264 encoder in the GPU, or part of the CPU?

Your suggestion of lowering the resolution and working my way up doesn't make sense, if "4k should work fine." Unless the resolution at which it starts working reliably will give you some hint at what the problem is. Other than lowering the resolution, I don't know how to scale this back any more to get it to work correctly. Can anyone else see something I'm doing wrong?

It really looks like its trying to encode in software. Is there an OMX daemon I need to start, or library to install, or something to get it to use the hardware encoder?

Here's a FLIR I took to confirm which chip was getting hot, and where the heat sink was needed. I think it's interesting to see that the heat is obviously only half the chip, showing that only two of the four CPUs were being stressed.

While working on my H.265 problem, I wanted to confirm that I could at least get 4k video off VOXL in the already supported modes. This is what I've done:

• Configured voxl-camera-server's preview profile for 3840x2160, 30fps, nv21 format. All defaults, except increase the resolution.
• Configured voxl-streamer's hires profile for 3840x2160, 50Mbps bitrate, everything else defaults.
• Restarted voxl-camera-server and started voxl-streamer.
• voxl-inspect-cam confirms voxl-camera-server is sending 3840x2160 at 29.9fps, NV21 format.
• With no clients, top claims 50% idle CPU, and no errors on voxl-streamer console.

As soon as I connect with VLC, I start getting errors on voxl-streamer console (see below), the CPU jumps to ~1% idle (about 50% user, 45% system, and a few percent in Idle, IRQ, SIRQ, etc.) with voxl-streamer taking about 65 to 70% of total CPU, mm-qcamera-daemon and voxl-camera-server are taking up most of the rest of it. Frame rate in VLC is a frame every few seconds. Video is often blocky (like missed frames). The "Statistics" tab of VLC is showing Lost frames, Decoded and Displayed don't match, etc.

In short, VOXL isn't keeping up with 4K Video, in the Default configs. Not by a long shot. Is there something else I need to do to get 4K video streamed off VOXL? Or do something differently?

Thanks for your help!

-Mark

@Chad-Sweet voxl-camera-server isn't doing the encoding to H.265? What ends up in /data/misc/camera/modalai_video.h265 then? Just the raw frames? Or did you mean that voxl-camera-server doesn't write H.265 to MPA? (See below.)

On a side note: I enabled both preview and video to see if I could write to MPA and to disk at the same time, and this is what voxl-inspect-cam shows:

|size(bytes)| height | width  |exposure(ms)| gain | frame id |latency(ms)|Framerate(hz)| format
|  1179648  |    768 |   1024 |       8.33 |   81 |   15811  |      54.7 |      0.0    | H265
|   460800  |    480 |    640 |       8.33 |   81 |   15812  |      45.4 |      0.0    | NV21
|  1179648  |    768 |   1024 |       8.33 |   81 |   15813  |      51.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15814  |      44.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15815  |      50.8 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15816  |      47.6 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15817  |      49.7 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15818  |      49.2 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15819  |      50.3 |      0.0    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15820  |      50.3 |     30.1    | H265
|  1179648  |    768 |   1024 |       8.33 |   81 |   15821  |      52.2 |     30.1    | H265
[...]

Basically, a whole lot of H.265 frames interleaved with a few nv21 frames. This is coming from MPA. The H.265 data is also being written to disk. The fact that the Size in Bytes is exactly the same for every frame, and it's exactly 12 bits per pixel (1179648/(1024*768) = 1.5), suggests that this is not actually getting encoded before transmission.

@Chad-Sweet Oh, interesting. Sorry, I missed that detail the first time you said it, my bad.

If voxl-camera-server is writing H.265 to MPA instead of NV21, will voxl-streamer attempt to transcode that to H.264? Or will it just pass it through to RTMP?

@Eric-Katzfey said in Streaming H.265/HEVC from hires camera on VOXL:

@SmittyHalibut Yes, I believe you have most of that correct. On the 821 part Qualcomm supports, primarily, their own proprietary QMMF framework. This will generate h265 and it will use hardware. voxl-streamer is based on GStreamer and uses the Qualcomm OMX element which only supports h264.

Clarification on this statement: OMX doesn't support H.265! I get it now. Looks like the latest proposed spec was v1.2.0 and it doesn't specify H.265 or HEVC anywhere. It's not that GStreamer hasn't been updated to use H.265, its that OMX, the underlying interface that GStreamer (and therefore voxl-streamer I understand the problem now.

So the next thing to figure out: Is QMMF available on VOXL? Or am I looking at installing it myself? Or am I looking at a whole different OS? Let's find out!

Again, thanks for all your help.

@Chad-Sweet Well, I'll be... Look at that, I see the file now.

Ok, so video writes to disk, and preview streams to MPA. Gotcha.

@Eric-Katzfey Bah. Different frameworks. Ok, I'll poke at it. I've just signed up for QDN (I did say I'm just getting started at this, right? ) so hopefully I'll be able to get some documentation. If you have any pointers on getting started with this: which forums, documentation, people to talk to, etc... I'd appreciate it.

I can copy-search/replace-hack-until-it-works with the best of them, but I am NOT a video expert, so don't hold your breath for any PRs any time soon.

Again, thanks for filling in the blanks in my understanding and pointing me in the right direction.

@Chad-Sweet I've got to at least modify voxl-camera-server to send 4k video to MPA, right? Is the preview vs video thing a red herring then?

@Eric-Katzfey The Snapdragon 821 page claims it'll do H.265 at 30fps. Is that "it has enough CPU horsepower to do H.265 at 30fps in software", or "It has a hardware encoder, but the OMX driver support just isn't there yet?" Or is that a question for Qualcomm?

Thank you both for your help in understanding how all this fits together.

So, if I'm understanding correctly:

• voxl-camera-server only takes data directly from the camera and sends it to MPA fs-sockets. Whatever format the camera presents, is what shows up on MPA. So if the CAMERA did H.265 internally, then I could use the video configed for h265 and that would work? But since the IMX214 I'm using doesn't, I have to use nv21.)
• voxl-streamer is what actually does the transcoding from what the camera presents (nv21) to what sends up on the stream (OMX based H.264 in this case).

It looks like my next steps are to see what H.265 encode options Qualcomm was talking about, and figuring out how to add it to voxl-streamer.

If I get this working, y'all take pull-requests? I hate maintaining my own branches of supported-by-someone-else code.

@Chad-Sweet I assume that’s where it writes to the FIFO socket in /run/mpa/hires/.

voxl-streamer Gets its frames from voxl-camera-server via the FIFO mentioned above. Unless there’s a way to get it to read from MIPI directly, but the documentation reads like voxl-streamer only knows how to read from MPA.

The part that’s not working is getting voxl-camera-server to write an H.265 stream to MPA.

Or, alternatively, is voxl-streamer Capable of reencoding from whatever it gets from MPA, into H.265?

I'm trying to stream H.265/HEVC from the hires camera over the network. Eventually, I want 4k H.265, but right now I'm having trouble doing ANY H.265.

Following the instructions here:

• https://docs.modalai.com/voxl-camera-server/
• https://docs.modalai.com/voxl-streamer/

What works: I can stream using the preview profile in voxl-camera-server using the nv21 format. I've tried up to 1080p (haven't bothered trying higher yet) and it streams pretty well (some blips) to VLC on my laptop. VLC claims it's encoded with H.264 (not H.265).

What doesn't work:

• Changing the preview format from nv21 to h265, voxl-camera-server just stops running. When I look at the config file after, the format has changed to raw8.
  • I looked around the source code and it looks like preview and video have different formats they accept. h265 is listed as a video format. What I'm seeing is consistent with this.
• Disabling preview and enabling video, no matter whether the video format is h264 or h265 (the only two supported formats according to the source code), voxl-camera-server runs, but never sends any data to the MPA sockets.
  • I've confirmed this with both voxl-streamer and voxl-inspect-cam, both read from /run/mpa/hires/*. Neither of them see any frames, and voxl-streamer never opens a UDP socket.

Can someone help me understand what I'm missing? It looks like the problem is with voxl-camera-server, but I can't figure out what I'm doing wrong.

Here's /etc/modalai/voxl-camera-server.conf (but keep in mind the changes I mentioned above:

{
	"version":	0.2,
	"port_J2":	{
		"name":	"hires",
		"enabled":	true,
		"type":	"hires",
		"api":	"hal3",
		"frame_rate":	30,
		"override_id":	-1,
		"auto_exposure_mode":	"isp",
		"preview":	{
			"enabled":	true,
			"width":	1920,
			"height":	1080,
			"format":	"nv21"
		},
		"video":	{
			"enabled":	false,
			"width":	1920,
			"height":	1080,
			"format":	"h265"
		},
		"snapshot":	{
			"enabled":	false,
			"width":	1024,
			"height":	768,
			"format":	"jpg"
		}
	},
	"port_J3":	{
		"name":	"empty",
		"enabled":	false
	},
	"port_J4":	{
		"name":	"empty",
		"enabled":	false
	}
}

Here's /etc/modalai/voxl-streamer.conf:

{
    "conf-version":"1.1",
    "configuration": "tracking",
    "hires": {
        "input": {
            "interface": "mpa",
            "mpa-camera": "hires" },
        "output": {
            "stream": {
                "rotation": 0,
                "width": 1920,
                "height": 1080,
                "decimator": 2,
                "bitrate": 50000000 } }
    },
[trimmed a bunch of not-relevant-here configs]
}

Thank you for your help.

-Mark