Streaming 4k video from VOXL,

SmittyHalibut

While working on my H.265 problem, I wanted to confirm that I could at least get 4k video off VOXL in the already supported modes. This is what I've done:

Configured voxl-camera-server's preview profile for 3840x2160, 30fps, nv21 format. All defaults, except increase the resolution.
Configured voxl-streamer's hires profile for 3840x2160, 50Mbps bitrate, everything else defaults.
Restarted voxl-camera-server and started voxl-streamer.
voxl-inspect-cam confirms voxl-camera-server is sending 3840x2160 at 29.9fps, NV21 format.
With no clients, top claims 50% idle CPU, and no errors on voxl-streamer console.

As soon as I connect with VLC, I start getting errors on voxl-streamer console (see below), the CPU jumps to ~1% idle (about 50% user, 45% system, and a few percent in Idle, IRQ, SIRQ, etc.) with voxl-streamer taking about 65 to 70% of total CPU, mm-qcamera-daemon and voxl-camera-server are taking up most of the rest of it. Frame rate in VLC is a frame every few seconds. Video is often blocky (like missed frames). The "Statistics" tab of VLC is showing Lost frames, Decoded and Displayed don't match, etc.

In short, VOXL isn't keeping up with 4K Video, in the Default configs. Not by a long shot. Is there something else I need to do to get 4K video streamed off VOXL? Or do something differently?

Thanks for your help!

-Mark

Chad Sweet

You'll need to isolate a bad network first. 50Mbps is really high. Typical debugging for this would be to start at a low bitrate and work up. 4k should work fine

Eric Katzfey

@SmittyHalibut Best to move everything up slowly. And one thing at a time. For example, move to 720p and see how the system is performing. Then increase bitrate to say 5Mbps and see how things are going. Another really nice way to debug things with voxl-streamer is to use the video-test configuration. That takes the camera and MPA part out of the picture and let's you just focus on resolution, frame rate, bit rate, etc. Then add in the camera after everything looks solid with video-test.

SmittyHalibut

It looks like my VOXL is over-heating and down-clocking.

I've stripped the config down to the following:

voxl-streamer only. voxl-camera-server is stopped.

voxl-streamer configured to use video-test at 3840x2160x30fps with a bit rate of 1Mbps (I also tried 5Mbps and 10Mbps; the problems mentioned below were the same.)

 "video-test": {                   
     "input": {                    
         "interface": "test",      
         "frame": {          
             "width": 3840,  
             "height": 2160,        
             "format": "uyvy" } },
     "output": {                  
         "stream": {              
             "rotation": 0,       
             "width": 3840,       
             "height": 2160,
             "rate": 30,    
             "bitrate": 1000000 } }

Observed results with one stream going:

The video stream is still choppy (and very blocky at 1Mbps, as expected), lots of dropped frames.
voxl-streamer is consumping about 75% total CPU. thermal-engine is about 5%, and the rest of "other" and about 12.5% idle.

CPU consumption is 100% on CPU2 and CPU3, and about 50% on CPU0 and CPU1.

Mem: 1182424K used, 2678896K free, 17100K shrd, 16348K buff, 215972K cached
CPU0: 45.9% usr 21.1% sys  0.0% nic 29.2% idle  0.0% io  2.5% irq  1.1% sirq
CPU1: 66.0% usr 12.8% sys  0.0% nic 19.2% idle  0.0% io  1.2% irq  0.5% sirq
CPU2: 99.4% usr  0.1% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq  0.3% sirq
CPU3: 98.5% usr  0.5% sys  0.0% nic  0.1% idle  0.0% io  0.7% irq  0.0% sirq
Load average: 5.98 4.01 2.74 5/591 3516
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 3491  3228 root     S     781m 20.6   0 76.8 voxl-streamer -c video-test
 1995     1 root     S <  2154m 56.8   0  5.6 /usr/bin/thermal-engine
 2378     2 root     SW       0  0.0   0  1.0 [msm_thermal:fre]
  188     2 root     SW       0  0.0   1  0.5 [kgsl_worker_thr]
  249     2 root     SW       0  0.0   2  0.4 [kworker/u8:6]
 2212     1 root     S    18872  0.4   0  0.3 /usr/bin/voxl-cpu-monitor

voxl-cpu-monitor is warning of overheating, and slow clock rates:

Name   Freq (MHz) Temp (C) Util (%)
-----------------------------------
cpu0        307.2     81.6    12.72
cpu1        307.2     81.6    15.26
cpu2       1248.0     92.5    58.04
cpu3       1248.0     93.8    58.04
Total                 93.8    36.01
10s avg                       35.95
-----------------------------------
GPU           0.0     74.6     0.00
GPU 10s avg                    0.00
-----------------------------------
memory temp:       77.1 C
memory used:  1156/3770 MB
-----------------------------------
Flags
CPU freq scaling mode: auto
GPU freq scaling mode: auto
CPU OVERHEAT WARNING
-----------------------------------

And this is AFTER putting a heat-sink on the CPU/RAM stack (An unused copper RPi sink I had laying about.)

Also note the GPU is at 0%. Is the hardware H.264 encoder in the GPU, or part of the CPU?

Your suggestion of lowering the resolution and working my way up doesn't make sense, if "4k should work fine." Unless the resolution at which it starts working reliably will give you some hint at what the problem is. Other than lowering the resolution, I don't know how to scale this back any more to get it to work correctly. Can anyone else see something I'm doing wrong?

It really looks like its trying to encode in software. Is there an OMX daemon I need to start, or library to install, or something to get it to use the hardware encoder?

Here's a FLIR I took to confirm which chip was getting hot, and where the heat sink was needed. I think it's interesting to see that the heat is obviously only half the chip, showing that only two of the four CPUs were being stressed.
FLIR of VOXL 2022-02-11.jpeg

SmittyHalibut

Switching tack: I've configured voxl-camera-server to encode H.265 and write to /data/misc/camera:

Mem: 1396848K used, 2464472K free, 17096K shrd, 16692K buff, 318292K cached
CPU0: 32.2% usr 21.0% sys  0.0% nic 42.8% idle  0.2% io  2.1% irq  1.4% sirq
CPU1: 22.3% usr 29.8% sys  0.0% nic 46.5% idle  0.0% io  0.8% irq  0.4% sirq
CPU2: 53.7% usr  1.2% sys  0.0% nic 44.5% idle  0.0% io  0.2% irq  0.2% sirq
CPU3: 50.6% usr  0.4% sys  0.0% nic 48.7% idle  0.0% io  0.2% irq  0.0% sirq
Load average: 5.62 4.10 3.21 3/582 3801
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 3600     1 root     S     850m 22.4   0 30.1 /usr/bin/voxl-camera-server
 1966     1 root     S     665m 17.5   0 19.6 /system/bin/mm-qcamera-daemon
 2321     1 root     S     3988  0.1   1  0.9 /sbin/leprop-service
 2159     1 system   S     372m  9.8   1  0.7 /sbin/logd
    7     2 root     SW       0  0.0   1  0.4 [rcu_preempt]
 3445     2 root     SW       0  0.0   1  0.3 [kworker/u8:14]

And the CPU is in good shape:

Name   Freq (MHz) Temp (C) Util (%)
-----------------------------------
cpu0        480.0     66.3    25.86
cpu1        480.0     65.6    22.50
cpu2       2150.4     70.1    81.82
cpu3       2150.4     66.9    22.68
Total                 70.1    38.21
10s avg                       38.38
-----------------------------------
GPU           0.0     60.3     0.00
GPU 10s avg                    0.00
-----------------------------------
memory temp:       62.1 C
memory used:  1376/3770 MB
-----------------------------------
Flags
CPU freq scaling mode: auto
GPU freq scaling mode: auto
-----------------------------------

And this is voxl-camera-server encoding H.265, at 3840x2160x30fps, writing to flash.

So either voxl-streamer is doing software encoding very poorly, or something about the network stack is VERY broken. I'm inclined to think the former.