RTSP connection from drone unreliable since update to SDK 1.4.1
-
Since updating from SDK 1.3.5 to 1.4.1 on a starling 2 max I am experiencing intermittent issues when connecting to the RTSP streams from the drone. Sometimes these connect quickly with no issues, other times they hang on
waiting for progress to finishin the gstream pipeline. For example, running the commandgst-launch-1.0 rtspsrc location=rtsp://192.168.254.138:8901/live latency=0 ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosinkmay result in:Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Progress: (open) Opening Stream Pipeline is PREROLLED ... Prerolled, waiting for progress to finish... Progress: (connect) Connecting to rtsp://192.168.254.138:8901/live ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading and writing. Additional debug info: ../gst/rtsp/gstrtspsrc.c(8130): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Failed to connect. (Generic error) ERROR: pipeline doesn't want to preroll. Setting pipeline to NULL ... Freeing pipeline ...after significant delay. These errors seem to correlate with how long the system has been on. Additionally, the VOXL 2 boards now get extremely hot and seem to be under significant load
-
When this does occurs I see nothing unusual in the systemctl status messages:
● voxl-streamer-hires-down.service - voxl-streamer Loaded: loaded (/usr/bin/voxl-streamer; enabled; vendor preset: enabled) Active: active (running) since Thu 2025-02-27 23:48:55 UTC; 15min ago Process: 1479 ExecStartPre=/bin/sleep 5 (code=exited, status=0/SUCCESS) Main PID: 1955 (voxl-streamer) Tasks: 2 (limit: 4915) CGroup: /system.slice/voxl-streamer-hires-down.service └─1955 /usr/bin/voxl-streamer -s -p 8901 -i hires_down_small_encoded Feb 27 23:48:55 m0054 voxl-streamer[1955]: w: 1024 h: 768 fps: 30 format: H264 Feb 27 23:48:55 m0054 voxl-streamer[1955]: Stream available at rtsp://127.0.0.1:8901/live Feb 28 00:02:55 m0054 voxl-streamer[1955]: A new client rtsp://192.168.254.44:52848(null) has connected, total clients: 1 Feb 28 00:02:55 m0054 voxl-streamer[1955]: Camera server Connected Feb 28 00:04:03 m0054 voxl-streamer[1955]: rtsp client disconnected, total clients: 0 Feb 28 00:04:03 m0054 voxl-streamer[1955]: no more rtsp clients, closing source pipe intentionally Feb 28 00:04:04 m0054 voxl-streamer[1955]: A new client rtsp://192.168.254.44:38768(null) has connected, total clients: 1 Feb 28 00:04:04 m0054 voxl-streamer[1955]: Camera server Connected Feb 28 00:04:10 m0054 voxl-streamer[1955]: rtsp client disconnected, total clients: 0 Feb 28 00:04:10 m0054 voxl-streamer[1955]: no more rtsp clients, closing source pipe intentionallyvoxl-inspect-cpureturns:Name Freq (MHz) Temp (C) Util (%) ----------------------------------- cpu0 1075.2 85.3 19.45 cpu1 1075.2 84.1 10.51 cpu2 1075.2 84.1 12.28 cpu3 1075.2 84.9 10.03 cpu4 710.4 84.9 19.34 cpu5 710.4 86.1 20.54 cpu6 710.4 84.5 19.73 cpu7 844.8 84.9 4.80 Total 84.9 14.59 10s avg 15.06 ----------------------------------- small cores only 13.07 big cores only 16.11 ----------------------------------- GPU 587.0 80.7 7.43 GPU 10s avg 7.31 ----------------------------------- memory temp: 85.3 C memory used: 4195/7671 MB ----------------------------------- Flags CPU freq scaling mode: auto Standby Not Active -----------------------------------and top:
top - 00:07:09 up 10 min, 0 users, load average: 3.28, 2.91, 1.67 Tasks: 476 total, 2 running, 300 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.0 us, 14.0 sy, 0.0 ni, 67.3 id, 0.0 wa, 2.8 hi, 0.9 si, 0.0 st KiB Mem : 7855496 total, 2772832 free, 4527956 used, 554708 buff/cache KiB Swap: 3927744 total, 3927744 free, 0 used. 3552948 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1461 root -81 0 8650380 231620 66028 S 114.9 2.9 11:17.22 voxl-camera-ser 2134 root -51 0 1345204 107148 39084 S 77.2 1.4 7:28.22 voxl-open-vins- 1751 root 20 0 1011024 92336 11144 S 26.8 1.2 2:39.99 px4 1915 root 20 0 269308 6100 5524 S 4.3 0.1 0:25.73 voxl-mavlink-se 1503 root 20 0 852088 11892 6696 S 3.3 0.2 0:21.72 voxl-vision-hub 366 root 20 0 0 0 0 I 2.6 0.0 0:05.38 kworker/u16:11- 1953 root 20 0 617468 15208 9424 S 2.3 0.2 0:07.13 voxl-streamer 407 root 20 0 0 0 0 I 2.0 0.0 0:03.25 kworker/u16:14- 408 root 20 0 0 0 0 I 2.0 0.0 0:04.61 kworker/u16:15- 1592 root -2 0 0 0 0 S 1.7 0.0 0:10.26 qrtr_rx 262 root -3 0 0 0 0 S 1.3 0.0 0:06.29 kgsl_worker_thr 1673 root -81 0 195696 6204 5624 S 1.3 0.1 0:07.66 voxl-imu-server 1813 root 0 -20 0 0 0 I 1.3 0.0 0:04.09 kworker/u17:8-c 2179 root 20 0 0 0 0 I 1.0 0.0 0:04.12 kworker/u18:2-k 2211 root 20 0 0 0 0 I 1.0 0.0 0:03.58 kworker/u18:4-q 21475 root 0 -20 0 0 0 I 1.0 0.0 0:02.39 kworker/u17:0-c 30554 root 20 0 0 0 0 I 1.0 0.0 0:02.28 kworker/u18:6-k 48 root 20 0 0 0 0 S 0.7 0.0 0:02.29 rcuop/4 1183 logd 30 10 382204 3076 2504 S 0.7 0.0 0:03.33 logd 1279 root 20 0 97136 7620 6976 S 0.7 0.1 0:03.49 voxl-rangefinde 2146 root 20 0 0 0 0 I 0.7 0.0 0:02.28 kworker/u18:0-k 10 root 20 0 0 0 0 S 0.3 0.0 0:02.20 ksoftirqd/0 11 root 20 0 0 0 0 I 0.3 0.0 0:02.37 rcu_preempt 21 root 20 0 0 0 0 S 0.3 0.0 0:01.98 ksoftirqd/1 29 root 20 0 0 0 0 S 0.3 0.0 0:01.95 ksoftirqd/2 32 root 20 0 0 0 0 S 0.3 0.0 0:00.40 rcuop/2 37 root 20 0 0 0 0 S 0.3 0.0 0:02.04 ksoftirqd/3 56 root 20 0 0 0 0 S 0.3 0.0 0:01.03 rcuop/5 62 root 20 0 0 0 0 I 0.3 0.0 0:00.67 kworker/6:0-eve 64 root 20 0 0 0 0 S 0.3 0.0 0:01.59 rcuop/6 89 root 20 0 0 0 0 S 0.3 0.0 0:02.05 qmp_aop 301 root rt 0 0 0 0 S 0.3 0.0 0:01.34 spi3 1449 root 20 0 0 0 0 I 0.3 0.0 0:00.70 kworker/4:3-eve 1465 root 20 0 18868 1616 1472 S 0.3 0.0 0:02.68 voxl-cpu-monito 1472 root 20 0 6780 3660 3316 S 0.3 0.0 0:01.64 voxl-portalThe cpu load seems to be quite high, noticeably higher than before updating.
-
@Alex-Kushleyev @tom any response or advice?
-
@griffin , this seems like an network issue. So please try the following:
- open a terminal and just run the following command to "inspect" the hires stream so that it is always going and you can check if it's running any time :
voxl-inspect-cam hires_down_small_encoded - start a continuous ping test from your host that is trying to receive the video feed
- keep testing the video feed until you see the issue occur and check if ping is dropping out and double check
voxl-inspect-camoutput to make sure the data is flowing.
What i expect is the output of
voxl-inspect-camshould be continuously updating and you may see ping having trouble. Please let me know what you observe.Regarding the voxl2 heating up, i will check with the team. open-vins-server is running (VIO) and it has been recently updated to use GPU for feature detection, so a slight increase in voxl2 temperature may be possible.
Alex
- open a terminal and just run the following command to "inspect" the hires stream so that it is always going and you can check if it's running any time :
