how to get H256 hires encoded imgs from sdk 1.0 ?
-
Hello Team,
I'm trying to get encoded hires images using sdk 1.0 and camera_server v1.6.2 from a voxl2 board with configuration 6 (TOF, HIRES and Tracking). The following is the output fromvoxl-versionvoxl2:~$ voxl-version -------------------------------------------------------------------------------- system-image: 1.6.2-M0054-14.1a-perf kernel: #1 SMP PREEMPT Fri May 19 22:19:33 UTC 2023 4.19.125 -------------------------------------------------------------------------------- hw version: M0054 -------------------------------------------------------------------------------- voxl-suite: 1.0.0 -------------------------------------------------------------------------------- Packages: Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.0/binary-arm64/ Last Updated: 2023-11-12 06:55:14 List: libmodal-cv 0.3.2 libmodal-exposure 0.0.8 libmodal-journal 0.2.2 libmodal-json 0.4.3 libmodal-pipe 2.8.5 libqrb5165-io 0.2.0 libvoxl-cci-direct 0.1.5 libvoxl-cutils 0.1.1 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-dfs-server 0.1.0 qrb5165-imu-server 0.6.0 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.2.2 qrb5165-tflite 2.8.0-2 voxl-bind-spektrum 0.1.0 voxl-camera-calibration 0.4.0 voxl-camera-server 1.6.2 voxl-configurator 0.2.7 voxl-cpu-monitor 0.4.6 voxl-docker-support 1.2.5 voxl-eigen3 3.4.0 voxl-elrs 0.0.7 voxl-esc 1.2.2 voxl-feature-tracker 0.2.3 voxl-flow-server 0.3.3 voxl-gphoto2-server 0.0.10 voxl-jpeg-turbo 2.1.3 voxl-lepton-server 1.1.2 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.3.4 voxl-mavcam-manager 0.5.1 voxl-mavlink 0.1.1 voxl-mavlink-server 1.2.0 voxl-modem 1.0.5 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.6 voxl-mpa-tools 1.0.4 voxl-opencv 4.5.5-1 voxl-portal 0.5.9 voxl-px4 1.14.0-2.0.34 voxl-px4-imu-server 0.1.2 voxl-px4-params 0.1.8 voxl-qvio-server 0.9.3 voxl-remote-id 0.0.8 voxl-slpi-uart-bridge 1.0.1 voxl-streamer 0.7.1 voxl-suite 1.0.0 voxl-tag-detector 0.0.4 voxl-tflite-server 0.3.1 voxl-utils 1.3.1 voxl-uvc-server 0.1.6 voxl-vision-hub 1.6.6 voxl2-system-image 1.6.2-r0 voxl2-wlan 1.0-r0 --------------------------------------------------------------------------------I was able to get the camera-server to work with configuration 6 and the pipes are showing up in
voxl-inspect-cam -aas shown below| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | hires_large_color |13271040 | 4096 | 2160 | 33.23 | 431 | 489 | 77.1 | 30.0 | 3190.0 | NV12 | hires_large_encoded | 26549 | 4096 | 2160 | 33.23 | 431 | 489 | 89.3 | 30.0 | 7.2 | H265 (P) | hires_large_grey | 8847360 | 4096 | 2160 | 33.23 | 431 | 489 | 71.8 | 30.0 | 2126.6 | RAW8 | hires_small_color | 1179648 | 1024 | 768 | 33.23 | 431 | 490 | 60.6 | 30.0 | 283.5 | NV12 | hires_small_encoded | 1528 | 1024 | 768 | 33.23 | 431 | 490 | 63.0 | 30.0 | 0.5 | H265 (P) | hires_small_grey | 786432 | 1024 | 768 | 33.23 | 431 | 490 | 59.4 | 30.0 | 189.0 | RAW8 | hires_snapshot | | tof_conf | 38528 | 224 | 172 | 0.00 | 0 | 1669 | 29.6 | 5.0 | 1.5 | RAW8 | tof_depth | 38528 | 224 | 172 | 0.00 | 0 | 1669 | 29.3 | 5.0 | 1.5 | RAW8 | tof_ir | 38528 | 224 | 172 | 0.00 | 0 | 1669 | 29.0 | 5.0 | 1.5 | RAW8 | tracking | 307200 | 640 | 480 | 5.61 | 303 | 10016 | 11.6 | 30.0 | 73.7 | RAW8However, I'm not able to see the output of the encoded hires neither in
voxl-portalnot when I subscribe to the encoded pipeline in the published topics fromvoxl-mpa-to-ros. In fact, when I try to subscribe to any encoded image topic, the node crashes immediately as shown below.voxl2:~$ roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch ... logging to /home/root/.ros/log/c26582ac-8191-11ee-99cc-9c5322cc8a09/roslaunch-m0054-8630.log Checking log directory for disk usage. This may take a while. Press Ctrl-C to interrupt Done checking log file disk usage. Usage is <1GB. started roslaunch server http://m0054:43361/ SUMMARY ======== PARAMETERS * /rosdistro: melodic * /rosversion: 1.14.13 NODES / voxl_mpa_to_ros_node (voxl_mpa_to_ros/voxl_mpa_to_ros_node) auto-starting new master process[master]: started with pid [8644] ROS_MASTER_URI=http://localhost:11311 setting /run_id to c26582ac-8191-11ee-99cc-9c5322cc8a09 process[rosout-1]: started with pid [8655] started core service [/rosout] process[voxl_mpa_to_ros_node-2]: started with pid [8658] MPA to ROS app is now running Found new interface: hires_large_color Found new interface: hires_large_encoded Found new interface: hires_large_grey Found new interface: hires_small_color Found new interface: hires_small_encoded Found new interface: hires_small_grey Found new interface: hires_snapshot Found new interface: tof_conf Found new interface: tof_depth Found new interface: tof_ir Found new interface: tracking Found new interface: imu_apps Found new interface: tof_pc Found new interface: voa_pc_out Interface tof_depth now publishing Interface hires_small_grey now publishing Interface tof_depth ceasing to publish Interface hires_large_encoded now publishing Interface hires_small_grey ceasing to publish terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc [voxl_mpa_to_ros_node-2] process has died [pid 8658, exit code -6, cmd /opt/ros/melodic/lib/voxl_mpa_to_ros/voxl_mpa_to_ros_node __name:=voxl_mpa_to_ros_node __log:=/home/root/.ros/log/c26582ac-8191-11ee-99cc-9c5322cc8a09/voxl_mpa_to_ros_node-2.log]. log file: /home/root/.ros/log/c26582ac-8191-11ee-99cc-9c5322cc8a09/voxl_mpa_to_ros_node-2*.logIs there any possible solution for this issue? Please help!
-
@aashry the following is the
voxl-camera-server.conffile as well for your reference. Should I change anything here to fix that issue? Please elaborate./** * voxl-camera-server Configuration File * * Each camera has configurations for up to 4 HAL3 streams: * - `preview` stream for raw unprocessed images from CV cameras * - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming * - `large_video` 4k (ish) h264/h265 for onboard video recording to disk * - `snapshot` ISP-processed JPG snapshots that get saved to disk * * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled * * This file is generated from default values by voxl-configure-cameras. * Do not expect arbitrary resolutions to work, the ISP and video compression * pipelines only support very specific resolutions. * * The default video compression mode is cqp or Constant Quantization Parameter * * * */ { "version": 0.1, "cameras": [{ "type": "pmd-tof", "name": "tof", "enabled": true, "camera_id": 0, "fps": 5, "en_preview": true, "preview_width": 224, "preview_height": 1557, "pre_format": "tof", "ae_mode": "off", "standby_enabled": false, "decimator": 5 }, { "type": "imx214", "name": "hires", "enabled": true, "camera_id": 2, "fps": 30, "en_preview": false, "preview_width": 640, "preview_height": 480, "pre_format": "nv21", "en_small_video": true, "small_video_width": 1024, "small_video_height": 768, "small_venc_mode": "h265", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "en_large_video": true, "large_video_width": 4096, "large_video_height": 2160, "large_venc_mode": "h265", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 38, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 30, "en_snapshot": true, "en_snapshot_width": 4160, "en_snapshot_height": 3120, "ae_mode": "isp" }, { "type": "ov7251", "name": "tracking", "enabled": true, "camera_id": 1, "fps": 30, "en_preview": true, "preview_width": 640, "preview_height": 480, "pre_format": "raw8", "ae_mode": "lme_msv", "ae_desired_msv": 60, "ae_filter_alpha": 0.600000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 1 }] } -
Also, it seems like
voxl-camera-serverrandomly stops working. Is SDK 1.0 completely unusable so far? or Is there a workaround that we can do?| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps t | hires_large_color | Server Disconnected | hires_large_encoded | Server DisconnectedMost likely the server stopped without cleaning up Client is cleaning up pipes for the serverdMost likely the server stopped without cleaning up ERROR in pipe_client_init_channel opening request pipe: No such device or addressing request pipe: Ns | hires_small_encoded | Server DisconnectedERROR in pipe_client_init_channel opening request pipe: Ns | hires_small_grey | Server DisconnectedClient is cleaning up pipes for the server ERROR in pipe_client_init_channel opening request pipe: No such device or address cleaning up Most likely the server stopped without cleaning up Client is cleaning up pipes for the serverdMost likely the server stopped without cleaning up ERROR in pipe_client_init_channel opening request pipe: No such device or address Most likely the server stopped without cleaning upn pipe_client_init_channel opening request pipe: Ns Client is cleaning up pipes for the serveraning up ERROR in pipe_client_init_channel opening request pipe: No such device or address Most likely the server stopped without cleaning up Client is cleaning up pipes for the server ERROR in pipe_client_init_channel opening request pipe: No such device or address Most likely the server stopped without cleaning up Client is cleaning up pipes for the server ERROR in pipe_client_init_channel opening request pipe: No such device or address Most likely the server stopped without cleaning up Client is cleaning up pipes for the server ERROR in pipe_client_init_channel opening request pipe: No such device or address Most likely the server stopped without cleaning up Client is cleaning up pipes for the server -
Edit1: Everything above applies to the
h264encoding as well, I haven't been able to view it in the portal and it has the same problem with the ros node. -
Anyone can help?
-
@aashry What UAV are you flying? Is this an off the shelf one from modalai or did you assemble it yourself?
Usually that error means there is a hardware issue with the cameras
-
Hi @aashry, did you actually solve this problem?
I am running into the same issue.Camera config is alright, stream config seems to be right as well, but I cannot get an encoded stream in either voxl-portal or vlc.
Any help would be greatly appreciated.
BR
-
Hi @aashry voxl-camera-server with H265 support was implemented and included in an SDK past 1.0 - any reason why you are not upgrading to 1.3.3?
The other option is you mix and match versions of voxl-camera-server with your current SDK version, but odds are that will have quite a bit of conflicts as it is dependent on new versions of modalpipe and such.
Zach
-
Also @Bernhard - you will never be able to view the encoded stream on voxl-portal, just on VLC or something like QGC. SDK 1.0 does support h264 which is software encoding though so innately it will be slower - does this work out the gate for you?
-
@Bernhard , you may be missing a step, if you want to stream the encoded stream using RTSP protocol, please take a look at
voxl-streamer, which is our standard package : https://docs.modalai.com/voxl-streamer/voxl-streamer can perform the following functions:
- receive unencoded yuv (or raw8 monochrome) images from camera server and encode them using VOXL2's hardware encoder (using gstreamer + OMX api) and serve the encoded video as rtsp service
- receive encoded video stream from camera server and serve the encoded video directly
To play the h264 / h265 stream you can use VLC as Zach mentioned, or ffmpeg / ffplay:
ffplay rtsp://<board-ip>:port/liveport number is printed when you run
voxl-streamer(port can be modified with a command line argument)Alex
-
@Alex-Kushleyev I have tried using VLC as well, that does not work.
voxl-streamer is configured correctly (I think):"input-pipe": "hires_small_encoded", "bitrate": 1500000, "rotation": 0, "decimator": 1, "port": 8900But when I try to connect via rtsp://<IP>:8900/live nothing happens.
-
@Bernhard , please follow these steps:
- make sure
voxl-camera-serveris running either in the background service or manually started in a terminal - confirm that
hires_small_encodedstream is being generated by the camera server:voxl-inspect-cam hires_small_encodedshould show stream statistics like FPS, frame resolution and size- if you do not see any stats using the inspect-cam tool, make sure the
small encodedstream is enabled in your/etc/modalai/voxl-camera-server.conf
- assuming the encoded stream is working, launch
voxl-streamer:voxl-streamer -i hires_small_encoded- it should also tell you the URL to connect to, although often the ip in the url is local (
127.0.0.1) so you need to replace that with actual network IP address
Alex
- make sure
-
@Alex-Kushleyev thanks - I think the voxl-streamer was enabled, but not running.
Problem solved, thank you!
