Intermittent crashing of voxl-camera-server during VOA testing
-
We have calibrated our stereo cameras per these instructions: https://docs.modalai.com/calibrate-cameras/#calibrate-stereo-cameras.
We are using the dev version of the voxl-vision-px4 (that fixed a previous bug) and which also tightened the range of accurate obstacle detections from 0.2 -> 20 meters to 0.5 -> 8 meters. In our experience, detections within this range are still quite noisy (sometimes swinging 2-3 meters in tolerance), but they do serve as coarse-grained estimates. In our VOXL, we have enabled the voxl-camera-server and voxl-dfs-server. We confirm this is working by looking at the OBSTACLE_DISTANCE field in the MAVLink Inspector in QGC.
In QGC -> Vehicle Setup -> Safety -> Collision Avoidance, we have enabled Obstacle Prevention. We have also set the MPC_COL_PREV_D distance to 6.0 meters. We have taken the drone outside, gotten GPS sync, turned on Position mode, and manually attempted to fly it into a large tree with dense foliage. As advertised, the VOA will kick in and prevent us from flying into the tree. (This is great!)
During this testing, when we fly the drone towards the tree, we see a spike in the CPU usage of the voxl-dfs-server. (We are not surprised by this, but share it as a data point along with our belief that this service is operating as expected and uses more CPU to compute points as it approaches obstacles.)
Sometimes, when we fly the drone towards the tree, the voxl-camera-server will crash. One time, we saw the voxl-dfs-server crash as well. The crash message from the voxl-camera-server contains an ESTOP message on a single camera stream, although the exact camera stream varies (e.g., hires, tracking, or stereo). When one of these services crash, QGC oscillates between Position Mode and Hold Mode. On our controller, we lose the ability to pitch and roll. We can yaw, ascend and descend, but nothing more. Without being able to fly in any direction, we eventually land.
We suspect that when the voxl-camera-server crashes, the voxl-dfs-server is unable to get data and does not function. We do not know if this fact is communicated to QGC and is the mechanism that triggers the Hold mode.
We believe this intermittent crashing is likely due to some bug (perhaps due in some part to the interaction between the voxl-camera-server and voxl-dfs-server in the midst of a VOA event). Please advise if this is likely the case and how we might handle this.
-
Can you paste the log from
journalctl -u voxl-camera-serverhere so that we can see exactly what it's saying. Camera server doesn't know or care about what its clients do with the data it provides, it only knows if someone is listening for each camera. These camera server crashes are almost always caused by hardware issues with regards to the camera cables, though it is unusual that the errors you're seeing are coming from different cameras each time. You can also run camera server manually with debug flagsvoxl-camera-server -d 0to have it print additional outputs about every single thing that it does. If you can show me a log either from journalctl or a manually run camera server instance that'd be very helpful. -
Hi Jared,
Quick follow up to our call earlier, I've published the latest version of camera server(0.7.5) to stable, this has the increased pipe sizes for the hires camera and will print out the available resolutions for each camera when run with
voxl-camera-server -d 1(or -d 0, though this will be VERY verbose and will start printing each new exposure/gain). -
Awesome, thank you, @Alex-Gardner. We will try this when we are back in the office and let you know what we find.
-
@Alex-Gardner The available resolutions do print out accordingly, about the same as the previous list you send us. Thanks!
-
Thanks for printing out those resolutions. Very helpful!
yocto:~$ voxl-camera-server -d 1 exising instance of voxl-camera-server found, attempting to stop it ------ voxl-camera-server INFO: Port J2 ------ voxl-camera-server: Done configuring hires camera Port : port_J2 Name : hires Enabled : 1 Type : hires Api : hal3 P-W : 640 P-H : 480 P-Fmt : nv21 FPS : 30 OverrideId : -1 AEAlgo : isp ------ voxl-camera-server INFO: Port J3 ------ voxl-camera-server: Done configuring stereo camera Port : port_J3 Name : stereo Enabled : 1 Type : stereo Api : hal3 P-W : 1280 P-H : 480 P-Fmt : nv21 FPS : 30 OverrideId : -1 AEAlgo : modalai =================MODALAI Auto Exposure Settings================== gain_min: 0 gain_max: 1000 exposure_min_us: 100 exposure_max_us: 33000 desired_msv: 58.000000 k_p_ns: 32000.000000 k_i_ns: 20.000000 max_i: 250.000000 p_good_thresh: 3 exposure_period: 1 gain_period: 2 display_debug: no exposure_offset_for_gain_calc: 8000 ================================================================= ------ voxl-camera-server INFO: Port J4 ------ voxl-camera-server: Done configuring tracking camera Port : port_J4 Name : tracking Enabled : 1 Type : tracking Api : hal3 P-W : 640 P-H : 480 P-Fmt : raw8 FPS : 30 OverrideId : -1 AEAlgo : modalai =================MODALAI Auto Exposure Settings================== gain_min: 0 gain_max: 1000 exposure_min_us: 100 exposure_max_us: 33000 desired_msv: 58.000000 k_p_ns: 32000.000000 k_i_ns: 20.000000 max_i: 250.000000 p_good_thresh: 3 exposure_period: 1 gain_period: 2 display_debug: no exposure_offset_for_gain_calc: 8000 ================================================================= Writing new configuration to /etc/modalai/voxl-camera-server.conf Created pipe: hires_preview channel: 0 Created pipe: stereo channel: 1 Created pipe: tracking channel: 2 Starting Camera: tracking SUCCESS: Camera module opened ----------- Number of cameras: 3 ------ voxl-camera-server INFO: Client: qvio-server0 connected to channel: 2 ------ voxl-camera-server ERROR: Client connected to invalid channel: 0 ------ voxl-camera-server ERROR: Client connected to invalid channel: 1 ----------- HiRes camera id: 0 ----------- Stereo camera id: 2 ----------- Mono camera id: 1 ------ voxl-camera-server INFO: Client: voxl-streamer0 connected to channel: 2 Available resolutions for camera: tracking: 640 x 480 320 x 240 Resolution: 640 x 480 Found! ------ voxl-camera-server: Preview buffer allocations Internal Exposure for: tracking 2 /builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface Creating ModalAI exposure interface Port : port_J4 Name : tracking Enabled : 1 Type : tracking Api : hal3 P-W : 640 P-H : 480 P-Fmt : raw8 FPS : 30 OverrideId : -1 AEAlgo : modalai =================MODALAI Auto Exposure Settings================== gain_min: 0 gain_max: 1000 exposure_min_us: 100 exposure_max_us: 33000 desired_msv: 58.000000 k_p_ns: 32000.000000 k_i_ns: 20.000000 max_i: 250.000000 p_good_thresh: 3 exposure_period: 1 gain_period: 2 display_debug: no exposure_offset_for_gain_calc: 8000 ================================================================= Starting Camera: hires Available resolutions for camera: hires: 4160 x 3120 4056 x 3040 4000 x 3000 3040 x 3040 3016 x 3016 3840 x 2160 3648 x 2736 3264 x 2448 3200 x 2400 2976 x 2976 3044 x 1720 2704 x 2028 2704 x 1520 2592 x 1944 2688 x 1512 2028 x 1144 2160 x 2160 1920 x 2160 1920 x 1920 1880 x 1880 2048 x 1536 1920 x 1440 1920 x 1080 1600 x 1600 1600 x 1200 1520 x 1520 1440 x 1080 1080 x 1080 1280 x 960 1340 x 760 1280 x 768 1280 x 720 1200 x 1200 1280 x 640 1280 x 480 1040 x 780 1024 x 768 960 x 960 720 x 720 800 x 600 960 x 720 848 x 480 858 x 480 864 x 480 800 x 480 720 x 480 640 x 480 640 x 240 640 x 360 480 x 640 480 x 480 480 x 360 480 x 320 432 x 240 352 x 288 320 x 240 Resolution: 640 x 480 Found! ------ voxl-camera-server: Preview buffer allocations Port : port_J2 Name : hires Enabled : 1 Type : hires Api : hal3 P-W : 640 P-H : 480 P-Fmt : nv21 FPS : 30 OverrideId : -1 AEAlgo : isp Starting Camera: stereo Available resolutions for camera: stereo: 1280 x 480 848 x 480 858 x 480 864 x 480 800 x 480 720 x 480 640 x 480 640 x 240 640 x 360 480 x 480 480 x 360 480 x 320 432 x 240 352 x 288 320 x 240 Resolution: 1280 x 480 Found! ------ voxl-camera-server: Preview buffer allocations Internal Exposure for: stereo 1 /builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface Creating ModalAI exposure interface Received raw10 frame, checking to see if is actually raw8 Frame was actually 8 bit, sending as is Port : port_J3 Name : stereo Enabled : 1 Type : stereo Api : hal3 P-W : 1280 P-H : 480 P-Fmt : nv21 FPS : 30 OverrideId : -1 AEAlgo : modalai =================MODALAI Auto Exposure Settings================== gain_min: 0 gain_max: 1000 exposure_min_us: 100 exposure_max_us: 33000 desired_msv: 58.000000 k_p_ns: 32000.000000 k_i_ns: 20.000000 max_i: 250.000000 p_good_thresh: 3 exposure_period: 1 gain_period: 2 display_debug: no exposure_offset_for_gain_calc: 8000 ================================================================= ------ voxl-camera-server: Camera server is now running
