sysimg 4.0.0 voxl-camera-server crashing tracking

Steve Turner

With a fresh install on one of our platforms of System Image 4.0.0, the camera server crashes and QVIO becomes unresponsive. The system is configured with Stereo, Tracking, and TOF.

voxl-camera-server:

Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:00:10 apq8096 bash[2543]: ------ voxl-camera-server: Camera server is now running
Nov 16 21:25:39 apq8096 bash[2543]: ERROR:   Recieved Fatal error from camera: tof
Nov 16 21:25:39 apq8096 bash[2543]: ERROR:   Sending request 397, ErrorCode: -ENODEV
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tracking result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tof request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo_child result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tof result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tracking request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo_child request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: ------ voxl-camera-server INFO: Camera server is now stopping
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Deleting buffers: 3 of 16 still in use
Nov 16 21:25:40 apq8096 bash[2543]: WARNING: Deleting buffers: 5 of 16 still in use
Nov 16 21:25:47 apq8096 bash[2543]: WARNING: Deleting buffers: 1179656 of 1179668 still in use
Nov 16 21:25:47 apq8096 bash[2543]: Segmentation fault:
Nov 16 21:25:47 apq8096 bash[2543]: Fault thread: voxl-camera-ser(tid: 2543)
Nov 16 21:25:47 apq8096 bash[2543]: Fault address: 0xa
Nov 16 21:25:47 apq8096 bash[2543]: Address not mapped.
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m

voxl-qvio-server:

-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Wed 2022-11-16 21:29:40 UTC. --
Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-qvio-server.
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: setting scheduler
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: INFO: set FIFO priority successfully!
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading config file
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: =================================================================
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: imu_name:                         imu0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_name:                         tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: lens_cal_path:                    /data/modalai/opencv_tracking_intrinsics.yml
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_extrinsics_name:              tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: use_second_cam_if_stereo:         0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: odr_hz:                           30.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: use_camera_height_bootstrap:      1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: camera_height_off_ground_m:        0.085
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_init_while_moving:         0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_imu_timeshift_s:              -0.002
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_imu_timeshift_s_uncertainty:   0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_cam_wrt_imu_uncertainty:         0.001  0.001  0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: R_cam_to_imu_uncertainty:          0.005  0.005  0.005
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: accl_fsr_ms2:                     156.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gyro_fsr_rad:                     34.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: accl_noise_std_dev:                0.316
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gyro_noise_std_dev:                0.010
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_noise_std_dev:                100.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: min_std_pixel_noise:               0.500
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: fail_high_pixel_noise_points:      1.665
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: limited_imu_bw_trigger:           25.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gps_imu_time_alignment_s:          0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_gps_wrt_imu:                     0.000  0.000  0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_mapping:                   1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_gps_vel:                   0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: show_extra_points_on_overlay;     0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto-reset params:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: en_auto_reset:                    1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_velocity:          10.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov_instant:      0.100
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov:              0.010
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov_timeout_s:    0.500
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_min_features:          3
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_min_feature_timeout_s:  1.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: =================================================================
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading extrinsics config file
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: #0:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: parent:                imu0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: child:                 tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_child_wrt_parent:    0.109  -0.012   0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: RPY_parent_to_child: 180.0    45.0    90.0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: R_child_to_parent:     0.000  -0.707   0.707
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: -1.000  -0.000  -0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: 0.000  -0.707  -0.707
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: tbc:   0.109 -0.012  0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: ombc:  2.504 -2.504  1.037 (axis angle)
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: ombc: 180.000 45.000 90.000 (RPY deg)
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading /data/modalai/opencv_tracking_intrinsics.yml
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: using fisheye camera intrinsics:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: distortion: 0.008810 -0.003841 0.000000 0.000000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: focal lengths: 269.295927 268.516850
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: principle points: 340.767010 221.351666
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: WARNING failed to init in time, trying again
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                FIT:FeatureName=>SensorFusion, TSBDrift=>0.00059 (0.004)
                                                FIT:FeatureName=>SensorFusion, TSBDrift0=>-0.000 (0.003)
                                                FIT:FeatureName=>SensorFusion, TSBDrift1=>0.000 (0.003)
                                                FIT:FeatureName=>SensorFusion, TSBDrift2=>-0.000 (0.002)
                                                FIT:FeatureName=>SensorFusion, RSBDrift=>0.00062
                                                FIT:FeatureName=>SensorFusion, AccelBias0=>-0.00278
                                                FIT:FeatureName=>SensorFusion, AccelBias1=>-0.00121
                                                FIT:FeatureName=>SensorFusion, AccelBias2=>0.12533
                                                FIT:FeatureName=>SensorFusion, GyroBias0=>-0.00470
                                                FIT:FeatureName=>SensorFusion, GyroBias1=>-0.00426
                                                FIT:FeatureName=>SensorFusion, GyroBias2=>-0.00017
                                                FIT:FeatureName=>SensorFusion, GravityBias0=>0.30783
                                                FIT:FeatureName=>SensorFusion, GravityBias1=>0.16675
                                                FIT:FeatureName=>SensorFusion, GravityBias2=>-9.78417
                                                FIT:FeatureName=>SensorFusion, FrameCount=>7
                                                FIT:FeatureName=>SensorFusion, ImuCount=>304
                                                FIT:FeatureName=>SensorFusion, AvgTimeFrameUpdate=>5.84ms
                                                FIT:FeatureName=>SensorFusion, NumResets=>0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                TrckLnDst:6 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                Distribution of num of consec keyframes w/ GN failing all features:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                Distribution of num of consec frames w/ increasing depth uncertainty:2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: waiting for imu
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: waiting for cam
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: connected to imu server
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: connected to camera server
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                FIT:FeatureName=>SensorFusion, TSBDrift=>0.76085 (0.011)
                                                FIT:FeatureName=>SensorFusion, TSBDrift0=>0.596 (0.008)
                                                FIT:FeatureName=>SensorFusion, TSBDrift1=>-0.458 (0.006)
                                                FIT:FeatureName=>SensorFusion, TSBDrift2=>-0.114 (0.005)
                                                FIT:FeatureName=>SensorFusion, RSBDrift=>0.30760
                                                FIT:FeatureName=>SensorFusion, AccelBias0=>0.12854
                                                FIT:FeatureName=>SensorFusion, AccelBias1=>0.19535
                                                FIT:FeatureName=>SensorFusion, AccelBias2=>-0.10591
                                                FIT:FeatureName=>SensorFusion, GyroBias0=>-0.00679
                                                FIT:FeatureName=>SensorFusion, GyroBias1=>-0.00544
                                                FIT:FeatureName=>SensorFusion, GyroBias2=>-0.00180
                                                FIT:FeatureName=>SensorFusion, GravityBias0=>0.49261
                                                FIT:FeatureName=>SensorFusion, GravityBias1=>0.21229
                                                FIT:FeatureName=>SensorFusion, GravityBias2=>-9.78607
                                                FIT:FeatureName=>SensorFusion, FrameCount=>2375
                                                FIT:FeatureName=>SensorFusion, ImuCount=>180880
                                                FIT:FeatureName=>SensorFusion, AvgTimeFrameUpdate=>178.39ms
                                                FIT:FeatureName=>SensorFusion, NumResets=>1
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                TrckLnDst:182 46 36 69 45 37 32 43 20 22 21 16 16 11 8 11 4 8 7 2 4 2 6 4 2 2 5 3 2 2 3 0 2 3 4 1 1 1 4 1 2 1 0 1 5 1 5 1 0 3 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 1 2 0 1 2 1 1 0 0 1 0 0 0 2 0 0 0 2 0 1 0 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 108
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                Distribution of num of consec keyframes w/ GN failing all features:42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
                                                Distribution of num of consec frames w/ increasing depth uncertainty:2354 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: WARNING: disconnected from camera server, resetting VIO
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: client "voxl-portal0" connected to overlay

Here are our versions etc.

Service Name         |  Version  |  Enabled  |   Running   |  CPU Usage
---------------------------------------------------------------------------
 docker-autorun       |    1.2.4  | Disabled  | Not Running |
 docker-daemon        |    1.2.4  | Disabled  | Not Running |
 modallink-relink     |   0.16.1  | Disabled  | Not Running |
 voxl-camera-server   |    1.3.5  |  Enabled  |   Running   |    12.7
 voxl-cpu-monitor     |    0.3.0  |  Enabled  |   Running   |     0.0
 voxl-dfs-server      |    0.3.1  | Disabled  | Not Running |
 voxl-imu-server      |    1.0.3  |  Enabled  |   Running   |     0.0
 voxl-mavlink-server  |    0.2.0  |  Enabled  |   Running   |     1.8
 voxl-modem           |   0.16.1  | Disabled  | Not Running |
 voxl-portal          |    0.4.2  |  Enabled  |   Running   |     0.0
 voxl-qvio-server     |    0.8.2  |  Enabled  |   Running   |     9.1
 voxl-streamer        |    0.4.1  | Disabled  | Not Running |
 voxl-tag-detector    |    0.0.4  | Disabled  | Not Running |
 voxl-tflite-server   |    0.3.1  | Disabled  | Not Running |
 voxl-time-sync       |    1.2.2  | Disabled  | Not Running |
 voxl-vision-px4      |    1.4.0  |  Enabled  |   Running   |     0.0
 voxl-wait-for-fs     |    1.2.2  |  Enabled  |  Completed  |

This happens with or without the portal. Other units seem OK and this unit was OK in a different environment. Trying to isolate the issue.

Chad Sweet

camera server errors are very likely hardware related. usually a sensor is not fully plugged in

Steve Turner

Resurrecting this topic. We have two drones with identical hardware builds.
Only camera's plugged in are tracking cameras to J4.
Both are running the latest system image.

Drone 1's camera server never crashes while flying Indoors/outdoors.
Drone 2's camera server sometimes crashes and seems to be triggered by exposure changes when outside in bright environments. It is running a slightly newer version of voxl-camera-server.

The failing drone has this in the logs from logcat | grep "camera":

voxl:~$ logcat | grep camera


01-26 20:06:14.004  5732  5732 V HAL     : loaded HAL id=camera path=/usr/lib/hw/camera.msm8996.so hmi=(nil) handle=0x64450
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> get_num_of_cameras: 2767: dev_info[id=0,name='video2']
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> sort_camera_info: 2611: Camera id: 0 facing: 1, type: 1 is_yuv: 0
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> sort_camera_info: 2617: Number of cameras 1 sorted 1
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> get_num_of_cameras: 2785: num_cameras=1
01-26 20:06:14.006  5732  5732 I QCamera : <HAL><INFO> QCamera2Factory: 142: 1 camera devices detected!
01-26 20:06:14.107  5732  5732 E QCamera : <HAL><ERROR> initStaticMetadata: 9846: static int qcamera::QCamera3HardwareInterface::initStaticMetadata(uint32_t)@9846: from gCamCapability QCAMERA3_EXP_TIME_RANGE= (19425, 1020461640)
01-26 20:06:14.107  5732  5732 E QCamera : <HAL><ERROR> initStaticMetadata: 9877: static int qcamera::QCamera3HardwareInterface::initStaticMetadata(uint32_t)@9877: cameraId 0 raw_dim[0].w 640 raw_dim[0].h 480
01-26 20:06:14.107  5732  5732 I QCamera : <HAL><INFO> getCamInfo: 10272: camera 0 resource cost is 100
01-26 20:06:14.107  5732  5732 I QCamera : <HAL><INFO> cameraDeviceOpen: 414: Open camera id 0 API version 768
01-26 20:06:14.117  5732  5732 I QCamera : <HAL><INFO> openCamera: 851: [KPI Perf]: E PROFILE_OPEN_CAMERA camera id 0
01-26 20:06:14.197  5732  5732 I QCamera : <HAL><INFO> openCamera: 861: [KPI Perf]: X PROFILE_OPEN_CAMERA camera id 0, rc: 0
01-26 20:06:14.197  5732  5732 E QCamera : <HAL><ERROR> initParameters: 8374: int qcamera::QCamera3HardwareInterface::initParameters()@8374 HAL params are initialized


01-26 20:07:55.283  5732  5789 E QCamera : <MCI><ERROR> mm_camera_event_notify: 192: Camera Event DAEMON DIED received
01-26 20:07:55.284  5732  5788 E QCamera : <HAL><ERROR> camEvtHandle: 803: Fatal, camera daemon died

journalctl on the drone with crashed voxl-camera-server shows:

Jan 01 00:00:52 apq8096 systemd[1]: Started voxl-camera-server.
Jan 26 20:01:17 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:23 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:24 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:35 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:37 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:42 apq8096 bash[2966]: ERROR:   Recieved Fatal error from camera: tracking
Jan 26 20:01:42 apq8096 bash[2966]: ERROR:   Sending request 7188, ErrorCode: -ENODEV
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Thread: tracking request thread recieved ESTOP
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Thread: tracking result thread recieved ESTOP
Jan 26 20:01:42 apq8096 bash[2966]: ------ voxl-camera-server INFO: Camera server is now stopping
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Deleting buffers: 24 of 32 still in use
Jan 26 20:01:42 apq8096 bash[2966]: Segmentation fault:
Jan 26 20:01:42 apq8096 bash[2966]: Fault thread: voxl-camera-ser(tid: 2966)
Jan 26 20:01:42 apq8096 bash[2966]: Fault address: 0x8
Jan 26 20:01:42 apq8096 bash[2966]: Address not mapped.
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m

They have not been crashed... and the failure happens regularly when in bright environments. I matched up software versions as well and the crash still happens, so it has to be some kind of hardware issue and the below can be disregarded.

Hardware problems again with expansion or something weird in this specific camera/cable?

Drone 1 package versions (a bit older/stable repo):

libvoxl-cci-direct - 0.1.5
libvoxl-cutils - 0.1.1
voxl-camera-calibration - 0.2.3
voxl-camera-server - 1.3.5
voxl-cpu-monitor - 0.3.0
voxl-docker-support - 1.2.4
voxl-gphoto2-server - 0.0.10
voxl-jpeg-turbo - 2.1.3-4
voxl-libgphoto2 - 0.0.4
voxl-libuvc - 1.0.7
voxl-logger - 0.3.4
voxl-mavlink - 0.1.0
voxl-mavlink-server - 0.2.0
voxl-modem - 0.16.1
voxl-mongoose - 7.7.0-1
voxl-mpa-to-ros - 0.3.6
voxl-mpa-tools - 0.7.6
voxl-opencv - 4.5.5-1
voxl-portal - 0.4.2
voxl-qvio-server - 0.9.0
voxl-streamer - 0.4.1
voxl-suite - 0.9.4
voxl-system-image - 4.0-r0
voxl-tag-detector - 0.0.4
voxl-tflite-server - 0.3.1
voxl-utils - 1.2.2
voxl-uvc-server - 0.1.3
voxl-vision-px4 - 1.4.0
voxl-vpn - 0.0.6

Drone 2 package versions (a bit newer, dev repos):

libvoxl-cci-direct - 0.1.5
libvoxl-cutils - 0.1.1
voxl-camera-calibration - 0.3.0
voxl-camera-server - 1.3.7
voxl-cpu-monitor - 0.4.1
voxl-docker-support - 1.2.5
voxl-gphoto2-server - 0.0.10
voxl-jpeg-turbo - 2.1.3-4
voxl-lepton-server - 1.0.1
voxl-libgphoto2 - 0.0.4
voxl-libuvc - 1.0.7
voxl-logger - 0.3.4
voxl-mavlink - 0.1.0
voxl-mavlink-server - 0.2.0
voxl-modem - 0.16.2
voxl-mongoose - 7.7.0-1
voxl-mpa-to-ros - 0.3.6
voxl-mpa-tools - 0.8.4
voxl-opencv - 4.5.5-1
voxl-portal - 0.4.3
voxl-qvio-server - 0.9.0
voxl-streamer - 0.4.3
voxl-suite - 1.0.0
voxl-system-image - 4.0-r0
voxl-tag-detector - 0.0.4
voxl-tflite-server - 0.3.1
voxl-utils - 1.2.3
voxl-uvc-server - 0.1.3
voxl-vision-px4 - 1.4.1
voxl-vpn - 0.0.6

Chad Sweet

These are usually hardware issues. Some possibilities

crimped flex cable
exposed copper on the flex cable
loose or damaged connector

ModalAI Forum

sysimg 4.0.0 voxl-camera-server crashing tracking