With a fresh install on one of our platforms of System Image 4.0.0, the camera server crashes and QVIO becomes unresponsive. The system is configured with Stereo, Tracking, and TOF.
voxl-camera-server
:
Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:00:10 apq8096 bash[2543]: ------ voxl-camera-server: Camera server is now running
Nov 16 21:25:39 apq8096 bash[2543]: ERROR: Recieved Fatal error from camera: tof
Nov 16 21:25:39 apq8096 bash[2543]: ERROR: Sending request 397, ErrorCode: -ENODEV
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tracking result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tof request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo_child result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tof result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo result thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: tracking request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Thread: stereo_child request thread recieved ESTOP
Nov 16 21:25:39 apq8096 bash[2543]: ------ voxl-camera-server INFO: Camera server is now stopping
Nov 16 21:25:39 apq8096 bash[2543]: WARNING: Deleting buffers: 3 of 16 still in use
Nov 16 21:25:40 apq8096 bash[2543]: WARNING: Deleting buffers: 5 of 16 still in use
Nov 16 21:25:47 apq8096 bash[2543]: WARNING: Deleting buffers: 1179656 of 1179668 still in use
Nov 16 21:25:47 apq8096 bash[2543]: Segmentation fault:
Nov 16 21:25:47 apq8096 bash[2543]: Fault thread: voxl-camera-ser(tid: 2543)
Nov 16 21:25:47 apq8096 bash[2543]: Fault address: 0xa
Nov 16 21:25:47 apq8096 bash[2543]: Address not mapped.
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Nov 16 21:25:49 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m
voxl-qvio-server
:
-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Wed 2022-11-16 21:29:40 UTC. --
Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-qvio-server.
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: setting scheduler
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: INFO: set FIFO priority successfully!
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading config file
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: =================================================================
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: imu_name: imu0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_name: tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: lens_cal_path: /data/modalai/opencv_tracking_intrinsics.yml
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_extrinsics_name: tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: use_second_cam_if_stereo: 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: odr_hz: 30.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: use_camera_height_bootstrap: 1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: camera_height_off_ground_m: 0.085
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_init_while_moving: 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_imu_timeshift_s: -0.002
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_imu_timeshift_s_uncertainty: 0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_cam_wrt_imu_uncertainty: 0.001 0.001 0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: R_cam_to_imu_uncertainty: 0.005 0.005 0.005
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: accl_fsr_ms2: 156.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gyro_fsr_rad: 34.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: accl_noise_std_dev: 0.316
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gyro_noise_std_dev: 0.010
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: cam_noise_std_dev: 100.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: min_std_pixel_noise: 0.500
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: fail_high_pixel_noise_points: 1.665
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: limited_imu_bw_trigger: 25.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: gps_imu_time_alignment_s: 0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_gps_wrt_imu: 0.000 0.000 0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_mapping: 1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: enable_gps_vel: 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: show_extra_points_on_overlay; 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto-reset params:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: en_auto_reset: 1
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_velocity: 10.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov_instant: 0.100
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov: 0.010
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_max_v_cov_timeout_s: 0.500
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_min_features: 3
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: auto_reset_min_feature_timeout_s: 1.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: =================================================================
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading extrinsics config file
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: #0:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: parent: imu0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: child: tracking
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: T_child_wrt_parent: 0.109 -0.012 0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: RPY_parent_to_child: 180.0 45.0 90.0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: R_child_to_parent: 0.000 -0.707 0.707
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: -1.000 -0.000 -0.000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: 0.000 -0.707 -0.707
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: tbc: 0.109 -0.012 0.001
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: ombc: 2.504 -2.504 1.037 (axis angle)
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: ombc: 180.000 45.000 90.000 (RPY deg)
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: loading /data/modalai/opencv_tracking_intrinsics.yml
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: using fisheye camera intrinsics:
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: distortion: 0.008810 -0.003841 0.000000 0.000000
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: focal lengths: 269.295927 268.516850
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: principle points: 340.767010 221.351666
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: WARNING failed to init in time, trying again
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
FIT:FeatureName=>SensorFusion, TSBDrift=>0.00059 (0.004)
FIT:FeatureName=>SensorFusion, TSBDrift0=>-0.000 (0.003)
FIT:FeatureName=>SensorFusion, TSBDrift1=>0.000 (0.003)
FIT:FeatureName=>SensorFusion, TSBDrift2=>-0.000 (0.002)
FIT:FeatureName=>SensorFusion, RSBDrift=>0.00062
FIT:FeatureName=>SensorFusion, AccelBias0=>-0.00278
FIT:FeatureName=>SensorFusion, AccelBias1=>-0.00121
FIT:FeatureName=>SensorFusion, AccelBias2=>0.12533
FIT:FeatureName=>SensorFusion, GyroBias0=>-0.00470
FIT:FeatureName=>SensorFusion, GyroBias1=>-0.00426
FIT:FeatureName=>SensorFusion, GyroBias2=>-0.00017
FIT:FeatureName=>SensorFusion, GravityBias0=>0.30783
FIT:FeatureName=>SensorFusion, GravityBias1=>0.16675
FIT:FeatureName=>SensorFusion, GravityBias2=>-9.78417
FIT:FeatureName=>SensorFusion, FrameCount=>7
FIT:FeatureName=>SensorFusion, ImuCount=>304
FIT:FeatureName=>SensorFusion, AvgTimeFrameUpdate=>5.84ms
FIT:FeatureName=>SensorFusion, NumResets=>0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
TrckLnDst:6 4 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
Distribution of num of consec keyframes w/ GN failing all features:0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
Distribution of num of consec frames w/ increasing depth uncertainty:2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: waiting for imu
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: waiting for cam
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: connected to imu server
Jan 01 00:00:10 apq8096 voxl-qvio-server[2539]: connected to camera server
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
FIT:FeatureName=>SensorFusion, TSBDrift=>0.76085 (0.011)
FIT:FeatureName=>SensorFusion, TSBDrift0=>0.596 (0.008)
FIT:FeatureName=>SensorFusion, TSBDrift1=>-0.458 (0.006)
FIT:FeatureName=>SensorFusion, TSBDrift2=>-0.114 (0.005)
FIT:FeatureName=>SensorFusion, RSBDrift=>0.30760
FIT:FeatureName=>SensorFusion, AccelBias0=>0.12854
FIT:FeatureName=>SensorFusion, AccelBias1=>0.19535
FIT:FeatureName=>SensorFusion, AccelBias2=>-0.10591
FIT:FeatureName=>SensorFusion, GyroBias0=>-0.00679
FIT:FeatureName=>SensorFusion, GyroBias1=>-0.00544
FIT:FeatureName=>SensorFusion, GyroBias2=>-0.00180
FIT:FeatureName=>SensorFusion, GravityBias0=>0.49261
FIT:FeatureName=>SensorFusion, GravityBias1=>0.21229
FIT:FeatureName=>SensorFusion, GravityBias2=>-9.78607
FIT:FeatureName=>SensorFusion, FrameCount=>2375
FIT:FeatureName=>SensorFusion, ImuCount=>180880
FIT:FeatureName=>SensorFusion, AvgTimeFrameUpdate=>178.39ms
FIT:FeatureName=>SensorFusion, NumResets=>1
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
TrckLnDst:182 46 36 69 45 37 32 43 20 22 21 16 16 11 8 11 4 8 7 2 4 2 6 4 2 2 5 3 2 2 3 0 2 3 4 1 1 1 4 1 2 1 0 1 5 1 5 1 0 3 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 1 1 2 0 1 2 1 1 0 0 1 0 0 0 2 0 0 0 2 0 1 0 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 108
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
Distribution of num of consec keyframes w/ GN failing all features:42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: LIBMV: AR INFO:
Distribution of num of consec frames w/ increasing depth uncertainty:2354 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: WARNING: disconnected from camera server, resetting VIO
Nov 16 21:25:48 apq8096 voxl-qvio-server[2539]: client "voxl-portal0" connected to overlay
Here are our versions etc.
Service Name | Version | Enabled | Running | CPU Usage
---------------------------------------------------------------------------
docker-autorun | 1.2.4 | Disabled | Not Running |
docker-daemon | 1.2.4 | Disabled | Not Running |
modallink-relink | 0.16.1 | Disabled | Not Running |
voxl-camera-server | 1.3.5 | Enabled | Running | 12.7
voxl-cpu-monitor | 0.3.0 | Enabled | Running | 0.0
voxl-dfs-server | 0.3.1 | Disabled | Not Running |
voxl-imu-server | 1.0.3 | Enabled | Running | 0.0
voxl-mavlink-server | 0.2.0 | Enabled | Running | 1.8
voxl-modem | 0.16.1 | Disabled | Not Running |
voxl-portal | 0.4.2 | Enabled | Running | 0.0
voxl-qvio-server | 0.8.2 | Enabled | Running | 9.1
voxl-streamer | 0.4.1 | Disabled | Not Running |
voxl-tag-detector | 0.0.4 | Disabled | Not Running |
voxl-tflite-server | 0.3.1 | Disabled | Not Running |
voxl-time-sync | 1.2.2 | Disabled | Not Running |
voxl-vision-px4 | 1.4.0 | Enabled | Running | 0.0
voxl-wait-for-fs | 1.2.2 | Enabled | Completed |
This happens with or without the portal. Other units seem OK and this unit was OK in a different environment. Trying to isolate the issue.