Open-vins keeps diverging and re-initializing.

[Yufei Jiang]

Hi,

I'm using VOXL2 with flight deck and trying to test the open-vins. Currently I'm only using one camera (tracking) and the output will diverge quickly once I slightly move the VOXL2. Is there any clue about why and how I can solve this?

Below is my system:
voxl2:/$ voxl-version
────────────────────────────────────────────────────────────────────────────────
system-image: 1.8.02-M0054-14.1a-perf
kernel: #1 SMP PREEMPT Mon Nov 11 22:08:01 UTC 2024 4.19.125
────────────────────────────────────────────────────────────────────────────────
hw platform: M0054
mach.var: 1.0.0
SKU: MDK-F0006-4-V2-C11-E1
────────────────────────────────────────────────────────────────────────────────
voxl-suite: 1.4.5
────────────────────────────────────────────────────────────────────────────────
Packages:
Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.4/binary-arm64/
Last Updated: 2025-06-05 19:03:39
List:
kernel-module-voxl-fsync-mod-4.19.125 1.0-r0
kernel-module-voxl-gpio-mod-4.19.125 1.0-r0
kernel-module-voxl-platform-mod-4.19.125 1.0-r0
libfc-sensor 1.0.7
libmodal-cv 0.5.18
libmodal-exposure 0.1.4
libmodal-journal 0.2.5
libmodal-json 0.4.7
libmodal-pipe 2.13.1
libqrb5165-io 0.5.0
libvoxl-cci-direct 0.3.2
libvoxl-cutils 0.1.5
modalai-slpi 1.1.20
mv-voxl 0.1-r0
qrb5165-bind 0.1-r0
qrb5165-dfs-server 0.2.0
qrb5165-imu-server 1.1.3
qrb5165-rangefinder-server 0.1.5
qrb5165-slpi-test-sig 01-r0
qrb5165-system-tweaks 0.3.5
qrb5165-tflite 2.8.0-2
voxl-bind-spektrum 0.1.1
voxl-camera-calibration 0.5.9
voxl-camera-server 2.2.1
voxl-ceres-solver 2:1.14.0-10
voxl-configurator 1.0.1
voxl-cpu-monitor 0.6.0
voxl-docker-support 1.3.1
voxl-elrs 0.4.7
voxl-esc 1.5.2
voxl-feature-tracker 0.5.2
voxl-flow-server 0.3.6
voxl-fsync-mod 1.0-r0
voxl-gphoto2-server 0.0.10
voxl-gpio-mod 1.0-r0
voxl-io-server 0.0.5
voxl-jpeg-turbo 2.1.3-7
voxl-lepton-server 1.3.3
voxl-lepton-tracker 0.0.4
voxl-libgphoto2 0.0.4
voxl-libuvc 1.0.7
voxl-logger 0.5.3
voxl-mavcam-manager 0.5.8
voxl-mavlink 0.1.5
voxl-mavlink-server 1.4.7
voxl-modem 1.1.6
voxl-mongoose 7.7.0-2
voxl-mpa-to-ros 0.3.9
voxl-mpa-tools 1.4.0
voxl-open-vins 0.4.19
voxl-open-vins-server 0.3.12
voxl-opencv 4.5.5-3
voxl-osd 0.1.7
voxl-platform-mod 1.0-r0
voxl-portal 0.7.10
voxl-px4 1.14.0-2.0.98
voxl-px4-imu-server 0.1.2
voxl-px4-params 0.7.3
voxl-qvio-server 1.2.0
voxl-remote-id 0.0.9
voxl-reset-slpi 0.0.1
voxl-state-estimator 0.0.5
voxl-streamer 0.7.5
voxl-suite 1.4.5
voxl-tag-detector 0.0.4
voxl-tflite-server 0.4.1
voxl-utils 1.4.8
voxl-uvc-server 0.1.7
voxl-vision-hub 1.8.21
voxl-vtx 1.3.14
voxl-wavemux 0.0.1
voxl2-io 0.0.3
voxl2-system-image 1.8.02-r0
voxl2-wlan 1.0-r0
────────────────────────────────────────────────────────────────────────────────

voxl-open-vins-server.conf:
{
"en_auto_reset": true,
"auto_reset_max_velocity": 20,
"auto_reset_max_v_cov_instant": 0.10000000149011612,
"auto_reset_max_v_cov": 0.10000000149011612,
"auto_reset_max_v_cov_timeout_s": 0.5,
"auto_reset_min_features": 1,
"auto_reset_min_feature_timeout_s": 3,
"auto_fallback_timeout_s": 3,
"auto_fallback_min_v": 0.600000023841858,
"en_cont_yaw_checks": false,
"fast_yaw_thresh": 5,
"fast_yaw_timeout_s": 1.75,
"do_fej": true,
"imu_avg": true,
"use_rk4_integration": true,
"cam_to_imu_refinement": true,
"cam_intrins_refinement": true,
"cam_imu_ts_refinement": true,
"max_clone_size": 8,
"max_slam_features": 35,
"max_slam_in_update": 10,
"max_msckf_in_update": 10,
"feat_rep_msckf": 4,
"feat_rep_slam": 4,
"cam_imu_time_offset": 0,
"slam_delay": 1,
"gravity_mag": 9.80665,
"init_window_time": 0.5,
"init_imu_thresh": 2,
"imu_sigma_w": 0.00013990944749616306,
"imu_sigma_wb": 4.1189724174615527e-07,
"imu_sigma_a": 0.0038947538150776763,
"imu_sigma_ab": 5.538346201712153e-05,
"msckf_chi2_multiplier": 1,
"slam_chi2_multiplier": 40,
"zupt_chi2_multiplier": 1,
"refine_features": true,
"pyr_levels": 6,
"grid_x": 5,
"grid_y": 5,
"msckf_sigma_px": 1,
"slam_sigma_px": 1.8,
"zupt_sigma_px": 1,
"try_zupt": true,
"zupt_max_velocity": 0.03,
"zupt_only_at_beginning": true,
"zupt_noise_multiplier": 1,
"zupt_max_disparity": 8,
"init_dyn_use": false,
"triangulate_1d": false,
"max_runs": 5,
"init_lamda": 0.001,
"max_lamda": 10000000000,
"min_dx": 1e-06,
"min_dcost": 1e-06,
"lam_mult": 10,
"min_dist": 0.1,
"max_dist": 60,
"max_baseline": 40,
"max_cond_number": 600000,
"use_mask": false,
"use_stereo": false,
"use_baro": false,
"num_opencv_threads": 4,
"fast_threshold": 15,
"histogram_method": 1,
"knn_ratio": 0.7,
"takeoff_accel_threshold": 0.5,
"takeoff_threshold": -0.1,
"use_stats": false,
"max_allowable_cep": 1,
"en_force_init": false,
"en_force_ned_2_flu": false,
"en_imu_frame_output": true,
"track_frequency": 15,
"publish_frequency": 5,
"en_vio_always_on": true,
"en_ext_feature_tracker": false,
"en_gpu_for_tracking": true,
"num_features_to_track": 10,
"raansac_gn": false,
"raansac_tri": true,
"en_thermal_enhance": false,
"en_overlay_landscape": false,
"thermal_brightness": 1,
"thermal_brightness_bos": 1

Output of open-vins-server:

voxl2:/etc/modalai$ voxl-open-vins-server
Setting printing level to: SILENT

=========================AUTO RESET==============================
en_auto_reset: 1
auto_reset_max_velocity: 20.000
auto_reset_max_v_cov_instant: 0.100
auto_reset_max_v_cov: 0.100
auto_reset_max_v_cov_timeout_s: 0.500
auto_reset_min_features: 1
auto_reset_min_feature_timeout_s: 3.000
auto_fallback_timeout_s: 3.000
auto_fallback_min_v: 0.600
en_cont_yaw_checks: false
fast_yaw_thresh: 5.000
fast_yaw_timeout_s: 1.750
===========================STATE=================================
do fej: true
imu avg: true
use rk4 integration: true
cam to imu refinement: true
cam intrins refinement: true
cam imu ts refinement: true
max clone size: 8
max slam features: 35
max slam in update: 10
max msckf in update: 10
feat rep msckf: ANCHORED_MSCKF_INVERSE_DEPTH
feat rep slam: ANCHORED_MSCKF_INVERSE_DEPTH
cam imu time offset: 0.00000
slam delay: 1.00000
pyr levels: 6
grid x: 5
grid y: 5

=====================INERTIAL INITIALIZER========================
gravity mag: 9.80665
init window time: 0.50000
init imu thresh: 2.00000
init dyn use: false

==========================IMU NOISE==============================
imu sigma w: 0.00014
imu sigma wb: 0.00000
imu sigma a: 0.00389
imu sigma ab: 0.00006

========================FEATURE NOISE============================
msckf chi^2 multiplier: 1.00000
msckf sigma px: 1.00000
slam chi^2 multiplier: 40.00000
slam sigma px: 1.80000
zupt chi^2 multiplier: 1.00000
zupt sigma px: 1.00000

=============================ZUPT================================
refine features: true
max runs: 5
try zupt: true
en_ext_feature_tracker: 0
en_gpu_for_tracking: 1
raansac_gn: false
raansac_tri: true
zupt max velocity: 0.03000
zupt only at beginning: true
zupt noise multiplier: 1.00000
zupt max disparity: 8.00000

use mask: false
use stereo: false
use baro: false

========================FEATURE INITIALIZER======================
triangulate 1d: false
init lamda: 0.00100
max lamda: 10000000000.00000
min dx: 0.00000
min dcost: 0.00000
lam mult: 10.00000
min dist: 0.10000
max dist: 60.00000
max baseline: 40.00000
num_opencv_threads: 4
fast_threshold: 15
histogram_method: HISTOGRAM
knn_ratio: 0.70000
track_frequency: 15.00000
publish_frequency: 5
imu takeoff accel thresh: 0.50000
force init on takeoff: false
takeoff_threshold(m): -0.100000
publish stats: false
max_allowable_cep: 1.00000
force FLU to NED transform: false
publish IMU frame: true
VIO always on (for bench testing): true
num_features_to_track: 10
Enable Thermal Enhance: false
Enable Overlay Landscape: false
Thermal Brightness: 1.00000
Thermal Brightness BOSON: 1.00000

=================================================================
vio_cam config:
cam 0:
enable: 1
name: tracking
pipe_for_preview: tracking
pipe_for_tracking: tracking
is_occluded_on_ground: 0
imu: imu_apps
is_extrinsic_present: 1
extrinsic cam wrt imu:
parent: imu_apps
child: tracking
T_cam_wrt_imu: 0.027 0.007 0.018
RPY_imu_to_cam: 0.0 45.0 90.0
R_cam_to_imu: 0.000 -0.707 0.707
1.000 0.000 -0.000
-0.000 0.707 0.707
cal file: opencv_tracking_intrinsics.yml
is_cal_present: 1
cam cal intrinsics:
width: 640
height: 480
fx: 277.659
fy: 277.763
cx: 349.269
cy: 219.049
n_coeffs: 4
is_fisheye: 1
-0.0240 0.0390 -0.0284 0.0077

vio cam: 0 is occluded: 0
[INFO] Camera: 0 -- converted extrinsics in FLU are: Roll: -0.000000, Pitch 135.000000, Yaw -90.000000 (-1.570796)
WORLD QUATERNION: -0.000000 0.000000 -0.000000 1.000000
OpenVINS using fisheye camera
OpenVINS FINAL Cam extrinsics to IMU for tracking: 7
-0.653281
0.653281
-0.270598
0.270598
0.0273
-0.0067
-0.0182
++++++++++++++++++++++++++

OpenVINS reports number of cameras being used, regardless of delivery method (pipe, merge, etc...): 1

++++++++++++++++++++++++++
Set camera intrinsics and extrinsics

====> Using Internal KLT feature tracker (w/ GPU) <====
initialized ocl manager

Current reference counts:
trackDB: 1
trackFEATS: 1
propagator: 2
updaterMSCKF: 1
updaterSLAM: 1
active_tracks_initializer: 1
initializer: 1
thread is locked to cores: 4 5 6 7
create_server_pipes
connecting client pipes
imu pipe name: imu_apps
Number of Cameras active: 1
Camera merge --- > ch: 1 to cam id: 0
Opening camera pipe: tracking
tmp_camera_pipe_names.push_back(): tracking
client "voxl-portal66662777" connected to overlay
[WARN] (0.005064 and 0.235771) Cannot initialize FRD to IMU transform--too much drift: your IMU may need to be recalibrated or is OVERHEATED!
[WARN] (0.004868 and 0.241071) Cannot initialize FRD to IMU transform--too much drift: your IMU may need to be recalibrated or is OVERHEATED!
[WARN] (0.003866 and 0.231757) Cannot initialize FRD to IMU transform--too much drift: your IMU may need to be recalibrated or is OVERHEATED!
[INFO] world : 1.000000 1.000000 (2)
[WARN] (0.003118 and 0.237088) Cannot initialize FRD to IMU transform--too much drift: your IMU may need to be recalibrated or is OVERHEATED!
[WARN] (0.005283 and 0.226763) Cannot initialize FRD to IMU transform--too much drift: your IMU may need to be recalibrated or is OVERHEATED!
FAILED TO FETCH IMG RINGBUF at time 0 -2 -2 -2 -2 -2
[INFO] FRD to IMU NED configuration done. g direction: -1
[INFO] gravity direction (2) : -1
[INFO] *** FRD to IMU transform READY! ***
FAILED TO FETCH IMG RINGBUF at time 0 -2 -2 -2 -2 -2
waiting for enough clone states (1 of 5)....
[WARN] Motion detected, going to full processing
waiting for enough clone states (2 of 5)....
waiting for enough clone states (3 of 5)....
waiting for enough clone states (4 of 5)....
[VIO_BAD_STATE] Quality less than 1.
WARNING auto-resetting, Bad VIO, State: true Q: -1 Vel: 0.043736 Uncert: 0.949216 State: 0(1)
[INFO] restarting managers STATELESS
Set camera intrinsics and extrinsics

====> Using Internal KLT feature tracker (w/ GPU) <====
initialized ocl manager

Current reference counts:
trackDB: 1
trackFEATS: 1
propagator: 2
updaterMSCKF: 1
updaterSLAM: 1
active_tracks_initializer: 1
initializer: 1
Camera queue size after reset: 0
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1959.206927 - 1958.867053 (0.339874)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1959.237218 - 1958.867053 (0.370165)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1959.282138 - 1958.867053 (0.415084)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1959.311426 - 1958.867053 (0.444373)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1959.336785 - 1958.867053 (0.469732)
waiting for enough clone states (1 of 5)....
[INFO] TAKEOFF: DISPARITY disturbed, possible motion detected. Avg: 434.788239 (8.000000) # feats: 2 (20).
waiting for enough clone states (2 of 5)....
[INFO] TAKEOFF: DISPARITY disturbed, possible motion detected. Avg: 0.000000 (8.000000) # feats: 2 (20).
waiting for enough clone states (3 of 5)....
waiting for enough clone states (4 of 5)....
[WARN] Motion detected, going to full processing
[VIO_BAD_STATE] Quality less than 1.
WARNING auto-resetting, Bad VIO, State: true Q: -1 Vel: 7.194683 Uncert: 0.586549 State: 0(1)
[INFO] restarting managers STATELESS
Set camera intrinsics and extrinsics

====> Using Internal KLT feature tracker (w/ GPU) <====
initialized ocl manager

Current reference counts:
trackDB: 1
trackFEATS: 1
propagator: 2
updaterMSCKF: 1
updaterSLAM: 1
active_tracks_initializer: 1
initializer: 1
Camera queue size after reset: 0
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1965.271825 - 1964.916372 (0.355453)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1965.311861 - 1964.916372 (0.395488)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1965.342133 - 1964.916372 (0.425761)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1965.376322 - 1964.916372 (0.459950)
IMU samples too small, make your window smaller (params.init_window_time), can't initalize 1965.411513 - 1964.916372 (0.495141)
waiting for enough clone states (1 of 5)....

[zauberflote1]

Hello,
Please refer to the following and see if the problem persists:
https://forum.modalai.com/topic/4475/voxl-open-vins-server?_=1749222785639

Thanks,

[Yufei Jiang]

@zauberflote1 Hi, I recalibrated the imu and checked it using voxl-inspect-imu. The imu data looks good. In voxl-inspect-cam it can detect all the cameras. However, it's still diverging really fast once I moved the VOXL2. If I disabled the auto-reset of vins, it will keep drifting.

[Yufei Jiang]

@Yufei-Jiang And I used voxl-configure-cameras and voxl-configure-extrinsics D0006_Sentinel to generate the camera config.