Persistent PX4 Sensor/Accel Failure on Stinger (D0013) with SDK 1.6.2 + Dev IMU/OV Servers

YUUJI INOUE

Support Request: Persistent PX4 Sensor/Accel Failure on Stinger (D0013) with SDK 1.6.2 + Dev IMU/OV Servers

System Information

Hardware: ModalAI Stinger (D0013)
Base SDK Version: SDK 1.6.2 (Clean install performed recently)
Updated Packages:
- qrb5165-imu-server: Built from dev branch (to support body frame output)
- voxl-open-vins-server: Built from dev branch
Camera Configuration: Tracking Cameras (Front/Down) only.

Current Status

After a clean install of SDK 1.6.2 and upgrading the IMU and OpenVINS servers to the dev branch versions (as suggested to handle D0013's inverted IMU natively):

VIO Tracking: Success. The drone is tracking perfectly in the Web Portal's 3D view. The virtual drone follows the real drone's movements correctly (RPY and translation).
VOXL Vision Hub: Success. voxl-vision-hub reports result: true in the Web Portal system check.
Image Sensors: Success. Previously reported errors, but now imageSensor0 and imageSensor1 are detected correctly.
PX4 Flight Controller: FAILURE.
- voxl-px4 reports result: false in the Web Portal system check.
- Specific Error: sensor_accel: result false (e.g., x: -0.28, y: -0.28, z: 9.63).
- The drone cannot enter Position Flight Mode.
- QGroundControl allows Arming in Manual mode, but rejects Position mode.

Steps Taken

Clean Install: Re-flashed SDK 1.6.2 to ensure a clean state.
Custom Build: Built and installed qrb5165-imu-server and voxl-open-vins-server from their dev branches on the device (using voxl-cross docker).
- This resolved previous issues with "Upside Down" initialization and VIO coordinate frame mismatches.
Calibration: Performed clean IMU and Camera calibrations via voxl-configure-mpa and QGroundControl.
Verification: Checked Web Portal. VIO is healthy, but PX4 fails to initialize the accelerometer properly.

Logs & Errors

We have attached the full voxl-px4.log, voxl-imu-server.log, and voxl-vision-hub.log files. Below are the critical errors observed.

1. Parameter File Access Error (from voxl-px4.log)

PX4 fails to access the parameter file, which may be causing it to load incorrect defaults.

Dec 19 09:35:49 m0104 voxl-px4[3893]: INFO  [param] selected parameter default file /data/px4/param/parameters
Dec 19 09:35:49 m0104 voxl-px4[3893]: ERROR [param] open '/data/px4/param/parameters' failed (2)

2. Sensor Start Timeouts (from voxl-px4.log)

The system attempts to start the IMU driver (icm42688p) but times out.

Dec 19 09:35:49 m0104 voxl-px4[3893]: Starting IMU driver with rotation 12
Dec 19 09:36:09 m0104 voxl-px4[3893]: INFO  [qshell] Send cmd: 'icm42688p start -s -R 12'
Dec 19 09:36:09 m0104 voxl-px4[3893]: ERROR [qshell] command timed out
Dec 19 09:36:09 m0104 voxl-px4[3893]: ERROR [qshell] Command failed

3. Accelerometer Failure (from voxl-px4.log)

This is the direct cause of the sensor_accel: result false status.

Dec 19 09:36:49 m0104 voxl-px4[2171]: ERROR [muorb] [sensors] Accel #0 fail:  TIMEOUT!

4. IMU Server Analysis (from voxl-imu-server.log)

The voxl-imu-server appears to be running and detecting the IMU correctly (Detected M0104 VOXL2 mini, detected ICM42688).
It loads calibration data successfully.

Dec 19 10:13:42 m0104 voxl-imu-server[30843]: detected ICM42688 on spi bus[id] 3
Dec 19 10:13:42 m0104 voxl-imu-server[30843]: started fifo on imu0

IMU server seems healthy and publishing data.

5. Vision Hub Analysis (from voxl-vision-hub.log)

voxl-vision-hub successfully connects to the VIO pipe (ov), but reports an issue with finding a specific extrinsic relation.

Dec 19 10:07:19 m0104 voxl-vision-hub[1723]: Connected to VIO pipe: ov
Dec 19 10:07:19 m0104 voxl-vision-hub[1723]: Geometry module updating to use imu: imu_apps_body for VIO
...
Dec 19 10:07:17 m0104 voxl-vision-hub[1723]: ERROR: /etc/modalai/extrinsics.conf missing body to imu_apps_body, sticking with identity for now

Also, there are connection errors to voxl-mavlink-server:

Dec 19 10:07:19 m0104 voxl-vision-hub[1723]: Error trying to send control command to pipe ch4 /run/mpa/mavlink_onboard/, not connected

This suggests that while VIO itself is running, the communication chain to send this data to PX4 (via mavlink-server) might be broken or misconfigured.

Questions

Despite VIO working perfectly with the updated servers and voxl-imu-server appearing healthy, PX4 consistently fails to validate the accelerometer (sensor_accel: result false).

Is there a specific PX4 parameter or driver configuration in SDK 1.6.2 that conflicts with the Stinger (D0013) hardware, even with the updated IMU server?
The accelerometer values (z ~ 9.6) seem physically reasonable for a stationary drone. Why does the check return false?
Are there known compatibility issues with voxl-px4 and the dev branch qrb5165-imu-server?

voxl-vision-hub.log
voxl-imu-server.log
voxl-px4.log

Eric Katzfey

@YUUJI-INOUE Can you send a PX4 log? If you can arm in manual mode then the PX4 IMU is working fine. The last voxl-px4 message sequence shows the IMU start correctly. If the parameters file does not yet exist (e.g. no PX4 parameters have been updated yet) then there will be no /data/px4/param/parameters file and you will see the error message: "open '/data/px4/param/parameters' failed". That is fine and to be expected. Once you have updated some parameters (e.g. running voxl-px4-param) then that file will be created and that error message will no longer appear.

YUUJI INOUE

@Eric-Katzfey

Updated Support Request: Parameter Fix Attempt & Remaining Accelerometer/Position Mode Issues

Following the advice to update parameters, we have made progress but still cannot enter Position Mode due to persistent sensor checks failures.

Actions Taken

Parameter Reset & Load:
- Executed voxl-configure-px4-params --wizard and selected D0013 (Stinger).
- Confirmed that /data/px4/param/parameters is now created and populated.
Battery Configuration Fix:
- Manually corrected BAT_N_CELLS to 4 (was defaulting to 6) and BAT_LOW_VOLT to 14.0V.
- This successfully resolved the "Low Voltage" warning in QGC.
Re-Calibration:
- Performed fresh Accelerometer, Level Horizon, and Compass calibrations in QGC.
- Performed Camera calibration.

Current Status

Manual Mode: OK. We can Arm the drone and spin motors in Manual Mode. We have also performed a successful test flight in Manual Mode.
VIO: OK. The virtual drone in Web Portal follows the real drone perfectly.
Position Mode: FAILED.
- QGC shows "Ready for flight" briefly after reboot, but rejects Arming (Command denied).
- After disarming from a Manual Mode test, Position Mode switches to "Not Ready".
Web Portal System Check: FAILED.
- voxl-px4 still reports sensor_accel: result false.
- Values (e.g., z: 9.70) seem correct, but the check fails.

Persistent Logs Errors (New Logs Attached)

Despite the parameter fix, voxl-px4.log still shows communication timeouts with the sensors:

ERROR [muorb] [sensors] Accel #0 fail: TIMEOUT!
ERROR [qshell] Command failed
ERROR [muorb] [drivers__device] i2c probe failed

Questions

Since we can Arm in Manual Mode (suggesting the IMU hardware is working) and have loaded the correct D0013 parameters, why does voxl-px4 still report sensor_accel: result false with timeouts?
Is there a specific driver or timing parameter in SDK 1.6.2 (with the dev branch IMU server) that causes these spurious timeouts on the Stinger?
What is preventing Position Mode Arming even when VIO is healthy and Manual Arming works?

We have attached the latest logs (voxl-px4.log, voxl-vision-hub.log) from this new configuration state.
voxl-px4.log
voxl-vision-hub.log

YUUJI INOUE

@YUUJI-INOUE said in Persistent PX4 Sensor/Accel Failure on Stinger (D0013) with SDK 1.6.2 + Dev IMU/OV Servers:

@Eric-Katzfey

Updated Support Request: Parameter Fix Attempt & Remaining Accelerometer/Position Mode Issues

Following the advice to update parameters, we have made progress but still cannot enter Position Mode due to persistent sensor checks failures.

Actions Taken

Parameter Reset & Load:

Executed voxl-configure-px4-params --wizard and selected D0013 (Stinger).

Confirmed that /data/px4/param/parameters is now created and populated.

Battery Configuration Fix:

Manually corrected BAT_N_CELLS to 4 (was defaulting to 6) and BAT_LOW_VOLT to 14.0V.

This successfully resolved the "Low Voltage" warning in QGC.

Re-Calibration:

Performed fresh Accelerometer, Level Horizon, and Compass calibrations in QGC.

Performed Camera calibration.

Current Status

Manual Mode: OK. We can Arm the drone and spin motors in Manual Mode. We have also performed a successful test flight in Manual Mode.

VIO: OK. The virtual drone in Web Portal follows the real drone perfectly.

Position Mode: FAILED.

QGC shows "Ready for flight" briefly after reboot, but rejects Arming (Command denied).

After disarming from a Manual Mode test, Position Mode switches to "Not Ready".

Web Portal System Check: FAILED.

voxl-px4 still reports sensor_accel: result false.

Values (e.g., z: 9.70) seem correct, but the check fails.

Persistent Logs Errors (New Logs Attached)

Despite the parameter fix, voxl-px4.log still shows communication timeouts with the sensors:
ERROR [muorb] [sensors] Accel #0 fail: TIMEOUT!
ERROR [qshell] Command failed
ERROR [muorb] [drivers__device] i2c probe failed
Questions

Since we can Arm in Manual Mode (suggesting the IMU hardware is working) and have loaded the correct D0013 parameters, why does voxl-px4 still report sensor_accel: result false with timeouts?

Is there a specific driver or timing parameter in SDK 1.6.2 (with the dev branch IMU server) that causes these spurious timeouts on the Stinger?

What is preventing Position Mode Arming even when VIO is healthy and Manual Arming works?

We have attached the latest logs (voxl-px4.log, voxl-vision-hub.log) from this new configuration state.
voxl-px4.log
voxl-vision-hub.log

I've been hooked on this for two weeks now. Can someone tell me how to set up STINGER VER1 from scratch?

Eric Katzfey

@YUUJI-INOUE I'm very confused by the voxl-px4.log file you attached. It is filled with a bunch of -- Reboot -- lines. Where do those lines come from? How did you generate voxl-px4.log? The only Accel #0 fail: TIMEOUT! I see is after a shutdown command has been issued to PX4. The Command failed is normal. The startup script attempts to do some auto-configuration by looking for devices that may not be attached to the drone and will produce those messages. Same with the i2c probe failed messages. The fact that you can fly in manual mode means that the IMU is fine.

YUUJI INOUE

@Eric-Katzfey
Thank you for your reply. I tried various SKU configurations and was able to successfully enter position mode. Thank you.
The current SKU is MRB-D0013-6-V2-C33-T7-M22-X14.

YUUJI INOUE

@Eric-Katzfey

Support Request: VIO Instability on Custom Drone Frame (Compared to Stinger)

Summary

We have successfully configured a VOXL 2 Mini on a Stinger frame (D0013) with SDK 1.6.2, and VIO works perfectly in our indoor environment.
However, we are facing persistent VIO instability on our custom drone frame, which uses the same VOXL 2 Mini and camera sensors, despite trying extensive troubleshooting.

The core issue is that feature points drop to 0 and VIO quality drops to 0% specifically when the drone is hovering or stationary. Paradoxically, when we move the drone or shine a bright light on the floor, the feature count improves and VIO holds for longer.

Hardware Setup

Flight Controller: VOXL 2 Mini (SDK 1.6.2)
Cameras:
- tracking_front (Ar0144)
- tracking_down (Ar0144, rotated 90 degrees)
- Note: This custom frame DOES NOT have a Hi-Res camera. It only has 2 tracking cameras.
- However, since selecting "Tracking only" in the SKU configuration prevented us from entering Position Mode, we configured it as "Hi-Res + 2 Tracking" and then manually disabled the hires camera in voxl-camera-server.conf.
Frame: Custom frame (Carbon fiber).
- Note: We identified that the frame was reflecting into the camera view. We have covered the reflective parts with black tape, which improved stability slightly but did not solve the issue.

Symptoms

Hovering/Stationary: There are no issues when the drone is constantly moving. However, when it remains stationary for more than 10-20 seconds, feature points (num_pts) gradually decrease and eventually hit 0. VIO Quality then drops to 0%.
Moving/Flight: When moving the drone (in flight or by hand), feature points increase to the max of ~30 and VIO recovers.
Lighting: Shining a strong light on the floor extends the time before VIO fails, but it eventually drops to 0.
Comparison: A Stinger drone in the exact same indoor environment and lighting holds position perfectly with default settings.

Troubleshooting Steps Taken (All Reverted to Default)

We have tried the following configuration changes, but none provided a permanent solution, so we have reverted everything to default settings (except for necessary camera ID fixes):

VIO Config (estimator_config.yaml):
- Tried zupt_only_at_beginning: false (to enable ZUPT during flight). -> No effect / Worsened.
- Tried increasing num_pts (50 -> 150) and lowering fast_threshold (15 -> 5). -> No effect.
Camera Config (voxl-camera-server.conf):
- Tried increasing exposure_max_us (12ms -> 33ms). -> QGC rejected position mode more strictly.
- Tried increasing gain_max (8000 -> 16000). -> No significant improvement.
IMU Config (voxl-imu-server.conf):
- Tried lowering imu0_lp_cutoff_freq_hz (92Hz -> 42Hz, 20Hz). -> No effect.
Calibration:
- Performed voxl-calibrate-camera with the frame taped to prevent reflections.
- Updated extrinsics.conf with measured values.

Current Configuration (Clean State)

SDK: 1.6.2 (Clean install)
voxl-vision-hub.conf: en_vio: true, vio_pipe: ov (OpenVINS)
voxl-camera-server.conf: Default exposure (12ms), hires disabled.
estimator_config.yaml: Default (OpenVINS default).
voxl-imu-server.conf: Default (92Hz).

Question

Why does the custom drone fail to maintain feature points in the same environment where the Stinger works perfectly?
Since "shining a light" and "moving the drone" help, it seems related to signal-to-noise ratio or feature detection thresholds, but software parameter tuning has failed.

Are there any specific IMU noise characteristics or vibration issues on custom frames that could cause OpenVINS to reject static features? Or are there other low-level parameters we should look at for a custom frame build?

We have attached the following logs from a failed flight attempt:
voxl-px4.log
voxl-vision-hub.log
voxl-open-vins-server.log

YUUJI INOUE

@Eric-Katzfey
P.S.
Camera calibration has been completed for both the front and down cameras, with both being below 0.3.
Camera position and rotation have also been set (same rotation as STINGER).
Camera focus is also perfectly adjusted.

YUUJI INOUE

@Eric-Katzfey
maiden flight
CustomFrame

zauberflote1

@YUUJI-INOUE
Hello there,
Here are a few things to be aware of:

VVHUB "Error" for missing extrinsics for imu_body

This is not really an error, just tells you that the IMU in body frame and the body frame itself are connected by the identity. In other words, there's no transformation needed for the IMU data from VVHUB standpoint.

Position Mode not engaging

Assuming you're able to get the drone arming in Manual/Altitude mode. Are you using VFC in your VVHUB conf or using the classic PX4 position mode? -- If your drone resembles the Stinger D0013, VFC is the suggested format. -- you can use the wizard to configure VVHUB accordingly. Can you confirm your VVHUB setup? Please attach your vision-hub and vfc .conf

VIO not working

As you described, different frames, different noise characterization. You can change the IMU noise values inside the kalibr_imu_chain.yaml (in the directory you found estimator.yaml). Personally based on your description, I am inclined to believe your performance issue is rooted in your extrinsics values and VVHUB configuration. Did you use Kalibr? -- If you used Kalibr with imu_body pipe, all you have to do is to replace the SE(3) elements inside kalibr_imucam_chain.yaml and make sure the sync_config flag inside voxl-open-vins-server.conf is set to false. Otherwise, unless you have a proper CAD model, physically measured values will likely be inaccurate and cause the estimator to diverge. For future help, please attach all your VIO-related yamls and your voxl-open-vins-server.conf

All the best,
ZBFT

YUUJI INOUE

@zauberflote1
Thank you for your support. I followed your advice and changed the settings again today.
It's much better than yesterday, but the reduction in feature points is still there. I've made a video of the current situation, so please check it out.
https://www.youtube.com/watch?v=H-_lLwPrPOE

I would like to ask some questions today as well, so thank you in advance.

Support Request: VIO Instability on Custom Drone Frame (Compared to Stinger) - Follow-up

Summary

We have successfully configured a VOXL 2 Mini on a Stinger frame (D0013) with SDK 1.6.2, and VIO works perfectly in our indoor environment.
However, we are facing persistent VIO instability on our custom drone frame, which uses the same VOXL 2 Mini and camera sensors, despite trying extensive troubleshooting.

The core issue is that feature points drop to 0 and VIO quality drops to 0% specifically when the drone is hovering or stationary. Paradoxically, when we move the drone or shine a bright light on the floor, the feature count improves and VIO holds for longer.

Update: After fixing camera extrinsics and setting sync_config: false, VIO performance improved significantly ("much better than yesterday"). However, the issue persists: feature points still drop to 0 during hovering, while Stinger maintains ~22 features in the same conditions.

Hardware Setup

Flight Controller: VOXL 2 Mini (SDK 1.6.2)
Cameras:
- tracking_front (Ar0144)
- tracking_down (Ar0144, rotated 90 degrees)
- Note: This custom frame DOES NOT have a Hi-Res camera. It only has 2 tracking cameras.
- However, since selecting "Tracking only" in the SKU configuration prevented us from entering Position Mode, we configured it as "Hi-Res + 2 Tracking" and then manually disabled the hires camera in voxl-camera-server.conf.
Frame: Custom frame (Carbon fiber), designed in CAD by ourselves.
- Note: We identified that the frame was reflecting into the camera view. We have covered the reflective parts with black tape, which improved stability slightly but did not solve the issue.
Camera Positions relative to IMU (FRD coordinate system):
- FRONT camera:
  - Forward (X): +70.8mm
  - Right (Y): -19.4mm (Left direction)
  - Down (Z): -20mm (Up direction)
- DOWN camera:
  - Forward (X): +38mm
  - Right (Y): -19.4mm (Left direction)
  - Down (Z): +9mm (Down direction)
Camera Calibration: Both FRONT and DOWN cameras have calibration errors ≤ 0.3 (excellent calibration quality).
Camera Focus: Focus is properly adjusted for both cameras.

Symptoms

Hovering/Stationary: There are no issues when the drone is constantly moving. However, when it remains stationary for more than 10-20 seconds, feature points (num_pts) gradually decrease and eventually hit 0. VIO Quality then drops to 0%.
Moving/Flight: When moving the drone (in flight or by hand), feature points increase to the max of ~30 and VIO recovers.
Lighting: Shining a strong light on the floor extends the time before VIO fails, but it eventually drops to 0.
Comparison: A Stinger drone in the exact same indoor environment and lighting holds position perfectly with default settings, maintaining ~22 features even when stationary.

Recent Troubleshooting Steps (Today)

IMU Noise Parameter Testing

We compared kalibr_imu_chain.yaml between Stinger and our custom frame and found they have identical values. Since different frames should have different noise characteristics, we tested adjusting IMU noise parameters:

Increased to 1.5x: No noticeable difference
Decreased to 0.8x: Performance worsened
Increased to 2x: Position mode could be maintained, but the drone oscillated (wobbled) excessively
Reverted to default: Best performance

Conclusion: IMU noise parameters are not the root cause. Default values are optimal.

Previous Troubleshooting Steps (All Reverted to Default)

We have tried the following configuration changes, but none provided a permanent solution, so we have reverted everything to default settings (except for necessary fixes):

VIO Config (estimator_config.yaml):
- Tried zupt_only_at_beginning: false (to enable ZUPT during flight). -> No effect / Worsened.
- Tried increasing num_pts (50 -> 150) and lowering fast_threshold (15 -> 5). -> VIO stopped working completely.
Camera Config (voxl-camera-server.conf):
- Tried increasing exposure_max_us (12ms -> 33ms). -> QGC rejected position mode more strictly.
- Tried increasing gain_max (8000 -> 16000). -> No significant improvement.
IMU Config (voxl-imu-server.conf):
- Tried lowering imu0_lp_cutoff_freq_hz (92Hz -> 42Hz, 30Hz, 20Hz). -> No effect.
- FFT analysis showed 200-240Hz vibrations, but filtering them did not improve VIO. Stinger shows similar vibrations but stable VIO.
Calibration:
- Performed voxl-calibrate-camera with the frame taped to prevent reflections.
- Updated extrinsics.conf and kalibr_imucam_chain.yaml with precise physical measurements.
OpenVINS Config (voxl-open-vins-server.conf):
- Set sync_config: false (based on support advice). -> Significant improvement.
- Tried sync_config: true (to match Stinger). -> Performance worsened.
- Current: sync_config: false (required for custom frame).

Current Configuration

SDK: 1.6.2
voxl-vision-hub.conf: en_vio: true, vio_pipe: ov (OpenVINS), offboard_mode: vfc
voxl-open-vins-server.conf: sync_config: false (required for custom frame)
voxl-camera-server.conf: Default exposure (12ms), hires disabled.
estimator_config.yaml: Default (OpenVINS default).
voxl-imu-server.conf: Default (92Hz).
kalibr_imu_chain.yaml: Default (same as Stinger).
kalibr_imucam_chain.yaml: Updated with precise physical measurements for camera extrinsics.

Key Differences from Stinger

sync_config: Stinger works with sync_config: true, but our custom frame requires sync_config: false.
Feature point behavior: Stinger maintains ~22 features when stationary, while our custom frame drops to 0.
IMU noise parameters: Identical values, but custom frame may have different vibration characteristics.

Question

Why does the custom drone fail to maintain feature points in the same environment where the Stinger works perfectly?

Since:

Frame is designed in CAD with precise measurements
Extrinsics are correctly measured and configured (FRONT: Forward 70.8mm, Left 19.4mm, Up 20mm; DOWN: Forward 38mm, Left 19.4mm, Down 9mm)
Camera calibration errors are ≤ 0.3 for both cameras (excellent quality)
Camera focus is properly adjusted
sync_config: false is required (unlike Stinger)
IMU noise parameter adjustments don't help
Frame reflections are taped off

What other factors could cause OpenVINS to lose feature points specifically during hovering/stationary periods on a custom frame?

We have attached the following configuration files:

kalibr_imu_chain.yaml　https://drive.google.com/file/d/1xk3hV08xkCsr_76CaHUp71kmJrE4qDip/view?usp=sharing
kalibr_imucam_chain.yaml
https://drive.google.com/file/d/1fTq8KKdH1gERFFMk34GBBJd5IRFqAw9Z/view?usp=sharing
estimator_config.yaml
https://drive.google.com/file/d/1vtH50HK-VgsAOCAZEtjA27-kU0kD_aic/view?usp=sharing
voxl-open-vins-server.conf
https://drive.google.com/file/d/1seNpGRoWwLbVvCMlXYlQeMPOkTn04nc0/view?usp=sharing