voxl-mapper produces no map; VIO output rate halves to 15 Hz and does not recover until OpenVINS restart

[김진혁]

Hardware: Starling 2 Max, SKU MRB-D0012-4-V4-C29-T8-M36, hw platform M0054 (C29 config with PMD ToF). Received 14 Sep 2026, factory state.

Software version: (voxl-version)

• system-image 1.8.08-M0054-14.1a-perf, kernel 4.19.125
• voxl-suite 1.6.4~beta5 (factory; SDK not re-flashed)
• voxl-mapper 0.2.4, voxl-voxblox 1.1.5, voxl-nlopt 2.5.0-5 (installed via apt)

Configuration / services: (voxl-inspect-services)

• Running: voxl-camera-server, voxl-imu-server, voxl-open-vins-server, voxl-px4, voxl-vision-hub, voxl-mavlink-server, voxl-rangefinder-server, voxl-portal, voxl-streamer, voxl-mavcam-manager, voxl-cpu-monitor
• voxl-qvio-server: disabled. voxl-mapper: started manually for these tests, otherwise disabled.
• OpenVINS config: use_stereo false, max_cameras 2, track_frequency 30.0, imu_body_frame_mode true
• voxl-mapper.conf: defaults (voxel_size 0.1, esdf_max_distance 2, robot_radius 0.35, tof_0_rate 10, tof pipe /run/mpa/tof enabled, stereo depth pipes disabled)

What I am trying to do:
Build a map with voxl-mapper so we can move on to trajectory/planning based autonomy. VIO and manual autonomy already work on this airframe: on the same day we flew a manual Position-mode flight and an OFFBOARD figure_eight flight, and EKF fused vision at 99-100% (cs_ev_pos / cs_ev_hgt / cs_ev_vel / cs_ev_yaw) with 0.14 m horizontal drift over 20 s of stick-neutral hover.

What actually happens:
voxl-mapper starts and connects, but never produces any map output — and starting it also halves the VIO rate, which then does not recover when voxl-mapper is stopped.

1. With voxl-mapper stopped, voxl-inspect-vio reports dt = 33.3 ms (30 Hz).
2. Starting voxl-mapper drops it to 66.7 ms (15 Hz), with almost no jitter (spread ~0.1 ms). The raw ov pipe reads 15 Hz too, so this is not a voxl-vision-hub stage.
3. Stopping voxl-mapper does not restore 30 Hz. At that moment feature count (20) and quality (97%) are normal, so it is not feature starvation.
4. systemctl restart voxl-open-vins-server restores 30 Hz every time (reproduced 3x).
5. Running voxl-mapper in the foreground, initialisation completes normally:

Loading our own config file
Loading extrinsics config file
Trying to init tsdf server
created tsdf server
Connected to VIO server
Initializing ESDF structs
Connected to depth pipe
ESDF Thread is now locked to the following cores: 4 5 6

and then it repeats, indefinitely:

WARNING the requested timestamp was too new, VIO may have stopped
ERROR in pipe_server_write_point_cloud, received NULL data pointer

6. voxl-inspect-points voxl_mapper_aligned_ptcloud shows the header only, no data rows, while the ToF input pipe tof_pc is healthy at 43,200 points.

Note: "VIO may have stopped" looks like a misdiagnosis — VIO is running the whole time, just at half rate.

What we ruled out (all measured on the vehicle):

• CPU load: voxl-mapper uses 1.9-5.7% CPU, the system is ~72% idle (8 cores), and no other service rises when it starts.
• Config drift: voxl-camera-server.conf and voxl-open-vins-server.conf are byte-identical to a snapshot taken before voxl-mapper was installed.
• Camera input: tracking_front is a steady 30.0 fps (exposure ~7.4 ms); the processed pipes (misp_grey, misp_norm) are also 30 fps. So input is 30 Hz while only the OpenVINS output is halved.
• Clock mismatch: VIO and ToF timestamps share the same monotonic base. Sampled simultaneously they differ by ~137 ms (VIO 2841.870 s, ToF 2841.733 s).
• Extrinsics: body -> tof is defined in extrinsics.conf and voxl-mapper logs no extrinsics error.
• Subscriber count: two simultaneous voxl-inspect-vio clients do not change the rate (still 30 Hz).

Steps to reproduce:

1. systemctl restart voxl-open-vins-server, wait ~30 s, confirm voxl-inspect-vio shows 33.3 ms.
2. systemctl start voxl-mapper.
3. voxl-inspect-vio now shows 66.7 ms.
4. systemctl stop voxl-mapper — still 66.7 ms.
5. systemctl restart voxl-open-vins-server — back to 33.3 ms.

Questions:

1. Is there a known condition under which voxl-open-vins-server latches to half its configured rate and does not return until restarted?
2. Is a rate change expected when another process subscribes to the VIO pipe?
3. Is the "requested timestamp was too new" failure a consequence of the 15 Hz rate, or an independent problem? If we could hold 30 Hz, would voxl-mapper work?
4. Is there a supported way to detect or recover this state without restarting voxl-open-vins-server?

Logs / pictures: Happy to provide the full system snapshot (voxl-version, voxl-inspect-services, full PX4 parameter dump) and the .ulg flight logs on request.

[Zachary Lowell 0]

Hi! Planning on testing and getting into the weeds of this tomorrow to see if I can recreate and provide proper answers for this!

Zach