voxl-mapper produces no map; VIO output rate halves to 15 Hz and does not recover until OpenVINS restart

[김진혁]

Hardware: Starling 2 Max, SKU MRB-D0012-4-V4-C29-T8-M36, hw platform M0054 (C29 config with PMD ToF). Received 14 Sep 2026, factory state.

Software version: (voxl-version)

• system-image 1.8.08-M0054-14.1a-perf, kernel 4.19.125
• voxl-suite 1.6.4~beta5 (factory; SDK not re-flashed)
• voxl-mapper 0.2.4, voxl-voxblox 1.1.5, voxl-nlopt 2.5.0-5 (installed via apt)

Configuration / services: (voxl-inspect-services)

• Running: voxl-camera-server, voxl-imu-server, voxl-open-vins-server, voxl-px4, voxl-vision-hub, voxl-mavlink-server, voxl-rangefinder-server, voxl-portal, voxl-streamer, voxl-mavcam-manager, voxl-cpu-monitor
• voxl-qvio-server: disabled. voxl-mapper: started manually for these tests, otherwise disabled.
• OpenVINS config: use_stereo false, max_cameras 2, track_frequency 30.0, imu_body_frame_mode true
• voxl-mapper.conf: defaults (voxel_size 0.1, esdf_max_distance 2, robot_radius 0.35, tof_0_rate 10, tof pipe /run/mpa/tof enabled, stereo depth pipes disabled)

What I am trying to do:
Build a map with voxl-mapper so we can move on to trajectory/planning based autonomy. VIO and manual autonomy already work on this airframe: on the same day we flew a manual Position-mode flight and an OFFBOARD figure_eight flight, and EKF fused vision at 99-100% (cs_ev_pos / cs_ev_hgt / cs_ev_vel / cs_ev_yaw) with 0.14 m horizontal drift over 20 s of stick-neutral hover.

What actually happens:
voxl-mapper starts and connects, but never produces any map output — and starting it also halves the VIO rate, which then does not recover when voxl-mapper is stopped.

1. With voxl-mapper stopped, voxl-inspect-vio reports dt = 33.3 ms (30 Hz).
2. Starting voxl-mapper drops it to 66.7 ms (15 Hz), with almost no jitter (spread ~0.1 ms). The raw ov pipe reads 15 Hz too, so this is not a voxl-vision-hub stage.
3. Stopping voxl-mapper does not restore 30 Hz. At that moment feature count (20) and quality (97%) are normal, so it is not feature starvation.
4. systemctl restart voxl-open-vins-server restores 30 Hz every time (reproduced 3x).
5. Running voxl-mapper in the foreground, initialisation completes normally:

Loading our own config file
Loading extrinsics config file
Trying to init tsdf server
created tsdf server
Connected to VIO server
Initializing ESDF structs
Connected to depth pipe
ESDF Thread is now locked to the following cores: 4 5 6

and then it repeats, indefinitely:

WARNING the requested timestamp was too new, VIO may have stopped
ERROR in pipe_server_write_point_cloud, received NULL data pointer

6. voxl-inspect-points voxl_mapper_aligned_ptcloud shows the header only, no data rows, while the ToF input pipe tof_pc is healthy at 43,200 points.

Note: "VIO may have stopped" looks like a misdiagnosis — VIO is running the whole time, just at half rate.

What we ruled out (all measured on the vehicle):

• CPU load: voxl-mapper uses 1.9-5.7% CPU, the system is ~72% idle (8 cores), and no other service rises when it starts.
• Config drift: voxl-camera-server.conf and voxl-open-vins-server.conf are byte-identical to a snapshot taken before voxl-mapper was installed.
• Camera input: tracking_front is a steady 30.0 fps (exposure ~7.4 ms); the processed pipes (misp_grey, misp_norm) are also 30 fps. So input is 30 Hz while only the OpenVINS output is halved.
• Clock mismatch: VIO and ToF timestamps share the same monotonic base. Sampled simultaneously they differ by ~137 ms (VIO 2841.870 s, ToF 2841.733 s).
• Extrinsics: body -> tof is defined in extrinsics.conf and voxl-mapper logs no extrinsics error.
• Subscriber count: two simultaneous voxl-inspect-vio clients do not change the rate (still 30 Hz).

Steps to reproduce:

1. systemctl restart voxl-open-vins-server, wait ~30 s, confirm voxl-inspect-vio shows 33.3 ms.
2. systemctl start voxl-mapper.
3. voxl-inspect-vio now shows 66.7 ms.
4. systemctl stop voxl-mapper — still 66.7 ms.
5. systemctl restart voxl-open-vins-server — back to 33.3 ms.

Questions:

1. Is there a known condition under which voxl-open-vins-server latches to half its configured rate and does not return until restarted?
2. Is a rate change expected when another process subscribes to the VIO pipe?
3. Is the "requested timestamp was too new" failure a consequence of the 15 Hz rate, or an independent problem? If we could hold 30 Hz, would voxl-mapper work?
4. Is there a supported way to detect or recover this state without restarting voxl-open-vins-server?

Logs / pictures: Happy to provide the full system snapshot (voxl-version, voxl-inspect-services, full PX4 parameter dump) and the .ulg flight logs on request.

[Zachary Lowell 0]

Hi! Planning on testing and getting into the weeds of this tomorrow to see if I can recreate and provide proper answers for this!

Zach

[Zachary Lowell 0]

Can you put your voxl2 into performance mode and test the 15 vs 30hz please! voxl-configure-cpu perf

[Zachary Lowell 0]

Ok lastly - can you please list out your OpenVINS package version? Previous versions of vins has a callback that alternates between processing and dropping frames when VIO is intiialized and its static jerk test permits throttling - so that first the 66.7 ms spacing you described. Restarting it would also bypass this throttle during init which is why it would go back to 30 hz

[김진혁]

Hi Zach, thanks for looking into this.

OpenVINS package version

voxl-open-vins-server 0.6.1 — that is the only OpenVINS-related package installed on this unit (dpkg -l | grep -i vins returns exactly one row; there is no separate open-vins library package). Still factory voxl-suite 1.6.4~beta5, never re-flashed.

Performance mode test

voxl-configure-cpu does not exist on this image, so I used voxl-set-cpu-mode perf (governor went schedutil -> performance, confirmed via /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor). Please tell me if you meant a different tool.

The result is that I could not reproduce the problem at all today — in either CPU mode.

Condition	VIO dt	voxl-mapper output
Fresh boot, schedutil, mapper off	33.3 ms	-
schedutil, mapper off, 2 min	33.3 ms (320 samples)	-
schedutil, mapper on, 2 min	33.3 ms (320 samples)	working
performance, mapper off	33.3 ms	-
performance, mapper on, 90 s	33.3 ms (240 samples)	working
performance, restart voxl-open-vins-server x5	33.3 ms every time (400 samples)	-

23 measurements, ~1,760 samples, all 33.3 ms. Not a single 15 Hz reading.

More importantly: voxl-mapper works now.

voxl-inspect-points voxl_mapper_aligned_ptcloud produces real data — ~1,900 points per cloud, roughly every 500 ms in schedutil. On 22 Sep the same pipe showed the header and zero data rows. There are no pipe_server_write_point_cloud, received NULL data pointer and no requested timestamp was too new, VIO may have stopped messages in the journal at all today. voxl-mapper CPU is now ~100% of one core, versus the 1.9% I reported — it was low then because it was not actually doing any work.

Performance mode does help the mapper, even though it did not change the VIO rate: the aligned point cloud comes out roughly every 95-190 ms instead of ~500 ms.

What this suggests about the causality

I think I had the direction backwards in my first post. The likely chain is:

OpenVINS comes up at 15 Hz -> mapper cannot match pose timestamps ("requested timestamp was too new") -> no map output

rather than "mapper is started -> VIO halves". That matches your static-jerk-throttle explanation: whatever decides the throttle is decided at VIO init, and on 22 Sep the vehicle happened to be in that state for the whole session (my very first reading that day, right after boot and before voxl-mapper had ever been started, was already 15 Hz).

Nothing was changed in between: dpkg status and voxl-open-vins-server.conf / voxl-camera-server.conf are untouched since 21 Sep. So the difference is runtime state, not configuration.

One observation that may not fit the throttle theory

Today the vehicle is completely static and features reads 0, yet VIO runs at 30 Hz. On 22 Sep it was showing features 20 and quality 97% while stuck at 15 Hz. If the throttle is permitted by a static-jerk test, I would have expected today's fully static case to be the one that throttles. Is the test based on IMU motion rather than tracked features, and is there any way to see its decision at runtime?

Remaining questions

1. Is there a log line, pipe field, or debug flag that shows whether the throttle is currently engaged? Right now the only way I can tell is by timing the output, and the state is invisible in state / quality / features.
2. Was the throttle behaviour changed after 0.6.1? If a newer voxl-open-vins-server fixes it, which SDK version should we move to?
3. Given it is intermittent, is it safe to treat "dt == 33.3 ms" as a pre-flight check, or is there a more reliable condition we should test before an autonomous flight?

I will keep the vehicle in performance mode and report back if the 15 Hz state shows up again, with the full journal from that boot.

Thanks,
JinHyeok