AprilTag relocalization fails with down-facing camera due to camera extrinsics mismatch (roll/pitch out of bounds)

kgn-mdlai

Re: Apriltag relocalization not relocalizing?

Hello, I have the following issue similar to one above about apriltag relocalization with fixed frame:

Platform / Version:
Starling 2 (C27)
voxl-suite v1.5.0
Front-facing camera used for VIO
Down-facing camera used for AprilTag detection

Issue
AprilTag relocalization works correctly when the tag detector runs on the front-facing camera, but consistently fails when running on the down-facing camera, even though:

The down-facing camera intrinsics are correct
The camera→body extrinsics in /etc/modalai/extrinsics.conf are correct
R_tag_to_fixed is correctly set for a flat ground tag R_tag_to_fixed= [[0, -1, 0], [1, 0, 0], [0, 0, 1]]
The tag is physically flat and the vehicle is level
With the down-facing camera, voxl-vision-hub repeatedly reports: "WARNING, apriltag roll/pitch out of bounds" does not relocalize.

Confirmation
Using the same tag, same fixed-frame configuration, and same environment:

Detection from front camera → relocalization succeeds.
Detection from down-facing camera → relocalization rejected.
Swapping only the camera input is sufficient to reproduce the failure.

Conclusion
This strongly suggests that the AprilTag relocalization path in voxl-vision-hub is composing R_tag_to_cam → R_cam_to_imu/body using the wrong camera extrinsics (appears to default to the front/tracking camera), rather than the extrinsics corresponding to the camera that actually produced the detection. As a result, the inferred R_tag_to_local ∘ R_fixed_to_tag contains spurious roll/pitch and is rejected by the ±10° gate.

Could you please solve the issue in new SDK? (You might add additional parameters in /etc/modalai/voxl-vision-hub.conf to select fixed frame pipe)

Best,
Kağan

ApoorvThapliyal

@kgn-mdlai Thank you for bringing this up, a fix to this is currently being worked on.
Thanks

Eric Katzfey

@kgn-mdlai Unfortunately, the April Tag code is quite outdated and hasn't been tested on any of our latest hardware. As it is now it is provided as reference code and if you are able to get things working correctly on your own we are happy to accept PRs to the repo. We do plan an update of this in a future revision of the SDK but there is no scheduled date that we can provide right now.