VIO errors with new AR0144 tracking camera unstable flight/crashing

Gary Holmgren

Hello, I recently upgraded from the older MSU-M0014 tracking camera to the AR0144 based sensor, and the flight on our drone is now very unstable and crashes constantly

Were running a voxl2 in a custom configuration as shown here

J6_LOWER_SENSOR="ar0144"
J6_LOWER_NAME="tracking"
J6_LOWER_ROTATE="true"


J6_UPPER_SENSOR=""
J6_UPPER_NAME=""
J6_UPPER_ROTATE="false"


J7_LOWER_SENSOR="pmd-tof-liow2"
J7_LOWER_NAME="tof"
J7_LOWER_ROTATE="false"


J7_UPPER_SENSOR=""
J7_UPPER_NAME=""
J7_UPPER_ROTATE="false"


J8_LOWER_SENSOR="imx214"
J8_LOWER_NAME="hires"
J8_LOWER_ROTATE="true"


J8_UPPER_SENSOR=""
J8_UPPER_NAME=""
J8_UPPER_ROTATE="false"

While I understand there is no guarantee this should work, we had it working with the older tracking camera perfectly fine, we just wanted a higher-resolution sensor.

The tracking camera does calibrate to ~.4 projection error and passes consistently. also, the extrinsic has been quadruple checked to be accurate as nothing changed from the location of the previous tracking camera which did fly stably.

I followed the troubleshooting vio tips and saw there are some issues here

this is before takeoff

and you can see whenever the drone takes off in position mode it has this error

it goes back and forth between okay and sometimes bam_cam_cal

which is confusing because it succeeded in the camera calibration
I used a glass panel with a low glare print of a 5x4 chessboard from opencv with 40mm squares and made sure it was perfectly flat by using adhesive and a roller. I don't know how it could get any flatter.

I used: voxl-calibrate-camera tracking -f -s 5x4 -l 0.04

Vibration is certainly not an issue as the stack is isolated and it worked before.

The qvio overlay in the portal is very laggy and even during calibration, this I'm sure is due to the higher resolution, could this be the problem? and if it is, is the voxl2 capable of handling this resolution for tracking?

I am happy to share more config stuff if needed as I cannot figure out what the problem is.

Alex Kushleyev

@Gary-Holmgren ,

Here are a few suggestions to help debug this:

I noticed that you have a 180 degree rotation specified for the camera config. This rotation is achieved by reversing the readout order from the camera, which affects the intrinsic calibration. Make sure you calibrate your camera's intrinsics with the same rotation (enabled or disabled) as you are testing, otherwise the intrinsics will not be valid.
whenever working on a new setup, always test VIO in hand-held tests in order to eliminate other variables. If VIO does not work in hand-held tests, there is no hope that it will work in flight
do you ever have cases where VIO works for some time, maybe 5-10 seconds while you are moving (and then blows up) or is it always blowing up as soon as you start moving? If the latter is the case, majority of the cases is incorrect extrinsic parameters (specifically, camera to IMU rotation). did you calculate your extrinsics based on rotated or not rotated camera configuration?
please share the results of your intrinsic calibration, just to double check it
double check camera focus, anything farther than 6 inches or so should be in clear focus (fisheye lenses have very short focal distance and large depth of field)

Alex

Gary Holmgren

@Alex-Kushleyev
Thankyou for the advice!

There are few seconds of flight where the drone does want to stay in place and VIO seems to be working but the drift then starts to go haywire and I have to fight the controls to keep it in one spot, I am testing this in a very well lit feature-rich space as well.

When you said "Make sure you calibrate your camera's intrinsics with the same rotation (enabled or disabled)" did you mean to calibrate after all these parameters are set correctly? I calibrated the camera after the hardware was all installed in the drone with the rotate flag set in the custom can config. Or are you implying there is a rotate flat you have to specify when doing the actual calibration procedure ex: voxl-calibrate-camera tracking -f -s -l... or in the intrinsic YAML file there needs to be a rotate parameter?

Maybe you can see what might have gone wrong, I got a fairly good calibration recently so here are my results.
Also, I am running px4 off the flight core V2 connected to voxl2 over uart. the parameters and everything else stayed the same when switching to the AR0144 tracking sensor so I don't think anything is wrong there. Just to reiterate I had the older tracking camera working with follow tag as well and it was very stable so the only issues I believe could be causing this are related to the new AR0144 setup I have. I am also on the newest SDK 1.3.0 as it just came out a couple of days ago.

For reference here is a pic of the cameras mounting configuration to make sense of the extrinsics config

I am under the assumption the bottom "positive Y" axis of the camera is where the MIPI cables come out from which seems to be the convention with all of the image sensors I've used. thus I have rotated the Z on the tracking cam to -90 so the MIPI points upwards and the X axis is pointing out the left side of the drone.

In order to install the tracking cam the lens does have to be taken off but it's refocused during installation I ensured there's no contamination on the sensor or lens during install as well.

Below are my current extrinsics and intrinsics.

	"extrinsics":	[{
			"parent":	"imu_apps",
			"child":	"tracking",
			"T_child_wrt_parent":	[0.0501, 0.0060, 0.0093],
			"RPY_parent_to_child":	[0, 45, -90]
		}, {
			"parent":	"body",
			"child":	"imu_apps",
			"T_child_wrt_parent":	[0.1167, 0.0060, 0.0053],
			"RPY_parent_to_child":	[0, 0, 0]
		}, {
			"parent":	"body",
			"child":	"tof",
			"T_child_wrt_parent":	[0.1752, 0, 0.0034],
			"RPY_parent_to_child":	[0, 90, -90]
		}, {
			"parent":	"body",
			"child":	"ground",
			"T_child_wrt_parent":	[0, 0, 0.0328],
			"RPY_parent_to_child":	[0, 0, 0]
		}]
}

%YAML:1.0
---
M: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 5.6009618399934220e+02, 0., 6.4970452549577044e+02, 0.,
       5.6164176916165263e+02, 4.0012359829168020e+02, 0., 0., 1. ]
D: !!opencv-matrix
   rows: 4
   cols: 1
   dt: d
   data: [ -4.7175359821771193e-02, 3.8442932109403055e-02,
       -1.7319222703866245e-02, -3.5236303868186605e-04 ]
reprojection_error: 3.0046768943236762e-01
width: 1280
height: 800
distortion_model: fisheye
calibration_time: "2024-06-04 22:33:29"

Gary Holmgren

@Alex-Kushleyev

Here is a quick takeoff video I screen-recorded with the qvio inspector so you can maybe get a better idea of what's going on.

https://drive.google.com/file/d/1tThv3vlY8RCXSmvu3UycyTQ6EDm7ZxVB/view?usp=sharing

Alex Kushleyev

@Gary-Holmgren , I think your imu -> tracking camera rotation should be:

"RPY_parent_to_child": [0, 45, 90] (+90, not -90). Even though your flex is pointing up, you have specified in camera server config to rotate the image (in the camera ISP itself).

Try that..

What i meant about the calibration / rotation.. make sure that you set the "en_rotate": true in voxl-camera-server.conf and then perform the camera calibration, so that you calibrate the camera with the correct internal (to camera) image rotation. If you perform intrinsics calibration and then change the internal camera rotation, the calibration will not be correct (principal points will be flipped, at least).

I checked, the video, it basically shows ZERO feature tracking, so the transform issue is probably causing this.

Alex

Gary Holmgren

@Alex-Kushleyev Will try this! thank you very much for taking a look. will report back with the result.

Gary Holmgren

@Alex-Kushleyev Hi Alex, I flipped the extrinsic to +90 for the tracking and it fixed the issue, the recalibration worked first try and the flight is now stable. Your help was much appreciated!

Alex Kushleyev

@Gary-Holmgren , great! The transforms can be tricky sometimes especially with the added complexity of the image rotation in the camera…

Alex