Troubleshooting/Improving VIO Accuracy
-
Hi all,
I have three Seeker drones that I am using for VIO/SLAM experimentation. All three are running fresh installs of VOXL SDK 0.9.5 (system image 4.0.0). The newest drone (~February 2023) can fly well with VIO for short periods but will sometimes lose its position estimate and crash indoors. The other two (late 2021/early 2022) perform significantly worse despite being configured the same. An interesting side note is that the PX4 parameters from the newest Seeker drone are NOT compatible with the PX4 parameters on the older two drones, which I pulled from the GitLab. I also included the EKF2 indoor VIO parameter file.
In particular, one of the older drones is not airworthy using VIO; it claims 100% VIO quality with 20-30 tracked features on the ground, but a few seconds after takeoff, it invariably loses its position estimate and becomes unstable. Inspecting the qvio service with voxl-inspect-qvio shows an immediate drop to -1% quality.
1.) Are there any important troubleshooting steps that I can take to improve the VIO on the drone that can't keep a position estimate? It has a fresh firmware image, fresh configuration, fresh PX4 parameters, and fresh IMU calibration.
2.) Are there any tips for improving VIO quality in general, even for the drone that works relatively well? I added small orange traffic cones on the floor of my test environment, but those don't seem to be tracked well by the VIO algorithm. I even added an 8'x4' foam wall with a black/white checkerboard pattern (1' diameter squares) for the camera to track, but VIO flight is still erratic at best. This video shows some of the instability, and note that it is often much more severe than what is shown.
Thanks in advance, and I'm happy to provide whatever information could be useful. Any advice is appreciated!
Best,
Nick B
Graduate student, The University of Arizona -
@nickyblanch What PX4 version is on each of the drones? Are they different?
-
There are troubleshooting vio tips here: https://docs.modalai.com/flying-with-vio-0_9/#troubleshooting-vio
-
@tom Thanks for your reply. I believe the PX4 versions are listed below:
VIO Challenged Drone #1
VIO Challenged Drone #2
VIO Successful Drone
It appears that the newer drone is on a different PX4 version than the two that are struggling - I thought that reflashing a new system image to the older two drones would have updated PX4 but I was obviously wrong. I can upgrade the PX4 firmware on the older two drones and see if that affects flight.
-
@Moderator Thank you for the tip. The vibration in particular could be a concern, I'll log some data and see.
-
@tom Hi Tom, in case you have any other suggestions, here is some more info on what is happening:
Sensors are calibrated.
VIO works like a charm up until takeoff in position mode; while on the ground or with the props not spinning, VIO quality is good and rarely fails even with quick movements. VIO immediately fails within the first second or two of drone takeoff.
There is data in both /data/modalai/opencv_tracking_intrinsics.yml and /data/modalai/voxl-imu-server.cal.
Extrinsics are configured for the FC1 seeker (VOXL cam).
The vibration analyzer in QGroundControl didn't show anything that appeared abnormal during flight, although I haven't totally ruled out vibration yet.
Altitude mode is also extremely unstable, with jumps of several feet in each direction.
I've had this drone working well with VIO in the past, running the same PX4 version. I guess it could also be a hardware fault but that is more difficult to diagnose.
Thanks!
-
@tom It's been a bit since the last update, but I after lots of fresh installs, calibrations, etc, I think the issue may be narrowed down to vibration. Shown below is the output of voxl-inspect-vibration for IMU0 during a relatively stable flight. The results are the same for IMU1.
It's clear that the vibration on the IMU is high, but is there anything I can do about it? The frame of the Seeker is unmodified, exactly as it came. Also, do you know if this vibration is high enough to cause occasional VIO failures?
Thanks,
Nick -
@nickyblanch that vibration does look high. We typically have much better on Seeker. Is the VOXL CAM portion loose or anything?
-
@Moderator Hi, thank you for your response. The VOXL cam is firmly attached to the frame, the four main mounting screws are tight and the rubber washers are under compression. I will go through and make sure everything else is tight, but it all appears to be. I swapped out the propellers for brand new ones yesterday.
I added foam padding between the VOXL CAM and the frame in an attempt to dampen vibration but it may have made it worse. VIO failed after about a minute of slow flying.
-
@Moderator Hi, just wanted to follow up - have you seen this problem before and/or how it was fixed? I tore the entire VOXL Cam portion apart, tightened everything, and re-secured the VOXL Cam to the airframe with a lot of thick foam between the VOXL and the airframe. Vibration still shows red (> 5.00m/s^2) after takeoff in Position Mode but is fine before takeoff. VIO flight technically works when moving slowly, but it is extremely unstable and the Seeker is constantly jerking up/down and left/right. Brand new propellers, motors appear to work fine. The drone has sustained a few crashes while indoors in the past.
I don't have the budget for a new VOXL cam and need to get the drone flying within the next week so I can finish my thesis and graduate - any suggestions? Is there possibly some software filtering in the VOXL sdk that I can check?
Thanks!
-
@nickyblanch I'll send an update here for anyone else with bad VIO quality that may be affected by vibration - tearing apart/reassembling the VOXL Cams may have helped a small amount, but the majority of the problem was definitely centered around the battery. The battery seemed fine and secure, but after adding foam all the way around the battery, the VIO quality improved significantly.
Curiously, although the vibration data is lower now, it still shows red ( > 5 m/s^2) in all three directions. I compared this against my brand new VOXL2 Seeker drone that doesn't have any VIO problems and that drone also shows red ( > 5 m/s^2) for vibration data. However, despite the high vibration, both the new VOXL2 drone and the older, previously unflyable VOXL1 drones fly semi-well in position mode.
-
@nickyblanch Hey Nicky Thank you for posting on here! I'm having issues with the VIO as well but my error is, "avoidance system not ready" with a "pre-arm check" error. This only happens when i enable obstetrical avoidance under safety on OGC. Was this occurring with you before by chance?
-
@AP123 Hi, I have not seen this before, but I also haven't tried enabling obstacle avoidance. I can't pretend to know much about the obstacle avoidance code running inn the VOXL SDK, but I suppose this error could be caused by some obstacle avoidance service not running on the VOXL when it should be, hence QGroundcontrol being confused.
