Occasional instability and crashes (possible positioning error)
-
Hi everyone,
From time to time, we experience difficulties in executing stable flights, whether a manual flight, a PX4 mission, or a Python-driven flight. At some point in the air, drones (I guess) lose their local position and go to (I cannot call this a drift as it happens very fast) a totally irrelevant position, then enter the descend mode and hit the ground. This happened multiple times. It is clear from the logs that there is something unusual, but we couldn't find the root cause as we have limited hands-on experience. Could anyone help us by reviewing the attached flight log belonging to a very short mission that ended up in an instant crash?
Log: https://review.px4.io/plot_app?log=0363540b-211b-4b83-a00e-2d3957565918
Best,
Umut
-
Here's today's update: we lost the drone, it is gone!
It was supposed to fly slowly in an orbit around a fixed point (r = 9 m) with altitude of 10 m. Though it started very well, after a shortwhile the reported altitude in QGC suddenly became -8.8 m (was precisely 10 m beforehand) and the drone started to gain altitude. It did not follow the land and return commands possibly due to the altitude error. The device continued to gain altitude until it disappeared at some point. It never turned back to the field although it had a 50% battery. Our mistake was obviously not forcing disarm, we forgot this with fuss and stuck to land. There is no log as the device was lost, but we have the previous one above.
Just to note, we were using WiFi, not the RC. This appears to be another bad decision but we are working on autonomous swarm ops, so we eventually cannot operate every single drone with an RC. Another note is GPS was active and healthy, EKF2 fusion included GPS data this time.
We apparently need urgent feedback on that one.
Best
-
If you look at the error messages in the log you posted, you can see it is losing GPS. Is there a chance you have poor GPS reception?
Look at the "Logged Messages" section. It implies you lost GPS.
-
Dear Chad,
Thanks for the reply. During that flight (which the log belongs to), we got signals from 14 sats and had stable GPS signal until the drone started rolling wildly around 3:11 just before crashing (please refer to the 3D flight demonstration - you can also verify this by looking at the very first plot under the map). Furthermore, at that time, EKF2 fusion excluded GPS data, relying on other positioning input like vision, baro, etc. (pretty much like an indoor mode). So, GPS had no effect on the strange behavior we experienced.
In the second flight where we lost the drone, GPS was included in the fusion this time as we first thought that this was the reason of such unstable movement, however it was apparently not. The drone somehow loses its position estimation even though GPS works correct.
-
VIO is environment dependent. Were you monitoring the QVIO Overlay to make sure vision was holding steady?
Check out this document on troubleshooting VIO
https://docs.modalai.com/flying-with-vio/#troubleshooting-vio
-
I'll definitely check that one out to see how stable the vision inputs are. However, in the second flight, when we included GPS in the EKF2 fusion, we excluded the vision inputs to isolate the problem. So it can't be the sole reason, either.
The result was that the drone flew away to the deep in the sky during a so-far pretty stable orbit move (unfortunately no logs for this one). Thus, I am thinking like we were occasionally getting faulty positioning data from IMU. Because we were orbiting at 10 m altitude, then suddenly alt reading changed to -8.8 m and the device started to rise possibly to reach 10 m again.
-
Still need some troubleshooting on the logs I've posted if anyone in the team have some time. Cheers to all!
