Compass Errors
-
On a recent Sentinel Drone test flight, the compass heading switched 180 degrees while in flight. All preflight checks were accomplished without any errors being detected. Approximately 2 and a half minutes into flight, the drone began an uncommanded right-hand turn. The drone was sent an RTL message, which it reported as received, but it did not respond. the drone continued to turn right at a faster rate and crashed. Review of the error logs after the crash showed that the drone yaw orientation prior to the error was =83 degrees. the Yaw after the error was a negative 89 degrees (-89 degrees). The logs indicated there were multiple compass errors requiring a calibration in mid-flight. the error messages were two "compass needs calibration - land now", followed by two "GPS spoofing detected", and then followed by two more "compass nees calibration - land now" messages. The elapsed time from the first message to the last message was 8 seconds. None of these messages were transmitted to the ground station and were found during the post-crash investigation.
Additional Information: The drone was a VTOL sUAS ModalAI Sentinel Drone utilizing a Remote Control Spektrum DSMX Reciever 2.4GHZ, WiFi Short Range IP Conectivity for nearby flights and a 5G LTE Longe Range BLOS capability. Navigation Sources were the Ublox Neo-M8N module GP. Navigation modes included Fail Safe Mode Px4 Safety Configuration and Manual Remote Control.
Any thoughts on what would have caused the compass to flip 180 degrees in flight and steps to take to prevent this from happening again? Have any other users reported this type of error?
-
On a recent Sentinel Drone test flight, the compass heading switched 180 degrees while in flight. All preflight checks were accomplished without any errors being detected. Approximately 2 and a half minutes into flight, the drone began an uncommanded right-hand turn. The drone was sent an RTL message, which it reported as received, but it did not respond. the drone continued to turn right at a faster rate and crashed. Review of the error logs after the crash showed that the drone yaw orientation prior to the error was =83 degrees. the Yaw after the error was a negative 89 degrees (-89 degrees). The logs indicated there were multiple compass errors requiring a calibration in mid-flight. the error messages were two "compass needs calibration - land now", followed by two "GPS spoofing detected", and then followed by two more "compass nees calibration - land now" messages. The elapsed time from the first message to the last message was 8 seconds. None of these messages were transmitted to the ground station and were found during the post-crash investigation.
Additional Information: The drone was a VTOL sUAS ModalAI Sentinel Drone utilizing a Remote Control Spektrum DSMX Reciever 2.4GHZ, WiFi Short Range IP Conectivity for nearby flights and a 5G LTE Longe Range BLOS capability. Navigation Sources were the Ublox Neo-M8N module GP. Navigation modes included Fail Safe Mode Px4 Safety Configuration and Manual Remote Control.
Any thoughts on what would have caused the compass to flip 180 degrees in flight and steps to take to prevent this from happening again? Have any other users reported this type of error?
You should look carefully at the flight log (using px4 flight review) and see what was happening to actual compass data just prior to the crash. You can look at the magnitude of magnetic field that should remain constant regardless of the vehicle orientation.
Also, heading change by close to 180 degrees does not mean that the compass readings themselves changed so much.
Were you flying in very cold weather? If so, it is possible that EKF filter in PX4 was not able to correctly track the IMU gyro biases (quickly changing while IMU was cooling down in flight) and incorrectly reported compass error (because it was disagreeing with IMU).
This may be a similar issue, please check out the analysis: https://discuss.px4.io/t/stopping-compass-mid-flight/42808
Alex
-
On a recent Sentinel Drone test flight, the compass heading switched 180 degrees while in flight. All preflight checks were accomplished without any errors being detected. Approximately 2 and a half minutes into flight, the drone began an uncommanded right-hand turn. The drone was sent an RTL message, which it reported as received, but it did not respond. the drone continued to turn right at a faster rate and crashed. Review of the error logs after the crash showed that the drone yaw orientation prior to the error was =83 degrees. the Yaw after the error was a negative 89 degrees (-89 degrees). The logs indicated there were multiple compass errors requiring a calibration in mid-flight. the error messages were two "compass needs calibration - land now", followed by two "GPS spoofing detected", and then followed by two more "compass nees calibration - land now" messages. The elapsed time from the first message to the last message was 8 seconds. None of these messages were transmitted to the ground station and were found during the post-crash investigation.
Additional Information: The drone was a VTOL sUAS ModalAI Sentinel Drone utilizing a Remote Control Spektrum DSMX Reciever 2.4GHZ, WiFi Short Range IP Conectivity for nearby flights and a 5G LTE Longe Range BLOS capability. Navigation Sources were the Ublox Neo-M8N module GP. Navigation modes included Fail Safe Mode Px4 Safety Configuration and Manual Remote Control.
Any thoughts on what would have caused the compass to flip 180 degrees in flight and steps to take to prevent this from happening again? Have any other users reported this type of error?
-
You should look carefully at the flight log (using px4 flight review) and see what was happening to actual compass data just prior to the crash. You can look at the magnitude of magnetic field that should remain constant regardless of the vehicle orientation.
Also, heading change by close to 180 degrees does not mean that the compass readings themselves changed so much.
Were you flying in very cold weather? If so, it is possible that EKF filter in PX4 was not able to correctly track the IMU gyro biases (quickly changing while IMU was cooling down in flight) and incorrectly reported compass error (because it was disagreeing with IMU).
This may be a similar issue, please check out the analysis: https://discuss.px4.io/t/stopping-compass-mid-flight/42808
Alex
-
@Alex-Kushleyev how cold is "very"? Ours crashed in November, probably between 30-40 F at the time of flight. Had been flying for a few minutes
@groupo , I know "cold" is a vague term, but i would say that if you have a rapid 20-30 degrees C change in temperature from idle (sitting on the ground) to flight, then it the is probably some risk of this issue happening. Rapid i mean within 1-2 minutes.
The number 180 degree is probably a coincidence because after the heading is wrong by more than 90 degrees, the vehicle will start an unstable swirling around (in position control mode). Unstable, meaning, that the trajectory of the vehicle will be a growing spiral (where if the heading is off by less than 90 degrees, it would be a shrinking spiral, but radius of the spiral growing as you get closer to 90 degrees).
When the heading is off between 90 and 180 degrees, the vehicle enters an unstable spiral (when you are in x-y gps position control mode), so probably by the end (crash) the yaw drifts closer to 180 degrees off. However, if you have the log, you can see at which point the behavior of the x-y position controller started to have spiral behavior (first stable spiral and then unstable spiral) and correlate that to the IMU temperature, if you have it, and see if the compass produced any inconsistent readings (like significantly changing magnitude of the magnetic field, which should stay constant regardless of the orientation of the vehicle).
I am not aware of this issue happening at ModalaAI during testing, so we don't have any logs and also default logging may not be sufficient to diagnose the issue to the fullest extent. You would need to look at the EKF status / output to see if there are any red flags.
Another way to debug this would be to look at the EKF's estimate of the IMU biases for gyro during the flight and separately perform the IMU calibration across the temperature range that was tested in flight). Then you could see if indeed EKF could not keep up with estimating the gyro bias drift in real time.
What I mean by "EKF not keeping up with estimating gyro bias drift in real time" is the following. EKF in PX4 has some sort of process model for how the gyro bias evolves over time. I am not 100% sure, but i don't think the EKF actually looks at the IMU temperature and uses it for estimating the gyro bias - it could be a random walk model or something like that (lets assume random walk). If the parameters of the random walk for the gyro bias drift model are conservative (the model assumes that the rate of the gyro bias drift "slowly"), then if the gyro bias indeed drifts faster than EKF assumes it can, the EKF will stop correctly tracking the bias drift and correctly subtracting the gyro bias from the raw gyro measurements.
Note that for x and y gyro bias, it is possible to estimate it using accelerometer readings (gravity vector), but the gravity vector does not provide any information for yaw drift of the vehicle (which is aligned with z axis gyro at hover). The compass comes into play here to help with this, since the magnetic field has horizontal and vertical component (unlike gravity only having vertical component w.r.t Earth). So ideally, the compass should aid with estimating the bias drift of the yaw gyro (z axis), however if the z axis gyro bias drift is larger than expected, the EKF can actually think that compass is malfunctioning, because typically the EKF trusts the gyro/accel more than other secondary sensors like compass and barometer, etc.
So, with this in mind, and keeping in mind that this analysis is an educated guess, the crash(es) may have resulted from bias from an uncalibrated IMU not being properly tracked during a sudden temperature drop of the IMU, resulting in compass being flagged as malfunctioning and heading drifting to the point that the vehicle starts spiraling out of control. However, looking at the actual compass data and imu bias data (if possible), local gps position, can provide more information.
If anyone has logs from their flights (if you want to strip out GPS coordinates, please do so, but having local position estimates would be good to see - look for spiraling behavior), you can share the log to help with the analysis..
Alex
-
@groupo , I know "cold" is a vague term, but i would say that if you have a rapid 20-30 degrees C change in temperature from idle (sitting on the ground) to flight, then it the is probably some risk of this issue happening. Rapid i mean within 1-2 minutes.
The number 180 degree is probably a coincidence because after the heading is wrong by more than 90 degrees, the vehicle will start an unstable swirling around (in position control mode). Unstable, meaning, that the trajectory of the vehicle will be a growing spiral (where if the heading is off by less than 90 degrees, it would be a shrinking spiral, but radius of the spiral growing as you get closer to 90 degrees).
When the heading is off between 90 and 180 degrees, the vehicle enters an unstable spiral (when you are in x-y gps position control mode), so probably by the end (crash) the yaw drifts closer to 180 degrees off. However, if you have the log, you can see at which point the behavior of the x-y position controller started to have spiral behavior (first stable spiral and then unstable spiral) and correlate that to the IMU temperature, if you have it, and see if the compass produced any inconsistent readings (like significantly changing magnitude of the magnetic field, which should stay constant regardless of the orientation of the vehicle).
I am not aware of this issue happening at ModalaAI during testing, so we don't have any logs and also default logging may not be sufficient to diagnose the issue to the fullest extent. You would need to look at the EKF status / output to see if there are any red flags.
Another way to debug this would be to look at the EKF's estimate of the IMU biases for gyro during the flight and separately perform the IMU calibration across the temperature range that was tested in flight). Then you could see if indeed EKF could not keep up with estimating the gyro bias drift in real time.
What I mean by "EKF not keeping up with estimating gyro bias drift in real time" is the following. EKF in PX4 has some sort of process model for how the gyro bias evolves over time. I am not 100% sure, but i don't think the EKF actually looks at the IMU temperature and uses it for estimating the gyro bias - it could be a random walk model or something like that (lets assume random walk). If the parameters of the random walk for the gyro bias drift model are conservative (the model assumes that the rate of the gyro bias drift "slowly"), then if the gyro bias indeed drifts faster than EKF assumes it can, the EKF will stop correctly tracking the bias drift and correctly subtracting the gyro bias from the raw gyro measurements.
Note that for x and y gyro bias, it is possible to estimate it using accelerometer readings (gravity vector), but the gravity vector does not provide any information for yaw drift of the vehicle (which is aligned with z axis gyro at hover). The compass comes into play here to help with this, since the magnetic field has horizontal and vertical component (unlike gravity only having vertical component w.r.t Earth). So ideally, the compass should aid with estimating the bias drift of the yaw gyro (z axis), however if the z axis gyro bias drift is larger than expected, the EKF can actually think that compass is malfunctioning, because typically the EKF trusts the gyro/accel more than other secondary sensors like compass and barometer, etc.
So, with this in mind, and keeping in mind that this analysis is an educated guess, the crash(es) may have resulted from bias from an uncalibrated IMU not being properly tracked during a sudden temperature drop of the IMU, resulting in compass being flagged as malfunctioning and heading drifting to the point that the vehicle starts spiraling out of control. However, looking at the actual compass data and imu bias data (if possible), local gps position, can provide more information.
If anyone has logs from their flights (if you want to strip out GPS coordinates, please do so, but having local position estimates would be good to see - look for spiraling behavior), you can share the log to help with the analysis..
Alex
-
W wilkinsaf referenced this topic on
-
@Alex-Kushleyev Thank you for the detailed breakdown! I am fairly certain OP is from another team on the same project and I believe we are describing the same event. He has custody of the drone/logs so I will let him follow up with you
@groupo We also had this issue, but we had modified the sentinel and was found to be more due to vibrations
Stopping Compass mid-flight
Hello, 2 weeks ago our 5 multicopters (voxl2 w/ m10 mro GPS/MAG) were working without fail. This week all 5 will stop listening to the compass mid flight. This results in a position hold that is not accurate, and eventu…
Dronecode Forum | Open Source Drone Development (discuss.px4.io)
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login