voxl-mavlink-server incorrectly sets UTC clock from GPS time
-
We have experienced an issue where the system clock of the voxl is set based on a system time message received via mavlink which causes all kinds of timing issues resulting in an immediate drone landing/crash. The time which is set is incorrect. In one instance I even saw that it sets it to a day ahead. Note that it only sometimes happens. Most of our flights don't have this issue but when it happens it is a direct problem.
In our logs we see the following log lines
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:15 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:16 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:17 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:18 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:19 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:20 2025 Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:21 2025The software versions on this drone are:
──────────────────────────────────────────────────────────────────────────────── system-image: 1.8.02-M0054-14.1a-perf kernel: #1 SMP PREEMPT Mon Nov 11 22:47:44 UTC 2024 4.19.125 ──────────────────────────────────────────────────────────────────────────────── hw platform: M0054 mach.var: 1.0.1 SKU: MCCA-M0054-C28-T0-M0-X0 ──────────────────────────────────────────────────────────────────────────────── voxl-suite: 1.4.5 ────────────────────────────────────────────────────────────────────────────────Service Name | Version | Enabled | Running | CPU Usage ─────────────────────────────────────────────────────────────────────────────── docker-autorun | 1.3.1 | Disabled | Not Running | modallink-relink | 1.1.6 | Disabled | Not Running | voxl-camera-server | 2.2.1 | Enabled | Not Running | voxl-cpu-monitor | 0.6.0 | Enabled | Running | 0.0% voxl-dfs-server | 0.2.0 | Disabled | Not Running | voxl-elrs-startup | 0.4.7 | Disabled | Not Running | voxl-feature-tracker | 0.5.2 | Disabled | Not Running | voxl-flow-server | 0.3.6 | Disabled | Not Running | voxl-imu-server | 1.1.3 | Enabled | Not Running | voxl-io-server | 0.0.5 | Disabled | Not Running | voxl-lepton-server | 1.3.3 | Disabled | Not Running | voxl-lepton-tracker | 0.0.4 | Disabled | Not Running | voxl-logger | 0.5.3 | Disabled | Not Running | voxl-mavcam-manager | 0.5.8 | Disabled | Not Running | voxl-mavlink-server | 1.4.7 | Enabled | Running | 3.8% voxl-modem | 1.1.6 | Disabled | Not Running | voxl-open-vins-server | 0.3.12 | Enabled | Not Running | voxl-osd | 0.1.7 | Disabled | Not Running | voxl-portal | 0.7.10 | Disabled | Not Running | voxl-px4-imu-server | 0.1.2 | Disabled | Not Running | voxl-px4 | 1.14.0 | Enabled | Running | 19.2% voxl-qvio-server | 1.2.0 | Disabled | Not Running | voxl-rangefinder-server | 0.1.5 | Disabled | Not Running | voxl-remote-id | 0.0.9 | Disabled | Not Running | voxl-seek-server | 0.3.12 | Disabled | Not Running | voxl-softap | 0.3.5 | Disabled | Not Running | voxl-state-estimator | 0.0.5 | Disabled | Not Running | voxl-static-ip | 0.3.5 | Disabled | Not Running | voxl-streamer | 0.7.5 | Disabled | Not Running | voxl-tag-detector | 0.0.4 | Disabled | Not Running | voxl-tflite-server | 0.4.1 | Disabled | Not Running | voxl-time-sync | 1.4.7 | Disabled | Not Running | voxl-uvc-server | 0.1.7 | Disabled | Not Running | voxl-vision-hub | 1.8.21 | Enabled | Not Running | voxl-vrx | 1.3.14 | Disabled | Not Running | voxl-vtx | 1.3.14 | Disabled | Not Running | voxl-wait-for-fs | 1.4.7 | Enabled | Completed |Where voxl-camera-server and voxl-open-vins-server are our own forked versions with some minor changes.
We don't have GPS enabled on our drone:
px4-param show -a | grep GPS ADSB_GPS_OFF_LAT [-1,5] : 0 ADSB_GPS_OFF_LON [-1,6] : 0 x COM_ARM_WO_GPS [208,483] : 1 x EKF2_GPS_CHECK [304,586] : 245 x + EKF2_GPS_CTRL [305,587] : 0 x EKF2_GPS_DELAY [306,588] : 110.0000 x EKF2_GPS_POS_X [307,589] : 0.0000 x EKF2_GPS_POS_Y [308,590] : 0.0000 x EKF2_GPS_POS_Z [309,591] : 0.0000 x EKF2_GPS_P_GATE [310,592] : 5.0000 x EKF2_GPS_P_NOISE [311,593] : 0.5000 x EKF2_GPS_V_GATE [312,594] : 5.0000 x EKF2_GPS_V_NOISE [313,595] : 0.3000 x EKF2_REQ_GPS_H [357,641] : 10.0000 FW_GPSF_LT [-1,721] : 30 FW_GPSF_R [-1,722] : 15.0000 GPS_1_GNSS [-1,841] : 0 GPS_1_PROTOCOL [-1,842] : 1 GPS_2_GNSS [-1,843] : 0 GPS_2_PROTOCOL [-1,844] : 1 GPS_DUMP_COMM [-1,845] : 0 GPS_PITCH_OFFSET [-1,846] : 0.0000 GPS_SAT_INFO [-1,847] : 0 GPS_UBX_BAUD2 [-1,848] : 230400 GPS_UBX_CFG_INTF [-1,849] : 0 x + GPS_UBX_DYNMODEL [418,850] : 6 GPS_UBX_MODE [-1,851] : 0 GPS_YAW_OFFSET [-1,852] : 0.0000 LPE_GPS_DELAY [-1,937] : 0.2900 LPE_GPS_VXY [-1,938] : 0.2500 LPE_GPS_VZ [-1,939] : 0.2500 LPE_GPS_XY [-1,940] : 1.0000 LPE_GPS_Z [-1,941] : 3.0000 x MAV_USEHILGPS [456,999] : 0 SENS_EN_GPSSIM [-1,1448] : 0 SENS_GPS_MASK [-1,1484] : 7 SENS_GPS_PRIME [-1,1485] : 0 SENS_GPS_TAU [-1,1486] : 10.0000 SIM_GPS_USED [-1,1555] : 10 x + SYS_HAS_GPS [764,1658] : 0 UAVCAN_SUB_GPS [-1,1930] : 1 UCAN1_GPS0_SUB [-1,2048] : -1 UCAN1_GPS1_SUB [-1,2049] : -1 UCAN1_GPS_PUB [-1,2050] : -1 UCAN1_UORB_GPS [-1,2053] : -1 UCAN1_UORB_GPS_P [-1,2054] : -1Have you seen this issues before and/or do you have any idea how this could be caused?
-
@Tjark The drone shouldn't land or crash when the system time is updated. To disable GPS on the drone edit the /etc/modalai/voxl-px4.conf file to set GPS=NONE. Is the system time message coming from PX4 or from a GCS?
-
@Eric-Katzfey Thanks for your prompt response! I think it is going down because it gets out of offboard mode because when the time is updated some of our processes also don't work properly anymore. But I don't know if it is only our processes which are affected.
The drone already had GPS disabled when this happened:
AIRFRAME=MULTICOPTER GPS=NONE RC=CRSF_MAV ESC=VOXL_ESC POWER_MANAGER=VOXLPM AIRSPEED_SENSOR=NONE DISTANCE_SENSOR=NONE OSD=DISABLE DAEMON_MODE=ENABLE SENSOR_CAL=ACTUAL ARTIFACT_MODE=DISABLE EXTRA_STEPS=()We are not connected to a GCS. Just PX4. We haven't seen this issue on sdk 1.1.3 (our previous version) but sometimes on sdk 1.4.5 (our current version) and also on multiple drones.
-
@Tjark Probably don't want to apply the offset if clock is REALTIME here: https://github.com/modalai/px4-firmware/blob/96011c76dc9fffb2f2a2528df1767cdf926b24ce/platforms/posix/src/px4/common/drv_hrt.cpp#L464
-
@Eric-Katzfey Only for MONOTONIC. I'll look at this more and try to figure out the correct thing to do.
-
@Eric-Katzfey Ok, thanks. Looking forward to it. For now, we have put a forked version of voxl-mavlink-server on 1 drone which had this issue a few times. This forked version will log the requested time offset instead of applying it. If we get a log from that, we will post it here if that helps.
-
@Tjark There is an updated voxl-px4 here: http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/voxl-px4_1.14.0-2.0.116-202510301051_arm64.deb. You will also need an updated libfc-sensor library here: http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/libfc-sensor_1.0.8-202508141142_arm64.deb. Can you try to update with these 2 and see if it solves the issue?
-
@Eric-Katzfey Thanks for your update. We tried it on one of our drones but we had some strange issues regarding invalid setpoints afterwards and we think it may be caused by a voxl-px4 version mismatch. We were running version 1.14.0-2.0.98 of voxl-px4 and the version you provided is 1.14.0-2.0.116. Would it be difficult to provide a patch for version 1.14.0-2.0.98?
We also had our first log of that it occurred again with our forked voxl-mavlink-server. The drone continued its flight now which is good. This is the changed function:
void mavlink_services_handle_system_time(mavlink_message_t* msg) { // collect and consolidate information int64_t time_now_ns = my_time_realtime_ns(); int64_t time_now_s = time_now_ns / 1e9; int64_t gps_time_us = mavlink_msg_system_time_get_time_unix_usec(msg); int64_t gps_time_s = gps_time_us / 1e6; // check gps time is valid if(gps_time_s == 0) return; bool within_10_seconds_original = abs(time_now_s - gps_time_s)<10; bool within_10_seconds_correct = llabs(time_now_s - gps_time_s)<10; if (within_10_seconds_original != within_10_seconds_correct) { printf( "WARNING! The check if we are within 10 seconds gives an incorrect result due to wrong usage of abs.\n" "Original: %s, Correct: %s (time_now_s=%lld, gps_time_s=%lld, diff_original=%d, diff_correct=%lld)\n", within_10_seconds_original ? "true" : "false", within_10_seconds_correct ? "true" : "false", (long long)time_now_s, (long long)gps_time_s, abs(time_now_s - gps_time_s), // truncated 32-bit diff llabs(time_now_s - gps_time_s) // correct 64-bit diff ); } // if we are within 10 seconds, good enough if(within_10_seconds_correct){ if(time_needs_setting){ printf("detected system time has already been set\n"); } time_needs_setting = 0; return; } else time_needs_setting = 1; if(!time_needs_setting) return; struct timespec ts_now; ts_now.tv_sec = time_now_s; ts_now.tv_nsec = 0; struct timespec ts_gps; ts_gps.tv_sec = gps_time_s; ts_gps.tv_nsec = 0; printf("WARNING! System wants to use GPS system time message to set UTC clock from %s to: %s", asctime(gmtime(&ts_now.tv_sec)), asctime(gmtime(&ts_gps.tv_sec))); printf("We will ignore this system time message and we won't update the clock"); // if (clock_settime(CLOCK_REALTIME, &ts) < 0) { // perror("Failed to set system time to GPS time"); // return; // } time_needs_setting = 0; return; }And we got log lines like this:
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov 7 09:07:29 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: to: Fri Nov 7 09:07:29 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov 7 09:07:30 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: to: Fri Nov 7 09:07:30 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov 7 09:07:31 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: to: Fri Nov 7 09:07:31 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov 7 09:07:32 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: to: Fri Nov 7 09:07:32 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov 7 09:07:33 2025 Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: to: Fri Nov 7 09:07:33 2025but I don't yet understand why it is triggering this log because both timestamps seem to be identical. Maybe you have an idea?
