voxl-mavlink-server incorrectly sets UTC clock from GPS time

Tjark

We have experienced an issue where the system clock of the voxl is set based on a system time message received via mavlink which causes all kinds of timing issues resulting in an immediate drone landing/crash. The time which is set is incorrect. In one instance I even saw that it sets it to a day ahead. Note that it only sometimes happens. Most of our flights don't have this issue but when it happens it is a direct problem.

In our logs we see the following log lines

Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:15 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:16 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:17 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:18 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:19 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:20 2025
Oct 27 08:21:29 JOHN voxl-mavlink-server[2280]: setting UTC clock from GPS to: Mon Oct 27 08:20:21 2025

The software versions on this drone are:

────────────────────────────────────────────────────────────────────────────────
system-image: 1.8.02-M0054-14.1a-perf
kernel:       #1 SMP PREEMPT Mon Nov 11 22:47:44 UTC 2024 4.19.125
────────────────────────────────────────────────────────────────────────────────
hw platform:  M0054
mach.var:     1.0.1
SKU:          MCCA-M0054-C28-T0-M0-X0
────────────────────────────────────────────────────────────────────────────────
voxl-suite:   1.4.5
────────────────────────────────────────────────────────────────────────────────

 Service Name             |  Version  |  Enabled  |   Running   |  CPU Usage
───────────────────────────────────────────────────────────────────────────────
 docker-autorun           |    1.3.1  | Disabled  | Not Running |
 modallink-relink         |    1.1.6  | Disabled  | Not Running |
 voxl-camera-server       |    2.2.1  |  Enabled  | Not Running |
 voxl-cpu-monitor         |    0.6.0  |  Enabled  |   Running   |     0.0%
 voxl-dfs-server          |    0.2.0  | Disabled  | Not Running |
 voxl-elrs-startup        |    0.4.7  | Disabled  | Not Running |
 voxl-feature-tracker     |    0.5.2  | Disabled  | Not Running |
 voxl-flow-server         |    0.3.6  | Disabled  | Not Running |
 voxl-imu-server          |    1.1.3  |  Enabled  | Not Running |
 voxl-io-server           |    0.0.5  | Disabled  | Not Running |
 voxl-lepton-server       |    1.3.3  | Disabled  | Not Running |
 voxl-lepton-tracker      |    0.0.4  | Disabled  | Not Running |
 voxl-logger              |    0.5.3  | Disabled  | Not Running |
 voxl-mavcam-manager      |    0.5.8  | Disabled  | Not Running |
 voxl-mavlink-server      |    1.4.7  |  Enabled  |   Running   |     3.8%
 voxl-modem               |    1.1.6  | Disabled  | Not Running |
 voxl-open-vins-server    |   0.3.12  |  Enabled  | Not Running |
 voxl-osd                 |    0.1.7  | Disabled  | Not Running |
 voxl-portal              |   0.7.10  | Disabled  | Not Running |
 voxl-px4-imu-server      |    0.1.2  | Disabled  | Not Running |
 voxl-px4                 |   1.14.0  |  Enabled  |   Running   |    19.2%
 voxl-qvio-server         |    1.2.0  | Disabled  | Not Running |
 voxl-rangefinder-server  |    0.1.5  | Disabled  | Not Running |
 voxl-remote-id           |    0.0.9  | Disabled  | Not Running |
 voxl-seek-server         |   0.3.12  | Disabled  | Not Running |
 voxl-softap              |    0.3.5  | Disabled  | Not Running |
 voxl-state-estimator     |    0.0.5  | Disabled  | Not Running |
 voxl-static-ip           |    0.3.5  | Disabled  | Not Running |
 voxl-streamer            |    0.7.5  | Disabled  | Not Running |
 voxl-tag-detector        |    0.0.4  | Disabled  | Not Running |
 voxl-tflite-server       |    0.4.1  | Disabled  | Not Running |
 voxl-time-sync           |    1.4.7  | Disabled  | Not Running |
 voxl-uvc-server          |    0.1.7  | Disabled  | Not Running |
 voxl-vision-hub          |   1.8.21  |  Enabled  | Not Running |
 voxl-vrx                 |   1.3.14  | Disabled  | Not Running |
 voxl-vtx                 |   1.3.14  | Disabled  | Not Running |
 voxl-wait-for-fs         |    1.4.7  |  Enabled  |  Completed  |

Where voxl-camera-server and voxl-open-vins-server are our own forked versions with some minor changes.

We don't have GPS enabled on our drone:

px4-param show -a | grep GPS
    ADSB_GPS_OFF_LAT [-1,5] : 0
    ADSB_GPS_OFF_LON [-1,6] : 0
x   COM_ARM_WO_GPS [208,483] : 1
x   EKF2_GPS_CHECK [304,586] : 245
x + EKF2_GPS_CTRL [305,587] : 0
x   EKF2_GPS_DELAY [306,588] : 110.0000
x   EKF2_GPS_POS_X [307,589] : 0.0000
x   EKF2_GPS_POS_Y [308,590] : 0.0000
x   EKF2_GPS_POS_Z [309,591] : 0.0000
x   EKF2_GPS_P_GATE [310,592] : 5.0000
x   EKF2_GPS_P_NOISE [311,593] : 0.5000
x   EKF2_GPS_V_GATE [312,594] : 5.0000
x   EKF2_GPS_V_NOISE [313,595] : 0.3000
x   EKF2_REQ_GPS_H [357,641] : 10.0000
    FW_GPSF_LT [-1,721] : 30
    FW_GPSF_R [-1,722] : 15.0000
    GPS_1_GNSS [-1,841] : 0
    GPS_1_PROTOCOL [-1,842] : 1
    GPS_2_GNSS [-1,843] : 0
    GPS_2_PROTOCOL [-1,844] : 1
    GPS_DUMP_COMM [-1,845] : 0
    GPS_PITCH_OFFSET [-1,846] : 0.0000
    GPS_SAT_INFO [-1,847] : 0
    GPS_UBX_BAUD2 [-1,848] : 230400
    GPS_UBX_CFG_INTF [-1,849] : 0
x + GPS_UBX_DYNMODEL [418,850] : 6
    GPS_UBX_MODE [-1,851] : 0
    GPS_YAW_OFFSET [-1,852] : 0.0000
    LPE_GPS_DELAY [-1,937] : 0.2900
    LPE_GPS_VXY [-1,938] : 0.2500
    LPE_GPS_VZ [-1,939] : 0.2500
    LPE_GPS_XY [-1,940] : 1.0000
    LPE_GPS_Z [-1,941] : 3.0000
x   MAV_USEHILGPS [456,999] : 0
    SENS_EN_GPSSIM [-1,1448] : 0
    SENS_GPS_MASK [-1,1484] : 7
    SENS_GPS_PRIME [-1,1485] : 0
    SENS_GPS_TAU [-1,1486] : 10.0000
    SIM_GPS_USED [-1,1555] : 10
x + SYS_HAS_GPS [764,1658] : 0
    UAVCAN_SUB_GPS [-1,1930] : 1
    UCAN1_GPS0_SUB [-1,2048] : -1
    UCAN1_GPS1_SUB [-1,2049] : -1
    UCAN1_GPS_PUB [-1,2050] : -1
    UCAN1_UORB_GPS [-1,2053] : -1
    UCAN1_UORB_GPS_P [-1,2054] : -1

Have you seen this issues before and/or do you have any idea how this could be caused?

Eric Katzfey

@Tjark The drone shouldn't land or crash when the system time is updated. To disable GPS on the drone edit the /etc/modalai/voxl-px4.conf file to set GPS=NONE. Is the system time message coming from PX4 or from a GCS?

Tjark

@Eric-Katzfey Thanks for your prompt response! I think it is going down because it gets out of offboard mode because when the time is updated some of our processes also don't work properly anymore. But I don't know if it is only our processes which are affected.

The drone already had GPS disabled when this happened:

AIRFRAME=MULTICOPTER
GPS=NONE
RC=CRSF_MAV
ESC=VOXL_ESC
POWER_MANAGER=VOXLPM
AIRSPEED_SENSOR=NONE
DISTANCE_SENSOR=NONE
OSD=DISABLE
DAEMON_MODE=ENABLE
SENSOR_CAL=ACTUAL
ARTIFACT_MODE=DISABLE
EXTRA_STEPS=()

We are not connected to a GCS. Just PX4. We haven't seen this issue on sdk 1.1.3 (our previous version) but sometimes on sdk 1.4.5 (our current version) and also on multiple drones.

Eric Katzfey

@Tjark Probably don't want to apply the offset if clock is REALTIME here: https://github.com/modalai/px4-firmware/blob/96011c76dc9fffb2f2a2528df1767cdf926b24ce/platforms/posix/src/px4/common/drv_hrt.cpp#L464

Eric Katzfey

@Eric-Katzfey Only for MONOTONIC. I'll look at this more and try to figure out the correct thing to do.

Tjark

@Eric-Katzfey Ok, thanks. Looking forward to it. For now, we have put a forked version of voxl-mavlink-server on 1 drone which had this issue a few times. This forked version will log the requested time offset instead of applying it. If we get a log from that, we will post it here if that helps.

Eric Katzfey

@Tjark There is an updated voxl-px4 here: http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/voxl-px4_1.14.0-2.0.116-202510301051_arm64.deb. You will also need an updated libfc-sensor library here: http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/libfc-sensor_1.0.8-202508141142_arm64.deb. Can you try to update with these 2 and see if it solves the issue?

Tjark

@Eric-Katzfey Thanks for your update. We tried it on one of our drones but we had some strange issues regarding invalid setpoints afterwards and we think it may be caused by a voxl-px4 version mismatch. We were running version 1.14.0-2.0.98 of voxl-px4 and the version you provided is 1.14.0-2.0.116. Would it be difficult to provide a patch for version 1.14.0-2.0.98?

We also had our first log of that it occurred again with our forked voxl-mavlink-server. The drone continued its flight now which is good. This is the changed function:

void mavlink_services_handle_system_time(mavlink_message_t* msg)
{
	// collect and consolidate information
	int64_t time_now_ns = my_time_realtime_ns();
	int64_t time_now_s = time_now_ns / 1e9;
	int64_t gps_time_us = mavlink_msg_system_time_get_time_unix_usec(msg);
	int64_t gps_time_s = gps_time_us / 1e6;

	// check gps time is valid
	if(gps_time_s == 0) return;

	bool within_10_seconds_original = abs(time_now_s - gps_time_s)<10;
	bool within_10_seconds_correct = llabs(time_now_s - gps_time_s)<10;

	if (within_10_seconds_original != within_10_seconds_correct) {
		printf(
			"WARNING! The check if we are within 10 seconds gives an incorrect result due to wrong usage of abs.\n"
			"Original: %s, Correct: %s (time_now_s=%lld, gps_time_s=%lld, diff_original=%d, diff_correct=%lld)\n",
			within_10_seconds_original ? "true" : "false",
			within_10_seconds_correct  ? "true" : "false",
			(long long)time_now_s,
			(long long)gps_time_s,
			abs(time_now_s - gps_time_s),               // truncated 32-bit diff
			llabs(time_now_s - gps_time_s)              // correct 64-bit diff
		);
	}

	// if we are within 10 seconds, good enough
	if(within_10_seconds_correct){
		if(time_needs_setting){
			printf("detected system time has already been set\n");
		}
		time_needs_setting = 0;
		return;
	}
	else time_needs_setting = 1;

	if(!time_needs_setting) return;

	struct timespec ts_now;
	ts_now.tv_sec = time_now_s;
	ts_now.tv_nsec = 0;

    struct timespec ts_gps;
    ts_gps.tv_sec = gps_time_s;
    ts_gps.tv_nsec = 0;
	
	printf("WARNING! System wants to use GPS system time message to set UTC clock from %s to: %s",
		   asctime(gmtime(&ts_now.tv_sec)),
		   asctime(gmtime(&ts_gps.tv_sec)));
	printf("We will ignore this system time message and we won't update the clock");

    // if (clock_settime(CLOCK_REALTIME, &ts) < 0) {
    //     perror("Failed to set system time to GPS time");
    //     return;
    // }

	time_needs_setting = 0;

	return;
}

And we got log lines like this:

Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov  7 09:07:29 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]:  to: Fri Nov  7 09:07:29 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov  7 09:07:30 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]:  to: Fri Nov  7 09:07:30 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov  7 09:07:31 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]:  to: Fri Nov  7 09:07:31 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov  7 09:07:32 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]:  to: Fri Nov  7 09:07:32 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]: We will ignore this system time message and we won't update the clockWARNING! System wants to use GPS system time message to set UTC clock from Fri Nov  7 09:07:33 2025
Nov 07 09:07:28 JOHN voxl-mavlink-server[2237]:  to: Fri Nov  7 09:07:33 2025

but I don't yet understand why it is triggering this log because both timestamps seem to be identical. Maybe you have an idea?

ModalAI Forum

voxl-mavlink-server incorrectly sets UTC clock from GPS time