I've solved the problem by setting the voxl-imu-server.conf
to
"imu0_enable": true,
"imu0_sample_rate_hz": 200,
"imu0_lp_cutoff_freq_hz": 92,
"imu0_rotate_common_frame": true,
"imu0_fifo_poll_rate_hz": 200,
...
Tested on 2 different VOXL2 boards, this reduces the number of lost messages sequences from 60% to <0.5% of all messages. Information on why this is the case would be interesting, as it is stated here that the default config is optimized for VIO.