Can you please try to use voxl-logger to log the IMU data concurrently with mpa-to-ros pipeline?
voxl-logger will save imu data with human readable timestamps, so you could easily plot that and double check if the IMU samples produced by qrb5165-imu-server are indeed jumping around. It is possible that something related to ROS is introducing a timestamp issue.
Alex