ROS IMU timing quality issues on RB5

[Moderator]

@dscott Can you please try updating to SDK 0.9.5? https://developer.modalai.com/asset/download/124

That 0.9 software version is pretty old

[dscott]

I reflashed using the linked image. voxl-mpa-to-ros appears to behave the same.

Here's all the same information as before. Like before, voxl-mpa-to-ros and rosbag record /imu_px4 were the only other things running that were not ran automatically.

--------------------------------------------------------------------------------
system-image: 1.4.1-M0052-14.1a-perf
kernel:       #1 SMP PREEMPT Thu Oct 13 18:59:44 UTC 2022 4.19.125
--------------------------------------------------------------------------------
hw version:   M0052
--------------------------------------------------------------------------------
voxl-suite:   0.9.5
--------------------------------------------------------------------------------
Packages:
Repo:  http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-0.9/binary-arm64/
Last Updated: 2023-08-25 17:16:14
List:
	libmodal-cv              0.2.3
	libmodal-exposure        0.0.7
	libmodal-journal         0.2.1
	libmodal-json            0.4.3
	libmodal-pipe            2.8.2
	libqrb5165-io            0.1.0
	libvoxl-cci-direct       0.1.5
	libvoxl-cutils           0.1.1
	mv-voxl                  0.1-r0
	qrb5165-bind             0.1-r0
	qrb5165-dfs-server       0.1.0
	qrb5165-imu-server       0.5.0
	qrb5165-slpi-test-sig    01-r0
	qrb5165-system-tweaks    0.1.5
	qrb5165-tflite           2.8.0-2
	voxl-bind                0.0.1
	voxl-camera-calibration  0.2.3
	voxl-camera-server       1.3.5
	voxl-cpu-monitor         0.3.0
	voxl-docker-support      1.2.4
	voxl-gphoto2-server      0.0.10
	voxl-jpeg-turbo          2.1.3-4
	voxl-libgphoto2          0.0.4
	voxl-libuvc              1.0.7
	voxl-logger              0.3.4
	voxl-mavlink             0.1.0
	voxl-mavlink-server      0.3.0
	voxl-modem               0.16.1
	voxl-mongoose            7.7.0-1
	voxl-mpa-to-ros          0.3.6
	voxl-mpa-tools           0.7.6
	voxl-opencv              4.5.5-1
	voxl-portal              0.5.0
	voxl-px4                 1.12.31
	voxl-px4-imu-server      0.1.2
	voxl-qvio-server         0.8.2
	voxl-remote-id           0.0.5
	voxl-streamer            0.4.1
	voxl-suite               0.9.5
	voxl-tag-detector        0.0.4
	voxl-tflite-server       0.3.1
	voxl-utils               1.2.2
	voxl-uvc-server          0.1.3
	voxl-vision-px4          1.4.0
--------------------------------------------------------------------------------

[Moderator]

@dscott A couple of ideas

1. use voxl-logger to log the IMU data and make sure there are no missing data points. We don't see dropped packets in our system, but we don't use ROS too frequently
2. increase the server's FIFO pipe size. Maybe change the code here

from

info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE;

to

info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE * 4;

[dscott]

@Moderator Sorry if it wasn't clear, but we have been using voxl-logger and haven't seen any of these issues there. I believe the "monotonic stamp" data on the second plot is taken from collecting data through both voxl-logger and voxl-mpa-to-ros at the same time.

From what I understand, is it not the same server provides the pipes that both voxl-logger and voxl-mpa-to-ros use?

[eric]

@dscott Indeed, the monotonic plot is displaying data captured from running voxl-logger in parallel with voxl-mpa-to-ros, after adjusting for initial offset between the realtime (ROS) clock and monotonic clock.

@Moderator Here is a plot showing dt from IMU data captured by voxl-logger. As you can see, there are a few outliers and a slight bias (making actual rate closer to 950Hz instead of 1KHz), but this is not causing any of the issues we reported in the original message when subscribing to the ROS IMU messages on the RB5.

[Moderator]

@eric yes, it's the same datasource. So, something in ROS is causing the delay. did you try increasing the FIFO size since ROS is slow to drain the FIFO?

[eric]

@Moderator Do you mean on the voxl-imu-server side? We've confirmed that data is not being dropped before getting to ROS.

In voxl-to-mpa-ros, IMU data is read from the pipe from imu-server, then published to ROS. We observed that IMU messages are read from the pipe in batches of about 10 or so. This explains why data was being dropped in ROS:

• A group of 10 or so IMU messages were read from the IMU pipe and placed inside a ROS publisher (with a queue size of 1).
• ROS subscribers can't copy data from the ROS queue faster than imu messages are copied into the queue.

However, we have already addressed this specific issue as described in the original post. We are not dropping any data after increasing the ROS publisher queue size in voxl-to-mpa-ros.

We are still having issues with latency on the ROS network, and "time travel" due to ROS usage of the realtime clock (instead of monotonic clock), and could use some help to resolve these issues, as described in the original post (refer to the second plot).

Apologies if I am misunderstanding your request

[eric]

@Moderator We've determined that the issue where time appears to be jumping backwards in ROS is actually caused by the system clock - see https://forum.modalai.com/topic/2612/rb5-system-clock-issues

This is a big issue and I hope that we and the rest of the community can get some support.

Regarding rosbag record latency, we've determined that a lot of the latency seems to be attributable to the recording utility we are using vs the ROS network as a whole. We did some experiments using ROS subscribers to directly measure latency (writing results to CSV), and performance seemed much better than indicated by rosbag timestamps.

[Moderator]

@eric Maybe the ROS timer can be changed to using the monotonic clock?

Regarding dropped IMU samples, these folks reduced the publishing rate and it seemed to help https://forum.modalai.com/topic/2570/imu_apps-seems-to-be-publishing-in-bursts/4

[eric]

@Moderator Unfortunately, there is not a way to configure ROS1 to use steady_clock instead of system_clock. In ROS2, it seems that this is configurable, but ModalAI does not support ROS2.