ROS IMU timing quality issues on RB5
-
can you please provide the software version? Do you have voxl-inspect-cpu, and if so, can you please provide the output? It sounds like you are starving the voxl-mpa-to-ros of CPU resources to empty the queue
Hi, I'm on the same team as @eric.
Here's the output of
voxl-version:-------------------------------------------------------------------------------- system-image: 1.4.1-M0052-14.1a-perf kernel: #1 SMP PREEMPT Thu Oct 13 18:59:44 UTC 2022 4.19.125 -------------------------------------------------------------------------------- hw version: M0052 -------------------------------------------------------------------------------- voxl-suite: -------------------------------------------------------------------------------- [1mPackages:[0m Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-0.9/binary-arm64/ Last Updated: 2023-08-18 19:06:41 List: libmodal-exposure 0.0.7 libmodal-journal 0.2.1 libmodal-json 0.4.3 libmodal-pipe 2.8.2 libqrb5165-io 0.1.0 libvoxl-cci-direct 0.1.5 libvoxl-cutils 0.1.1 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-imu-server 0.5.0 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.1.5 qrb5165-tflite 2.8.0-2 voxl-bind 0.0.1 voxl-camera-server 1.3.5 voxl-cpu-monitor 0.3.0 voxl-docker-support 1.2.4 voxl-gphoto2-server 0.0.10 voxl-jpeg-turbo 2.1.3-4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.3.4 voxl-mavlink 0.1.0 voxl-mavlink-server 0.3.0 voxl-modem 0.16.1 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.6-202308232023 voxl-mpa-tools 0.7.6 voxl-portal 0.5.0 voxl-px4 1.12.31 voxl-px4-imu-server 0.1.2 voxl-qvio-server 0.8.2 voxl-remote-id 0.0.5 voxl-streamer 0.4.1 voxl-utils 1.2.2 voxl-uvc-server 0.1.3 voxl-vision-px4 1.4.1 --------------------------------------------------------------------------------Here are screenshots from voxl-inspect-cpu and top. They are made while running voxl-mpa-to-ros and
rosbag record /imu_px4.
Here's the output of voxl-inspect-services:
-
Hi, I'm on the same team as @eric.
Here's the output of
voxl-version:-------------------------------------------------------------------------------- system-image: 1.4.1-M0052-14.1a-perf kernel: #1 SMP PREEMPT Thu Oct 13 18:59:44 UTC 2022 4.19.125 -------------------------------------------------------------------------------- hw version: M0052 -------------------------------------------------------------------------------- voxl-suite: -------------------------------------------------------------------------------- [1mPackages:[0m Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-0.9/binary-arm64/ Last Updated: 2023-08-18 19:06:41 List: libmodal-exposure 0.0.7 libmodal-journal 0.2.1 libmodal-json 0.4.3 libmodal-pipe 2.8.2 libqrb5165-io 0.1.0 libvoxl-cci-direct 0.1.5 libvoxl-cutils 0.1.1 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-imu-server 0.5.0 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.1.5 qrb5165-tflite 2.8.0-2 voxl-bind 0.0.1 voxl-camera-server 1.3.5 voxl-cpu-monitor 0.3.0 voxl-docker-support 1.2.4 voxl-gphoto2-server 0.0.10 voxl-jpeg-turbo 2.1.3-4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.3.4 voxl-mavlink 0.1.0 voxl-mavlink-server 0.3.0 voxl-modem 0.16.1 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.6-202308232023 voxl-mpa-tools 0.7.6 voxl-portal 0.5.0 voxl-px4 1.12.31 voxl-px4-imu-server 0.1.2 voxl-qvio-server 0.8.2 voxl-remote-id 0.0.5 voxl-streamer 0.4.1 voxl-utils 1.2.2 voxl-uvc-server 0.1.3 voxl-vision-px4 1.4.1 --------------------------------------------------------------------------------Here are screenshots from voxl-inspect-cpu and top. They are made while running voxl-mpa-to-ros and
rosbag record /imu_px4.
Here's the output of voxl-inspect-services:
@dscott Can you please try updating to SDK 0.9.5? https://developer.modalai.com/asset/download/124
That 0.9 software version is pretty old
-
@dscott Can you please try updating to SDK 0.9.5? https://developer.modalai.com/asset/download/124
That 0.9 software version is pretty old
I reflashed using the linked image. voxl-mpa-to-ros appears to behave the same.
Here's all the same information as before. Like before, voxl-mpa-to-ros and
rosbag record /imu_px4were the only other things running that were not ran automatically.-------------------------------------------------------------------------------- system-image: 1.4.1-M0052-14.1a-perf kernel: #1 SMP PREEMPT Thu Oct 13 18:59:44 UTC 2022 4.19.125 -------------------------------------------------------------------------------- hw version: M0052 -------------------------------------------------------------------------------- voxl-suite: 0.9.5 -------------------------------------------------------------------------------- [1mPackages:[0m Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-0.9/binary-arm64/ Last Updated: 2023-08-25 17:16:14 List: libmodal-cv 0.2.3 libmodal-exposure 0.0.7 libmodal-journal 0.2.1 libmodal-json 0.4.3 libmodal-pipe 2.8.2 libqrb5165-io 0.1.0 libvoxl-cci-direct 0.1.5 libvoxl-cutils 0.1.1 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-dfs-server 0.1.0 qrb5165-imu-server 0.5.0 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.1.5 qrb5165-tflite 2.8.0-2 voxl-bind 0.0.1 voxl-camera-calibration 0.2.3 voxl-camera-server 1.3.5 voxl-cpu-monitor 0.3.0 voxl-docker-support 1.2.4 voxl-gphoto2-server 0.0.10 voxl-jpeg-turbo 2.1.3-4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.3.4 voxl-mavlink 0.1.0 voxl-mavlink-server 0.3.0 voxl-modem 0.16.1 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.6 voxl-mpa-tools 0.7.6 voxl-opencv 4.5.5-1 voxl-portal 0.5.0 voxl-px4 1.12.31 voxl-px4-imu-server 0.1.2 voxl-qvio-server 0.8.2 voxl-remote-id 0.0.5 voxl-streamer 0.4.1 voxl-suite 0.9.5 voxl-tag-detector 0.0.4 voxl-tflite-server 0.3.1 voxl-utils 1.2.2 voxl-uvc-server 0.1.3 voxl-vision-px4 1.4.0 --------------------------------------------------------------------------------
-
I reflashed using the linked image. voxl-mpa-to-ros appears to behave the same.
Here's all the same information as before. Like before, voxl-mpa-to-ros and
rosbag record /imu_px4were the only other things running that were not ran automatically.-------------------------------------------------------------------------------- system-image: 1.4.1-M0052-14.1a-perf kernel: #1 SMP PREEMPT Thu Oct 13 18:59:44 UTC 2022 4.19.125 -------------------------------------------------------------------------------- hw version: M0052 -------------------------------------------------------------------------------- voxl-suite: 0.9.5 -------------------------------------------------------------------------------- [1mPackages:[0m Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-0.9/binary-arm64/ Last Updated: 2023-08-25 17:16:14 List: libmodal-cv 0.2.3 libmodal-exposure 0.0.7 libmodal-journal 0.2.1 libmodal-json 0.4.3 libmodal-pipe 2.8.2 libqrb5165-io 0.1.0 libvoxl-cci-direct 0.1.5 libvoxl-cutils 0.1.1 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-dfs-server 0.1.0 qrb5165-imu-server 0.5.0 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.1.5 qrb5165-tflite 2.8.0-2 voxl-bind 0.0.1 voxl-camera-calibration 0.2.3 voxl-camera-server 1.3.5 voxl-cpu-monitor 0.3.0 voxl-docker-support 1.2.4 voxl-gphoto2-server 0.0.10 voxl-jpeg-turbo 2.1.3-4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.3.4 voxl-mavlink 0.1.0 voxl-mavlink-server 0.3.0 voxl-modem 0.16.1 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.6 voxl-mpa-tools 0.7.6 voxl-opencv 4.5.5-1 voxl-portal 0.5.0 voxl-px4 1.12.31 voxl-px4-imu-server 0.1.2 voxl-qvio-server 0.8.2 voxl-remote-id 0.0.5 voxl-streamer 0.4.1 voxl-suite 0.9.5 voxl-tag-detector 0.0.4 voxl-tflite-server 0.3.1 voxl-utils 1.2.2 voxl-uvc-server 0.1.3 voxl-vision-px4 1.4.0 --------------------------------------------------------------------------------
@dscott A couple of ideas
- use voxl-logger to log the IMU data and make sure there are no missing data points. We don't see dropped packets in our system, but we don't use ROS too frequently
- increase the server's FIFO pipe size. Maybe change the code here
from
info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE;to
info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE * 4; -
@dscott A couple of ideas
- use voxl-logger to log the IMU data and make sure there are no missing data points. We don't see dropped packets in our system, but we don't use ROS too frequently
- increase the server's FIFO pipe size. Maybe change the code here
from
info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE;to
info.size_bytes = IMU_RECOMMENDED_PIPE_SIZE * 4;@Moderator Sorry if it wasn't clear, but we have been using voxl-logger and haven't seen any of these issues there. I believe the "monotonic stamp" data on the second plot is taken from collecting data through both voxl-logger and voxl-mpa-to-ros at the same time.
From what I understand, is it not the same server provides the pipes that both voxl-logger and voxl-mpa-to-ros use?
-
@Moderator Sorry if it wasn't clear, but we have been using voxl-logger and haven't seen any of these issues there. I believe the "monotonic stamp" data on the second plot is taken from collecting data through both voxl-logger and voxl-mpa-to-ros at the same time.
From what I understand, is it not the same server provides the pipes that both voxl-logger and voxl-mpa-to-ros use?
@dscott Indeed, the monotonic plot is displaying data captured from running voxl-logger in parallel with voxl-mpa-to-ros, after adjusting for initial offset between the realtime (ROS) clock and monotonic clock.
@Moderator Here is a plot showing dt from IMU data captured by voxl-logger. As you can see, there are a few outliers and a slight bias (making actual rate closer to 950Hz instead of 1KHz), but this is not causing any of the issues we reported in the original message when subscribing to the ROS IMU messages on the RB5.
-
@dscott Indeed, the monotonic plot is displaying data captured from running voxl-logger in parallel with voxl-mpa-to-ros, after adjusting for initial offset between the realtime (ROS) clock and monotonic clock.
@Moderator Here is a plot showing dt from IMU data captured by voxl-logger. As you can see, there are a few outliers and a slight bias (making actual rate closer to 950Hz instead of 1KHz), but this is not causing any of the issues we reported in the original message when subscribing to the ROS IMU messages on the RB5.
-
@eric yes, it's the same datasource. So, something in ROS is causing the delay. did you try increasing the FIFO size since ROS is slow to drain the FIFO?
@Moderator Do you mean on the voxl-imu-server side? We've confirmed that data is not being dropped before getting to ROS.
In voxl-to-mpa-ros, IMU data is read from the pipe from imu-server, then published to ROS. We observed that IMU messages are read from the pipe in batches of about 10 or so. This explains why data was being dropped in ROS:
- A group of 10 or so IMU messages were read from the IMU pipe and placed inside a ROS publisher (with a queue size of 1).
- ROS subscribers can't copy data from the ROS queue faster than imu messages are copied into the queue.
However, we have already addressed this specific issue as described in the original post. We are not dropping any data after increasing the ROS publisher queue size in voxl-to-mpa-ros.
We are still having issues with latency on the ROS network, and "time travel" due to ROS usage of the realtime clock (instead of monotonic clock), and could use some help to resolve these issues, as described in the original post (refer to the second plot).
Apologies if I am misunderstanding your request
-
@eric yes, it's the same datasource. So, something in ROS is causing the delay. did you try increasing the FIFO size since ROS is slow to drain the FIFO?
@Moderator We've determined that the issue where time appears to be jumping backwards in ROS is actually caused by the system clock - see https://forum.modalai.com/topic/2612/rb5-system-clock-issues
This is a big issue and I hope that we and the rest of the community can get some support.
Regarding rosbag record latency, we've determined that a lot of the latency seems to be attributable to the recording utility we are using vs the ROS network as a whole. We did some experiments using ROS subscribers to directly measure latency (writing results to CSV), and performance seemed much better than indicated by rosbag timestamps.
-
@Moderator We've determined that the issue where time appears to be jumping backwards in ROS is actually caused by the system clock - see https://forum.modalai.com/topic/2612/rb5-system-clock-issues
This is a big issue and I hope that we and the rest of the community can get some support.
Regarding rosbag record latency, we've determined that a lot of the latency seems to be attributable to the recording utility we are using vs the ROS network as a whole. We did some experiments using ROS subscribers to directly measure latency (writing results to CSV), and performance seemed much better than indicated by rosbag timestamps.
@eric Maybe the ROS timer can be changed to using the monotonic clock?
Regarding dropped IMU samples, these folks reduced the publishing rate and it seemed to help https://forum.modalai.com/topic/2570/imu_apps-seems-to-be-publishing-in-bursts/4
-
@eric Maybe the ROS timer can be changed to using the monotonic clock?
Regarding dropped IMU samples, these folks reduced the publishing rate and it seemed to help https://forum.modalai.com/topic/2570/imu_apps-seems-to-be-publishing-in-bursts/4
@Moderator Unfortunately, there is not a way to configure ROS1 to use steady_clock instead of system_clock. In ROS2, it seems that this is configurable, but ModalAI does not support ROS2.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login