Slow connection to QGC and missing params due to timeout with Doodle Labs Radios
-
I am currently running SDK 1.0.0 on a VOXL 2 and am using Doodle Labs Radios for the connection. This worked great for me in SDK .0.9.5, however in 1.0.0 the connection to QGC is very slow. The green bar at the top of the screens moves greater than 10x times slower than it does in the previous SDK. No settings on the radios have been changed between the SDK. I was able to test out SDK 1.0.0 using WIFI instead of the Doodle Labs Radios and everything works as it should.
Sometimes I am able to get a full set of params and a connection to QGC but am not able to run any of the sensor calibrations due to lag. The calibrations just timeout. Even when doing a radio calibration the slider on QGC move about 2 sec after I move the sticks.
I am hoping that someone from the dev team will be able to help me identify the issue.
@Chase-Riley There shouldn't be any difference between SDK 0.9.5 and SDK 1.0.0 in terms of how the doodle labs modems are handled.
Make sure that your ground station radio and drone side radio aren't too close too each other in order to prevent RF saturation.
Take a read through the "Troubleshooting Link Quality" section of this document from Doodle Labs, starting on page 6: https://doodlelabs.com/wp-content/uploads/2022/09/Smart-Radio-Trouble-Shooting-Guide-V0122.pdf
"
“The throughput is poor even at close range”
As the radios are optimized for long range, they do not work well when they are within a close
range of each other. At close range, the radios are likely to saturate the RF front-end, resulting
in poor performance. For bench evaluation, please use RF attenuators supplied in the Eval kit.
You should also reduce the output power of the radios and not use high gain antennas. , It is
further possible to enable Transmit-Power Control (TPC). Note that TPC is only recommended
for point-to-point networks.
If the radios are more than 5-10 meters apart and the throughput is still poor, then it could be
an interference or power supply issue. Please see the advice in the next section." -
@Chase-Riley There shouldn't be any difference between SDK 0.9.5 and SDK 1.0.0 in terms of how the doodle labs modems are handled.
Make sure that your ground station radio and drone side radio aren't too close too each other in order to prevent RF saturation.
Take a read through the "Troubleshooting Link Quality" section of this document from Doodle Labs, starting on page 6: https://doodlelabs.com/wp-content/uploads/2022/09/Smart-Radio-Trouble-Shooting-Guide-V0122.pdf
"
“The throughput is poor even at close range”
As the radios are optimized for long range, they do not work well when they are within a close
range of each other. At close range, the radios are likely to saturate the RF front-end, resulting
in poor performance. For bench evaluation, please use RF attenuators supplied in the Eval kit.
You should also reduce the output power of the radios and not use high gain antennas. , It is
further possible to enable Transmit-Power Control (TPC). Note that TPC is only recommended
for point-to-point networks.
If the radios are more than 5-10 meters apart and the throughput is still poor, then it could be
an interference or power supply issue. Please see the advice in the next section." -
@tom You can also try pinging from your ground station to the drone and see how cleanly the packets are coming through
@tom I have read through the troubleshooting manual from Doodle Labs multiple times. I do have the attenuators installed. I have tried bench top testing as well as removing the attenuators and trying range testing ( greater than 10m between GCS and AV). I have not switched on TPC because I am requiring a mesh setup for this project. Power supply should be solid. I have tried using a Lipo battery and the supplied 12V AC to DC convertor that came with the VOXL2 Dev kit. On the GCS side I am using a AC to DC convertor as well.
I understand that there should not be a difference in how Doodle Labs is handled between .0.9.5 and 1.0.0, however it is odd that if a flash .0.9.5 on the VOXL2 and run the same exact test there is not issue. I have run this test multiple times and have never had a issue on .0.9.5, but as soon as I switch back to 1.0.0 I encounter the problem again. Both test have been run with the same exact radio configurations, in the same RF environment, and the same antennas. The only variable I am seeing is the SDK version. I have included two ping test below. First one is from GCS to AV and the second is from AV to GCS running 1.0.0. These were captured while QGC was slowly getting link only to time out with missing params after sometime.
GCS to AV
ping 10.223.0.100 PING 10.223.0.100 (10.223.0.100) 56(84) bytes of data. 64 bytes from 10.223.0.100: icmp_seq=1 ttl=64 time=2.48 ms 64 bytes from 10.223.0.100: icmp_seq=2 ttl=64 time=2.90 ms 64 bytes from 10.223.0.100: icmp_seq=3 ttl=64 time=13.5 ms 64 bytes from 10.223.0.100: icmp_seq=4 ttl=64 time=2.92 ms 64 bytes from 10.223.0.100: icmp_seq=5 ttl=64 time=2.72 ms 64 bytes from 10.223.0.100: icmp_seq=6 ttl=64 time=2.29 ms 64 bytes from 10.223.0.100: icmp_seq=7 ttl=64 time=2.47 ms 64 bytes from 10.223.0.100: icmp_seq=8 ttl=64 time=32.8 ms 64 bytes from 10.223.0.100: icmp_seq=9 ttl=64 time=13.0 ms 64 bytes from 10.223.0.100: icmp_seq=10 ttl=64 time=3.13 ms 64 bytes from 10.223.0.100: icmp_seq=11 ttl=64 time=109 ms 64 bytes from 10.223.0.100: icmp_seq=12 ttl=64 time=4.34 ms 64 bytes from 10.223.0.100: icmp_seq=13 ttl=64 time=4.45 ms 64 bytes from 10.223.0.100: icmp_seq=14 ttl=64 time=7.97 ms 64 bytes from 10.223.0.100: icmp_seq=15 ttl=64 time=2.86 ms 64 bytes from 10.223.0.100: icmp_seq=16 ttl=64 time=2.39 ms 64 bytes from 10.223.0.100: icmp_seq=17 ttl=64 time=4.17 ms 64 bytes from 10.223.0.100: icmp_seq=18 ttl=64 time=4.50 ms 64 bytes from 10.223.0.100: icmp_seq=19 ttl=64 time=3.76 ms 64 bytes from 10.223.0.100: icmp_seq=20 ttl=64 time=2.62 ms 64 bytes from 10.223.0.100: icmp_seq=21 ttl=64 time=2.37 ms 64 bytes from 10.223.0.100: icmp_seq=22 ttl=64 time=8.58 ms 64 bytes from 10.223.0.100: icmp_seq=23 ttl=64 time=5.67 ms 64 bytes from 10.223.0.100: icmp_seq=24 ttl=64 time=2.46 ms 64 bytes from 10.223.0.100: icmp_seq=25 ttl=64 time=4.83 ms 64 bytes from 10.223.0.100: icmp_seq=26 ttl=64 time=2.44 ms 64 bytes from 10.223.0.100: icmp_seq=27 ttl=64 time=5.14 ms 64 bytes from 10.223.0.100: icmp_seq=28 ttl=64 time=2.66 ms 64 bytes from 10.223.0.100: icmp_seq=29 ttl=64 time=2.73 ms 64 bytes from 10.223.0.100: icmp_seq=30 ttl=64 time=3.12 ms ^C --- 10.223.0.100 ping statistics --- 30 packets transmitted, 30 received, 0% packet loss, time 29044ms rtt min/avg/max/mdev = 2.292/8.794/108.558/19.422 msAV to GCS
ping 10.223.0.150 PING 10.223.0.150 (10.223.0.150): 56 data bytes 64 bytes from 10.223.0.150: icmp_seq=0 ttl=64 time=3.496 ms 64 bytes from 10.223.0.150: icmp_seq=1 ttl=64 time=10.936 ms 64 bytes from 10.223.0.150: icmp_seq=2 ttl=64 time=3.443 ms 64 bytes from 10.223.0.150: icmp_seq=3 ttl=64 time=2.728 ms 64 bytes from 10.223.0.150: icmp_seq=4 ttl=64 time=3.497 ms 64 bytes from 10.223.0.150: icmp_seq=5 ttl=64 time=3.400 ms 64 bytes from 10.223.0.150: icmp_seq=6 ttl=64 time=3.168 ms 64 bytes from 10.223.0.150: icmp_seq=7 ttl=64 time=3.219 ms 64 bytes from 10.223.0.150: icmp_seq=8 ttl=64 time=2.796 ms 64 bytes from 10.223.0.150: icmp_seq=9 ttl=64 time=2.916 ms 64 bytes from 10.223.0.150: icmp_seq=10 ttl=64 time=64.166 ms 64 bytes from 10.223.0.150: icmp_seq=11 ttl=64 time=2.647 ms 64 bytes from 10.223.0.150: icmp_seq=12 ttl=64 time=4.295 ms 64 bytes from 10.223.0.150: icmp_seq=13 ttl=64 time=12.701 ms 64 bytes from 10.223.0.150: icmp_seq=14 ttl=64 time=6.396 ms 64 bytes from 10.223.0.150: icmp_seq=15 ttl=64 time=2.774 ms 64 bytes from 10.223.0.150: icmp_seq=16 ttl=64 time=5.131 ms 64 bytes from 10.223.0.150: icmp_seq=17 ttl=64 time=2.943 ms 64 bytes from 10.223.0.150: icmp_seq=18 ttl=64 time=5.845 ms 64 bytes from 10.223.0.150: icmp_seq=19 ttl=64 time=2.695 ms 64 bytes from 10.223.0.150: icmp_seq=20 ttl=64 time=8.678 ms 64 bytes from 10.223.0.150: icmp_seq=21 ttl=64 time=5.018 ms 64 bytes from 10.223.0.150: icmp_seq=22 ttl=64 time=2.625 ms 64 bytes from 10.223.0.150: icmp_seq=23 ttl=64 time=11.066 ms 64 bytes from 10.223.0.150: icmp_seq=24 ttl=64 time=8.588 ms 64 bytes from 10.223.0.150: icmp_seq=25 ttl=64 time=2.556 ms 64 bytes from 10.223.0.150: icmp_seq=26 ttl=64 time=3.735 ms 64 bytes from 10.223.0.150: icmp_seq=27 ttl=64 time=2.621 ms 64 bytes from 10.223.0.150: icmp_seq=28 ttl=64 time=9.022 ms 64 bytes from 10.223.0.150: icmp_seq=29 ttl=64 time=2.882 ms 64 bytes from 10.223.0.150: icmp_seq=30 ttl=64 time=7.700 ms ^C--- 10.223.0.150 ping statistics --- 31 packets transmitted, 31 packets received, 0% packet loss round-trip min/avg/max/stddev = 2.556/6.893/64.166/10.847 ms -
@Moderator Thanks. Let me know how it goes. I am pretty much at a stand still until I can get a reliable link from the Doodle Radios on 1.0.0.
-
between SDK 0.9.5 and SDK 1.0 we disabled the mavlink packet aggregation in voxl-mavlink-server which used to be on by default. This was originally a patch to reduce wifi traffic on VOXL1 + Flight Core systems that forwarded full offboard-rate traffic through WiFi since the implementation of PX4 on VOXL2's DSP only sends slow data rate mavlink telemetry to the ground control station, we didn't find this feature necessary anymore and disabled it by default.
You can re-enable the mavlink aggregation feature with the udp_mtu field in /etc/modalai/voxl-mavlink-server.conf, the JSON file contains the following description of the field:
* udp_mtu - maximum transfer unit for UDP packets back to GCS. voxl-mavlink-server * will bundle up packets for the GCS into a single UDP packet with * a maximum size of this. This might help with some radios. * This feature is off by default to let the network stack handle aggregation. * Set to 0 to disable this feature and send one UDP packet per msg. * Set to something like 500 to bundle a handful of packets together.I suspect doodlelabs might not be well optimized for very small mavlink messages or is not capable of doing its own aggregation. Try a few different values such as 100, 200, 300 etc and see if you get any improvement.
Best,
James -
between SDK 0.9.5 and SDK 1.0 we disabled the mavlink packet aggregation in voxl-mavlink-server which used to be on by default. This was originally a patch to reduce wifi traffic on VOXL1 + Flight Core systems that forwarded full offboard-rate traffic through WiFi since the implementation of PX4 on VOXL2's DSP only sends slow data rate mavlink telemetry to the ground control station, we didn't find this feature necessary anymore and disabled it by default.
You can re-enable the mavlink aggregation feature with the udp_mtu field in /etc/modalai/voxl-mavlink-server.conf, the JSON file contains the following description of the field:
* udp_mtu - maximum transfer unit for UDP packets back to GCS. voxl-mavlink-server * will bundle up packets for the GCS into a single UDP packet with * a maximum size of this. This might help with some radios. * This feature is off by default to let the network stack handle aggregation. * Set to 0 to disable this feature and send one UDP packet per msg. * Set to something like 500 to bundle a handful of packets together.I suspect doodlelabs might not be well optimized for very small mavlink messages or is not capable of doing its own aggregation. Try a few different values such as 100, 200, 300 etc and see if you get any improvement.
Best,
James@James-Strawson @tom @Moderator @Chad-Sweet So it does seem there was a change from SDK .0.9.5 to 1.0.0. This fixed the problem. I changed the value to
500and it links up quickly and I am able to complete calibrations. I would recommend that this be added to the Doodle Labs Modems Documentation. -
@James-Strawson @tom @Moderator @Chad-Sweet So it does seem there was a change from SDK .0.9.5 to 1.0.0. This fixed the problem. I changed the value to
500and it links up quickly and I am able to complete calibrations. I would recommend that this be added to the Doodle Labs Modems Documentation.@Chase-Riley Glad to hear. Does it also perform as expected at lower values such as 100 or 200? a value as high as 500 will increase latency when only sending small mavlink messages.
-
@Chase-Riley Glad to hear. Does it also perform as expected at lower values such as 100 or 200? a value as high as 500 will increase latency when only sending small mavlink messages.
@James-Strawson I had no luck with 100 or 200. I can try some other values, however I have not seen an issue with the value of 500 yet.
-
@Chase-Riley Glad to hear. Does it also perform as expected at lower values such as 100 or 200? a value as high as 500 will increase latency when only sending small mavlink messages.
@James-Strawson Quick update. I was able to lower the value to
300and it seems to work as well. -
C Chase Riley referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login