• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Register
  • Login
ModalAI Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
    • Register
    • Login

    Slow connection to QGC and missing params due to timeout with Doodle Labs Radios

    VOXL 2
    4
    11
    1.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      Chase Riley
      last edited by 6 Oct 2023, 20:21

      I am currently running SDK 1.0.0 on a VOXL 2 and am using Doodle Labs Radios for the connection. This worked great for me in SDK .0.9.5, however in 1.0.0 the connection to QGC is very slow. The green bar at the top of the screens moves greater than 10x times slower than it does in the previous SDK. No settings on the radios have been changed between the SDK. I was able to test out SDK 1.0.0 using WIFI instead of the Doodle Labs Radios and everything works as it should.

      Sometimes I am able to get a full set of params and a connection to QGC but am not able to run any of the sensor calibrations due to lag. The calibrations just timeout. Even when doing a radio calibration the slider on QGC move about 2 sec after I move the sticks.

      I am hoping that someone from the dev team will be able to help me identify the issue.

      T 1 Reply Last reply 6 Oct 2023, 21:44 Reply Quote 0
      • T
        tom admin @Chase Riley
        last edited by 6 Oct 2023, 21:44

        @Chase-Riley There shouldn't be any difference between SDK 0.9.5 and SDK 1.0.0 in terms of how the doodle labs modems are handled.

        Make sure that your ground station radio and drone side radio aren't too close too each other in order to prevent RF saturation.

        Take a read through the "Troubleshooting Link Quality" section of this document from Doodle Labs, starting on page 6: https://doodlelabs.com/wp-content/uploads/2022/09/Smart-Radio-Trouble-Shooting-Guide-V0122.pdf

        "
        “The throughput is poor even at close range”
        As the radios are optimized for long range, they do not work well when they are within a close
        range of each other. At close range, the radios are likely to saturate the RF front-end, resulting
        in poor performance. For bench evaluation, please use RF attenuators supplied in the Eval kit.
        You should also reduce the output power of the radios and not use high gain antennas. , It is
        further possible to enable Transmit-Power Control (TPC). Note that TPC is only recommended
        for point-to-point networks.
        If the radios are more than 5-10 meters apart and the throughput is still poor, then it could be
        an interference or power supply issue. Please see the advice in the next section."

        T 1 Reply Last reply 6 Oct 2023, 21:46 Reply Quote 0
        • T
          tom admin @tom
          last edited by 6 Oct 2023, 21:46

          @tom You can also try pinging from your ground station to the drone and see how cleanly the packets are coming through

          C 1 Reply Last reply 10 Oct 2023, 14:56 Reply Quote 0
          • C
            Chase Riley @tom
            last edited by Chase Riley 10 Oct 2023, 15:08 10 Oct 2023, 14:56

            @tom I have read through the troubleshooting manual from Doodle Labs multiple times. I do have the attenuators installed. I have tried bench top testing as well as removing the attenuators and trying range testing ( greater than 10m between GCS and AV). I have not switched on TPC because I am requiring a mesh setup for this project. Power supply should be solid. I have tried using a Lipo battery and the supplied 12V AC to DC convertor that came with the VOXL2 Dev kit. On the GCS side I am using a AC to DC convertor as well.

            I understand that there should not be a difference in how Doodle Labs is handled between .0.9.5 and 1.0.0, however it is odd that if a flash .0.9.5 on the VOXL2 and run the same exact test there is not issue. I have run this test multiple times and have never had a issue on .0.9.5, but as soon as I switch back to 1.0.0 I encounter the problem again. Both test have been run with the same exact radio configurations, in the same RF environment, and the same antennas. The only variable I am seeing is the SDK version. I have included two ping test below. First one is from GCS to AV and the second is from AV to GCS running 1.0.0. These were captured while QGC was slowly getting link only to time out with missing params after sometime.

            GCS to AV

            ping 10.223.0.100
            PING 10.223.0.100 (10.223.0.100) 56(84) bytes of data.
            64 bytes from 10.223.0.100: icmp_seq=1 ttl=64 time=2.48 ms
            64 bytes from 10.223.0.100: icmp_seq=2 ttl=64 time=2.90 ms
            64 bytes from 10.223.0.100: icmp_seq=3 ttl=64 time=13.5 ms
            64 bytes from 10.223.0.100: icmp_seq=4 ttl=64 time=2.92 ms
            64 bytes from 10.223.0.100: icmp_seq=5 ttl=64 time=2.72 ms
            64 bytes from 10.223.0.100: icmp_seq=6 ttl=64 time=2.29 ms
            64 bytes from 10.223.0.100: icmp_seq=7 ttl=64 time=2.47 ms
            64 bytes from 10.223.0.100: icmp_seq=8 ttl=64 time=32.8 ms
            64 bytes from 10.223.0.100: icmp_seq=9 ttl=64 time=13.0 ms
            64 bytes from 10.223.0.100: icmp_seq=10 ttl=64 time=3.13 ms
            64 bytes from 10.223.0.100: icmp_seq=11 ttl=64 time=109 ms
            64 bytes from 10.223.0.100: icmp_seq=12 ttl=64 time=4.34 ms
            64 bytes from 10.223.0.100: icmp_seq=13 ttl=64 time=4.45 ms
            64 bytes from 10.223.0.100: icmp_seq=14 ttl=64 time=7.97 ms
            64 bytes from 10.223.0.100: icmp_seq=15 ttl=64 time=2.86 ms
            64 bytes from 10.223.0.100: icmp_seq=16 ttl=64 time=2.39 ms
            64 bytes from 10.223.0.100: icmp_seq=17 ttl=64 time=4.17 ms
            64 bytes from 10.223.0.100: icmp_seq=18 ttl=64 time=4.50 ms
            64 bytes from 10.223.0.100: icmp_seq=19 ttl=64 time=3.76 ms
            64 bytes from 10.223.0.100: icmp_seq=20 ttl=64 time=2.62 ms
            64 bytes from 10.223.0.100: icmp_seq=21 ttl=64 time=2.37 ms
            64 bytes from 10.223.0.100: icmp_seq=22 ttl=64 time=8.58 ms
            64 bytes from 10.223.0.100: icmp_seq=23 ttl=64 time=5.67 ms
            64 bytes from 10.223.0.100: icmp_seq=24 ttl=64 time=2.46 ms
            64 bytes from 10.223.0.100: icmp_seq=25 ttl=64 time=4.83 ms
            64 bytes from 10.223.0.100: icmp_seq=26 ttl=64 time=2.44 ms
            64 bytes from 10.223.0.100: icmp_seq=27 ttl=64 time=5.14 ms
            64 bytes from 10.223.0.100: icmp_seq=28 ttl=64 time=2.66 ms
            64 bytes from 10.223.0.100: icmp_seq=29 ttl=64 time=2.73 ms
            64 bytes from 10.223.0.100: icmp_seq=30 ttl=64 time=3.12 ms
            ^C
            --- 10.223.0.100 ping statistics ---
            30 packets transmitted, 30 received, 0% packet loss, time 29044ms
            rtt min/avg/max/mdev = 2.292/8.794/108.558/19.422 ms
            

            AV to GCS

            ping 10.223.0.150
            PING 10.223.0.150 (10.223.0.150): 56 data bytes
            64 bytes from 10.223.0.150: icmp_seq=0 ttl=64 time=3.496 ms
            64 bytes from 10.223.0.150: icmp_seq=1 ttl=64 time=10.936 ms
            64 bytes from 10.223.0.150: icmp_seq=2 ttl=64 time=3.443 ms
            64 bytes from 10.223.0.150: icmp_seq=3 ttl=64 time=2.728 ms
            64 bytes from 10.223.0.150: icmp_seq=4 ttl=64 time=3.497 ms
            64 bytes from 10.223.0.150: icmp_seq=5 ttl=64 time=3.400 ms
            64 bytes from 10.223.0.150: icmp_seq=6 ttl=64 time=3.168 ms
            64 bytes from 10.223.0.150: icmp_seq=7 ttl=64 time=3.219 ms
            64 bytes from 10.223.0.150: icmp_seq=8 ttl=64 time=2.796 ms
            64 bytes from 10.223.0.150: icmp_seq=9 ttl=64 time=2.916 ms
            64 bytes from 10.223.0.150: icmp_seq=10 ttl=64 time=64.166 ms
            64 bytes from 10.223.0.150: icmp_seq=11 ttl=64 time=2.647 ms
            64 bytes from 10.223.0.150: icmp_seq=12 ttl=64 time=4.295 ms
            64 bytes from 10.223.0.150: icmp_seq=13 ttl=64 time=12.701 ms
            64 bytes from 10.223.0.150: icmp_seq=14 ttl=64 time=6.396 ms
            64 bytes from 10.223.0.150: icmp_seq=15 ttl=64 time=2.774 ms
            64 bytes from 10.223.0.150: icmp_seq=16 ttl=64 time=5.131 ms
            64 bytes from 10.223.0.150: icmp_seq=17 ttl=64 time=2.943 ms
            64 bytes from 10.223.0.150: icmp_seq=18 ttl=64 time=5.845 ms
            64 bytes from 10.223.0.150: icmp_seq=19 ttl=64 time=2.695 ms
            64 bytes from 10.223.0.150: icmp_seq=20 ttl=64 time=8.678 ms
            64 bytes from 10.223.0.150: icmp_seq=21 ttl=64 time=5.018 ms
            64 bytes from 10.223.0.150: icmp_seq=22 ttl=64 time=2.625 ms
            64 bytes from 10.223.0.150: icmp_seq=23 ttl=64 time=11.066 ms
            64 bytes from 10.223.0.150: icmp_seq=24 ttl=64 time=8.588 ms
            64 bytes from 10.223.0.150: icmp_seq=25 ttl=64 time=2.556 ms
            64 bytes from 10.223.0.150: icmp_seq=26 ttl=64 time=3.735 ms
            64 bytes from 10.223.0.150: icmp_seq=27 ttl=64 time=2.621 ms
            64 bytes from 10.223.0.150: icmp_seq=28 ttl=64 time=9.022 ms
            64 bytes from 10.223.0.150: icmp_seq=29 ttl=64 time=2.882 ms
            64 bytes from 10.223.0.150: icmp_seq=30 ttl=64 time=7.700 ms
            ^C--- 10.223.0.150 ping statistics ---
            31 packets transmitted, 31 packets received, 0% packet loss
            round-trip min/avg/max/stddev = 2.556/6.893/64.166/10.847 ms
            
            1 Reply Last reply Reply Quote 0
            • M
              Moderator ModalAI Team
              last edited by 10 Oct 2023, 16:00

              there isn't an obvious reason this would be the case, we will investigate

              C 1 Reply Last reply 10 Oct 2023, 19:12 Reply Quote 0
              • C
                Chase Riley @Moderator
                last edited by 10 Oct 2023, 19:12

                @Moderator Thanks. Let me know how it goes. I am pretty much at a stand still until I can get a reliable link from the Doodle Radios on 1.0.0.

                1 Reply Last reply Reply Quote 0
                • J
                  James Strawson ModalAI Team
                  last edited by 10 Oct 2023, 19:48

                  between SDK 0.9.5 and SDK 1.0 we disabled the mavlink packet aggregation in voxl-mavlink-server which used to be on by default. This was originally a patch to reduce wifi traffic on VOXL1 + Flight Core systems that forwarded full offboard-rate traffic through WiFi since the implementation of PX4 on VOXL2's DSP only sends slow data rate mavlink telemetry to the ground control station, we didn't find this feature necessary anymore and disabled it by default.

                  You can re-enable the mavlink aggregation feature with the udp_mtu field in /etc/modalai/voxl-mavlink-server.conf, the JSON file contains the following description of the field:

                   * udp_mtu - maximum transfer unit for UDP packets back to GCS. voxl-mavlink-server
                   *           will bundle up packets for the GCS into a single UDP packet with 
                   *           a maximum size of this. This might help with some radios.
                   *           This feature is off by default to let the network stack handle aggregation.
                   *           Set to 0 to disable this feature and send one UDP packet per msg.
                   *           Set to something like 500 to bundle a handful of packets together.
                  

                  I suspect doodlelabs might not be well optimized for very small mavlink messages or is not capable of doing its own aggregation. Try a few different values such as 100, 200, 300 etc and see if you get any improvement.

                  Best,
                  James

                  C 1 Reply Last reply 12 Oct 2023, 15:30 Reply Quote 0
                  • C
                    Chase Riley @James Strawson
                    last edited by 12 Oct 2023, 15:30

                    @James-Strawson @tom @Moderator @Chad-Sweet So it does seem there was a change from SDK .0.9.5 to 1.0.0. This fixed the problem. I changed the value to 500 and it links up quickly and I am able to complete calibrations. I would recommend that this be added to the Doodle Labs Modems Documentation.

                    J 1 Reply Last reply 12 Oct 2023, 19:22 Reply Quote 0
                    • J
                      James Strawson ModalAI Team @Chase Riley
                      last edited by 12 Oct 2023, 19:22

                      @Chase-Riley Glad to hear. Does it also perform as expected at lower values such as 100 or 200? a value as high as 500 will increase latency when only sending small mavlink messages.

                      C 2 Replies Last reply 16 Oct 2023, 12:45 Reply Quote 0
                      • C
                        Chase Riley @James Strawson
                        last edited by 16 Oct 2023, 12:45

                        @James-Strawson I had no luck with 100 or 200. I can try some other values, however I have not seen an issue with the value of 500 yet.

                        1 Reply Last reply Reply Quote 0
                        • C
                          Chase Riley @James Strawson
                          last edited by 20 Oct 2023, 18:45

                          @James-Strawson Quick update. I was able to lower the value to 300 and it seems to work as well.

                          1 Reply Last reply Reply Quote 1
                          • C Chase Riley referenced this topic on 28 Dec 2023, 21:51
                          1 out of 11
                          • First post
                            1/11
                            Last post
                          Powered by NodeBB | Contributors