ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    voxl-px4 service hangs at startup sometimes with SDK 1.1.2

    VOXL 2
    3
    18
    765
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Eric KatzfeyE
      Eric Katzfey ModalAI Team @Aaky
      last edited by

      @Aaky What drone are you using? Is it a custom drone? In your first post you included the output from px4. But it doesn't look complete. Are you sure that is the complete log? If so, does it look like that every time?

      A 1 Reply Last reply Reply Quote 0
      • A
        Aaky @Eric Katzfey
        last edited by

        @Eric-Katzfey Yes Eric its custom drone and the logs are actually complete posted in first post. Still if possible I will see if I can provide you with more logs. Any idea from these logs about the problem?

        Eric KatzfeyE 1 Reply Last reply Reply Quote 0
        • Eric KatzfeyE
          Eric Katzfey ModalAI Team @Aaky
          last edited by

          @Aaky There's only one log in the first message and it doesn't look complete. The command timed out and Command failed messages seem to be related to the attempt to start the voxl2_io driver. Can you comment that part out so that it skips trying to start the voxl2_io driver and see if it comes up then?

          A 1 Reply Last reply Reply Quote 0
          • A
            Aaky @Eric Katzfey
            last edited by Aaky

            @Eric-Katzfey Actually this driver failing happens very randomly, not always and also not because of any one particular driver in general. For example check one more dmesg and reset voxl-px4 log where voxl2_io driver spawned up correcly but still GPS LED wasn't blinking red and neither px4 got connected to GCS.

            voxl-px4-logs : https://drive.google.com/file/d/1PHqQLldXs5XFKhHvenWqy-0nDLUOfbDB/view?usp=sharing

            dmesg logs: https://drive.google.com/file/d/1ZOK7-ypm_9BfgbE4khD2NLKFdTecTHXD/view?usp=sharing

            Let me know any hints to further debug this issue.

            Alex KushleyevA Eric KatzfeyE 2 Replies Last reply Reply Quote 0
            • Alex KushleyevA
              Alex Kushleyev ModalAI Team @Aaky
              last edited by

              @Aaky , can you please confirm that you can reproduce the issue even with GPS driver completely disabled?

              A 1 Reply Last reply Reply Quote 0
              • A
                Aaky @Alex Kushleyev
                last edited by

                @Alex-Kushleyev Yes I confirm even if I disable GPS driver can reproduce this issue.

                1 Reply Last reply Reply Quote 0
                • Eric KatzfeyE
                  Eric Katzfey ModalAI Team @Aaky
                  last edited by

                  @Aaky Those 2 logs look normal except for the GPS taking a while to finally connect to the unit and advertise sensor_gps topic. But nothing to indicate there was any crash. So the GPS LED wasn't illuminated at all? Or was a different color? And PX4 not connecting to GCS doesn't necessarily mean that PX4 is having any problems.

                  A 1 Reply Last reply Reply Quote 0
                  • A
                    Aaky @Eric Katzfey
                    last edited by Aaky

                    @Eric-Katzfey Yes PX4 dosent crash but dosent get connected to QGC inspite of having good connectivity (I have been performing ping tests). GPS LED dosent blink at all when this particular scenario occurs. As I have told before this is very random occuring problem.

                    Eric KatzfeyE 1 Reply Last reply Reply Quote 0
                    • Eric KatzfeyE
                      Eric Katzfey ModalAI Team @Aaky
                      last edited by

                      @Aaky I'm having trouble following your comments. You say that px4 does not crash. But in the first log you posted it stops only partially through the startup script without any error indications. In that case is PX4 still running? If so, it seems like it is stuck somehow. If not, perhaps something is killing the px4 process. When this happens, adb into the voxl2 and look at voxl-inspect-services. Does it show voxl-px4 enabled and running? You say you are running this on a custom drone. Do you also have a reference drone from ModalAI? If so, does everything work fine on that drone? That is the main purpose that we sell those reference drones. It allows you to see everything working so that when you build a custom drone and something doesn't work you can compare against the reference drone.

                      A 1 Reply Last reply Reply Quote 0
                      • A
                        Aaky @Eric Katzfey
                        last edited by Aaky

                        @Eric-Katzfey Hi Eric, My apologies for the confusion.
                        I will correct my statement, PX4 never crashes, it just gets stuck which means the service is always running but never connects to QGC. On our custom drone post power boot up whenever we see GPS LED isn't blinking at all (neither red nor green) we come to know PX4 service is stuck somewhere. At this point if I ssh into VOXL2 and check voxl-inspect-services, I can see vozl-px4 service is definately running but GPS LED isn't blinking and also PX4 isn't connecting to QGC. These are the observations. Then if I do systemctl restart vozl-px4 , then sometimes problem gets solved and PX4 starts without getting stuck anywhere. I clarify once again, PX4 never crashes but just gets stuck somewhere. All all the logs starting from my first post involves the issue which I have described in this comment.
                        I am very old customer of ModalAI and have been working since voxl1 on multiple custom drones so I definitely understand the underlying architecture Eric. 🙂

                        I hope this comment clarifies the confusion.

                        Eric KatzfeyE 1 Reply Last reply Reply Quote 0
                        • Eric KatzfeyE
                          Eric Katzfey ModalAI Team @Aaky
                          last edited by

                          @Aaky Okay, this isn't anything we have encountered before. I'm not sure what could make PX4 get stuck like that. When it is in the stuck condition what does CPU utilization look like?

                          A 1 Reply Last reply Reply Quote 0
                          • A
                            Aaky @Eric Katzfey
                            last edited by

                            @Eric-Katzfey Here's screenshot of cpu utilization when this happpens. Everything looks normal.

                            Screenshot from 2024-03-24 20-07-53.png

                            Eric KatzfeyE 1 Reply Last reply Reply Quote 0
                            • Eric KatzfeyE
                              Eric Katzfey ModalAI Team @Aaky
                              last edited by

                              @Aaky Well, unfortunately it isn't obvious what is going wrong. You will have to start by disabling everything and when px4 comes up reliably start enabling things to find out the culprit.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Powered by NodeBB | Contributors