connected to voxl-mavlink-server but no messages from PX4

jay.miller

I'm having an issue with a couple of our m500 platforms (some running px4 v1.11, some running px4 v1.13) where we can no longer connect to QGC through wifi, but can connect to it via USB plugged into Flight Core.

Some symptoms:

I can ssh to VOXL board fine and have confirmed that the voxl-vision-px4.conf file is pointed to the correct qgc_ip. Other m500 platforms are able to connect to the QGC host laptop on the same wifi network at the same time, so I don't believe this is a network infrastructure issue.
I have run voxl-configure-mpa --factory-reset to bring the config files back in line (and then reconfigured the qgc_ip parameter).
At this point, from ssh'ed into the VOXL board, if I run voxl-px4-shell, I get into the PX4's Nuttx interactive shell. However, from that prompt, commands/messages hang. dmesg was the only command that would give some output, but then that stopped responding. At first, I thought this meant that the PX4 board was having issues.
However, if I go to the MAVLINK Console within QGC when I am connected through USB to Flight Core, I'm able to interact with px4 commands through that shell. I can run param reset_all successfully (as judged by examining parameters in QGC and seeing them change). Resetting the parameters to factory defaults didn't fix the connection issue. Loading parameters from a working v1.13 platform doesn't fix the connection issue. If I look at MALINK outputs in QGC, these values are updating.
Going back to ssh-ing to the VOXL board, and run voxl-vision-px4 status, I see my config file settings, then the log as things initialize, with the last line as WARNING: connected to voxl-mavlink-server but no messages from PX4

Maybe random, maybe relevant:

I'm having a hard time trying to re-flash the PX4 (in QGC, plugging in the USB cable directly to laptop the firmware update screen gives me Found device: Pixhawk Error: Get Board Info: Sync: Send Command: Get Command Response: Timeout waiting for bytes to be available
I've tried following these instructions through a Parallels VM (https://docs.modalai.com/flight-core-v2-firmware/), but things seem to hand waiting for the bootloader.

This feels like the connection between VOXL and PX4 is somehow broken, but both components seem OK?

Any help here would be much appreciated and I'm happy to provide more information or follow any ideas.

For the platform at question:

voxl-inspect-services --version gives:
All my package versions:
When I look at voxl-vision-px4 status I get (sorry for the multiple screenshots here):

jay.miller

Also, sorry, also saw issue 1586 and can confirm that external_fc parameter in voxl-mavlink-server.conf is false

jay.miller

@tom sorry for pinging you but I've seen you active in the discussions lately! Do you have any ideas here?

tom

@jay-miller Can you tell what the internal hardware of the M500 is? We've offered a few different variations:

these are the different possible configs:

single board solution: VOXL-Flight: https://docs.modalai.com/voxl-flight/
dual-board solution: VOXL + Flight Core: https://docs.modalai.com/flight-core/
dual-board solution: VOXL + Flight Core v2: https://docs.modalai.com/flight-core-v2/

Can you tell which variaton these M500s have?
You can also attach photos here and we can determine

It also never hurts to upgrade to the latest SDK: 0.9.5 as there are some small tweaks that may help

jay.miller

@tom Sorry for the delay, was having login issues.

We have a dual board with Flight Core 2. I'll try upgrading the SDK, but I'm a bit wary given we have a fleet of these puppies!

Thanks!

Jay

Eric Katzfey

@jay-miller For the m500 with a dual board solution the two boards communicate via a serial link. It's possible that there is an issue with this connection. Can you take the serial cable from one of the m500 that has this problem and swap it with one that doesn't have the problem? That could help isolate the cable as the problem. If you can convince yourself that the cable is fine then you can check each serial port individually.

jay.miller

@Eric-Katzfey Thanks for the tip!

The serial cable seemed to be OK, but updating the system to 0.9.5 via a fastboot recovery seemed to solve the problem. I think this issue relates to https://forum.modalai.com/topic/525/voxl-adb-or-wi-fi-no-longer-communicating but with the added symptom that I could ssh into the VOXL board over wifi as well as ADB in, but that the mavlink connection really seemed troublesome and QGC wasn't connecting over wifi. I just used VOXL Platform 0.9.5 for my update and that seemed to save things. (for those reading, make sure your extrinsics file is saved over on the /data/modalai partition! Mine was over there through good luck rather than foresight and confirmation on my part...)