Immediate crashes in HITL

shawn_ricardo

Hello,

My platform:
Starling 2
hw platform: M0054
mach. var: 1.0.1
voxl-suite 1.4.3
system-image: 1.8.02-M0054-14.1a-perf
kernel: #1 SMP PREEMPT Mon Nov 11 22:47:44 UTC 2024 4.19.125

I've gotten Starling 2 HITL up and running according to https://docs.modalai.com/voxl2-PX4-hitl/. I can control the drone in simulation via RC and see video stream and control modes via QGC.

I am using this FTDI connection (https://www.amazon.com/gp/product/B07TXVRQ7V/ref=ox_sc_act_title_3?smid=AFLYC5O31PGVX&psc=1), where the JST GH is plugged into VOXL2 port J18. I made sure to not use the voltage, only TX/RX/GND.

The issue is that after takeoff, the quad almost immediately crashes by flipping or by suddenly executing some extreme acceleration. I'm not sure how to identify whether the issue lies in the simulation (e.g., latency, sensor input error) or in the drone commands and am hoping a dev may provide a next troubleshooting step.

Thanks!

shawn_ricardo

The input was:

qshell commander arm
qshell commander takeoff

During the hold, the quad flipped.

Here is the PX4 flight log, which shows interesting actuator behavior:
https://review.px4.io/plot_app?log=518ccf49-36ed-4509-b928-da3dda44c7fd

The motor output diverges significantly around 2:12

At this time, the roll angle, pitch angle, and yaw angle executed deviated significantly from the commanded values:

Lastly, there were no failsafe flags indicating a loss of sensor input (e.g., EKF divergence, GPS loss, etc)....

[Only non-zero streams have been selected in the image below]

Could this be an issue with the actuation commands coming from the VOXL2?

shawn_ricardo

Here is a readout of the qshell dsp_hitl status command, just to validate the connection:

Eric Katzfey

@shawn_ricardo What command did you use to start HITL? Are you trying hold mode with GPS?

shawn_ricardo

Hi @Eric-Katzfey,

To start HITL, I used voxl-px4-hitl

I ran the commands:

1) qshell commander arm
2) qshell commander takeoff

which will eventually go into HOLD mode.

As far as the sensor positioning input, I'm assuming that I'm using GPS because I followed this instruction to turn off qvio:

NOTE: Ensure that voxl-qvio-server is disabled. If it is enabled, it will confuse gazebo due to gazebo sending fake sensor data to the qvio algorithm and causing poor flight dynamics in HITL. This can be done with the systemctl disable voxl-qvio-server command.

Here are the services running via voxl-inspect-services. Note that voxl-px4 is not running because the sim is not running:

shawn_ricardo

@Eric-Katzfey , would you happen to recommend next troubleshooting steps?

Eric Katzfey

@shawn_ricardo I'll try it on my setup and see what happens.

Eric Katzfey

@shawn_ricardo I just tried it and it worked fine. But I had all services disabled except for voxl-wait-for-fs, voxl-mavlink-server, and voxl-cpu-monitor. So maybe try that.

Eric Katzfey

@shawn_ricardo Can you capture the log of startup messages for voxl-px4-hitl so I can look for any error messages, or incorrect settings?

shawn_ricardo

Hi @Eric-Katzfey ,

I've tried those settings and the drone is able to stay hovering much long than expected. the positioning appears to be better, as well.

What service(s) might be getting in the way of HITL performance?

Also, I noticed during flight the drone vibrates significantly in HITL as soon as I publish offboard commands while in HOLD mode according to PX4's offboard guidelines. As soon as I stop publishing to /fmu/in/trajectory_setpoint, the vibrations stop. It is as if the drone is toggling between offboard and hold modes.

I have "offboard_mode" set to "off" in /etc/modalai/voxl-vision-hub.conf. I am running my own ros2 offboard node. I've enabled voxl-cpu-monitor, voxl-microdds-agent, voxl-modem, and voxl-wait-for-fs. I am running the deb package of voxl_mpa_to_ros2 . The command I am running is ros2 run voxl_mpa_to_ros2 voxl_mpa_to_ros2_node

I've tested the ability to publish offboard commands while in position hold mode in a completely separate px4 sim environment (just regular PX4 sim) and verified that drone position does not "vibrate" while offboard commands are being publishing and hold mode is the active mode.

What could be the cause of this behavior issue?

Eric Katzfey

@shawn_ricardo To figure out which service could be causing the issue just try starting each one and see which one causes the issue.

shawn_ricardo

@Eric-Katzfey

Okay, I'll keep this in mind as I develop toward offboard control. If I find any more information, I'll follow up here.

Just a side note: I don't see this behavior in real-world flight during manual or position, seems to be isolated to HITL at this point in time.

Thanks!