PX4 qmi_error abort

Rowan Dempster

Hey ModalAI PX4 users, has anyone been running into qmi_error causing the PX4 process to abort? At Cleo it happens at boot about 1/20 or 1/100 times. After booting successfully it's more rare about 1/200 times or 1/500 times.
Here's the full error from journalctl:

terminate called after throwing an instance of 'qmi_error'
Mar 19 15:33:57 m0054 voxl-px4[1854]:   what():  qmi_client_send_msg_sync() failed, (client_id=)0, result=0: qmi service error (-2)
Mar 19 15:33:57 m0054 voxl-px4[1854]: /usr/bin/voxl-px4: line 140:  1868 Aborted                 GPS=$GPS RC=$RC OSD=$OSD EXTRA_STEPS=$EXTRA_STEPS px4 $DAEMON -s /usr/bin/voxl-px4-start
Mar 19 15:33:57 m0054 systemd[1]: voxl-px4.service: Main process exited, code=exited, status=134/n/a
Mar 19 15:33:57 m0054 systemd[1]: voxl-px4.service: Failed with result 'exit-code'.

Eric Katzfey

@Rowan-Dempster Yes, we used to see this happen on older SDK versions. It was an indication that the DSP was crashing. It was happening at about that frequency. But there have been multiple bug fixes since then and as far as I know it no longer happens. Are you using a recent version of VOXL SDK? Have you made any modifications to the SDK?

Rowan Dempster

@Eric-Katzfey Thanks for the response!

Are you using a recent version of VOXL SDK?

Cleo branched off of your repo at this tag: https://github.com/modalai/px4-firmware/tree/v1.14.0-2.0.36-dev

Have you made any modifications to the SDK?

Yup we actively development on the PX4 modules, including the controllers and the EKF that run on the DSP.

So it may be our code running on the DSP causing the DSP crash, or it could be related to the bugs in the https://github.com/modalai/px4-firmware/tree/v1.14.0-2.0.36-dev tag itself that you mentioned have been fixed.

As far as a path forward, are there any methods you can suggest for inspecting the DSP to find the root cause of crashes? Things we can add to the code, perhaps a debug mode we can run the DSP modules in, etc

Also, do you know of bug fix commits in your repo's mainline that we at Cleo can attempt to backport to our fork and see if we also no longer see the DSP crashes?

Thank you for your help,
Rowan

Rowan Dempster

@Eric-Katzfey Any insight into this ^

Eric Katzfey

@Rowan-Dempster Sorry, not sure why I didn't see your response. Let me look through the commits to see if any of those important bug fixes have been added since then.

Eric Katzfey

@Rowan-Dempster Of course v1.14.0-2.0.36 is extremely old and there have been a lot of improvements / fixes since then. But the fixes for DSP crashes were made in the modalai-slpi codebase. What version of modalai-slpi are you running? One critical bug fix was added in v1.1.9 and another in v1.1.14.

Rowan Dempster

@Eric-Katzfey I am not familiar with the "modalai-slpi" codebase, could you elaborate on what that is.