voxl communications completely locking up
-
Hi,
I'm trying to run ros2 on the voxl and it appears to use more data that ros1 and when launching several nodes, sometimes after a while the voxl completely locks up and prevents adb, wifi, or microhard from communicating. My theory is the ipv4 buffer is getting overflown and stops all ip connections. This could make sense since adb also runs on ip.
Is there any way to verify this is the problem and not something else? Is there another way to shell into the voxl without using adb or ip so I can perform and soft-reboot? I've run tests on the power input and that all seems good. Also the flight controller still works when the voxl is locked up. The only fix for now is a hard-reboot
-
Edit, it appears after waiting some time the adb and wifi interfaces resume working, but the microhard interface refuses to work. The lights on the microhard indicate it's still connected, but I can't ping the voxl and the voxl can't ping the base station microhard
-
You can connect to ADB via USB, is that what you mean?
-
Yes via USB, however I think that still uses the ip protocol
-
I guess I don't know for sure, but I didn't think ADB used IP unless it was ADB over WiFi.
Can you run voxl-cpu-monitor? You might simply be overloading the CPU.
-
voxl-cpu-monitor doesn't appear to output anything other than "Init complete, entering main loop"
-
Maybe you're on an older release. voxl-perfmon is the older version of cpu-monitor
-
I wasn't able to run voxl-cpu-monitor but I made a simple bash script that blinks the voxl led every second and it stops blinking once the voxl stops responding. Also I tried tweaking the settings mentioned here https://docs.ros.org/en/ros2_documentation/dashing/Guides/DDS-tuning.html#cross-vendor-tuning, but that didn't help
-
@Chad-Sweet how do you recommend debugging this? Is there a system log I can view or any way to identify what is causing the lockup? Also what "release" has voxl-cpu-monitor? I don't think it's included in the latest image.
-
voxl-perfmon is here https://gitlab.com/voxl-public/utilities/voxl-utils/-/blob/master/bin/voxl-perfmon
it will let you know if CPU is maxing out which is the most likely scenario. Do you have a fan on the Snapdragon? that will help maximize CPU performance
If you're launching too much stuff at the same time, can you manually launch incrementally until you get it figured out?
-
Hi @Chad-Sweet We have a fan and the cpu doesn't appear to be overloaded. Is there any system log capabilities to help identify why the system is locking up and requiring a hard reboot?
-
If you run
dmesg -wHin a terminal it will show you all kernel messages and will continually update so you can see any last messages. -
@Eric-Katzfey We're having trouble identifying the errors in dmesg. Is there a watchdog timer that could automatically restart the voxl? This will help debugging uas in a remote location without needing a hard reset.
-
@benjamin-linne I'm not sure what you mean by a watchdog timer. Usually a watchdog requires a check-in from the threads / processes it is monitoring and will automatically reset the board if the check-in does not happen. You could certainly implement something like that on your own.
-
@Eric-Katzfey Could you point me to how I can setup watchdog timers on the voxl?
-
@benjamin-linne I'm sorry, but I'm really not sure what you are looking for. There are no generic watchdog timers that I am aware of. You would need to make something custom.
-
@Eric-Katzfey I'm looking for documentation on the hardware watchdog timers onboard the voxl's snapdragon microcontroller
-
Unfortunately, I'm not aware of any such documentation.
-
@Eric-Katzfey I was referring to the documentation for the specific registers that would need to be modified to enable and configure it since I'm familiar with embedded systems that allow that functionality. Another possibility is to add the /dev/watchdog kernel module, however that would require an update to the operating system. If this is possible, it would greatly improve the robustness since we are encountering many cpu lockups that require human intervention to perform hard reboots. For remote applications this is potentially a critical feature
-
The kernel build is posted at https://gitlab.com/voxl-public/system-image-build/voxl-build
