# voxl communications completely locking up

Source: https://forum.modalai.com/topic/258/voxl-communications-completely-locking-up
Category: Flight Core and Legacy VOXL (https://forum.modalai.com/category/9/flight-core-and-legacy-voxl)
Tags: voxl
Posted: 2021-05-28 15:46:48 UTC by benjamin linne
Replies: 19 · Views: 4207

## benjamin linne · 2021-05-28 15:46:48 UTC

Hi,

I'm trying to run ros2 on the voxl and it appears to use more data that ros1 and when launching several nodes, sometimes after a while the voxl completely locks up and prevents adb, wifi, or microhard from communicating. My theory is the ipv4 buffer is getting overflown and stops all ip connections. This could make sense since adb also runs on ip.

Is there any way to verify this is the problem and not something else? Is there another way to shell into the voxl without using adb or ip so I can perform and soft-reboot? I've run tests on the power input and that all seems good. Also the flight controller still works when the voxl is locked up. The only fix for now is a hard-reboot

## Reply by benjamin linne · 2021-05-28 15:56:12 UTC

Edit, it appears after waiting some time the adb and wifi interfaces resume working, but the microhard interface refuses to work. The lights on the microhard indicate it's still connected, but I can't ping the voxl and the voxl can't ping the base station microhard

## Reply by Chad Sweet (ModalAI staff) · 2021-05-28 16:04:46 UTC

You can connect to ADB via USB, is that what you mean?

## Reply by benjamin linne · 2021-05-28 16:52:29 UTC

Yes via USB, however I think that still uses the ip protocol

## Reply by Chad Sweet (ModalAI staff) · 2021-05-28 16:58:22 UTC

I guess I don't know for sure, but I didn't think ADB used IP unless it was ADB over WiFi.

Can you run [voxl-cpu-monitor](https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-cpu-monitor)? You might simply be overloading the CPU.

## Reply by benjamin linne · 2021-05-28 17:19:49 UTC

voxl-cpu-monitor doesn't appear to output anything other than "Init complete, entering main loop"

## Reply by Chad Sweet (ModalAI staff) · 2021-05-28 17:56:46 UTC

Maybe you're on an older release. voxl-perfmon is the older version of cpu-monitor

## Reply by benjamin linne · 2021-05-28 19:31:45 UTC

I wasn't able to run voxl-cpu-monitor but I made a simple bash script that blinks the voxl led every second and it stops blinking once the voxl stops responding. Also I tried tweaking the settings mentioned here https://docs.ros.org/en/ros2_documentation/dashing/Guides/DDS-tuning.html#cross-vendor-tuning, but that didn't help

## Reply by benjamin linne · 2021-06-01 11:40:02 UTC

@Chad-Sweet how do you recommend debugging this? Is there a system log I can view or any way to identify what is causing the lockup? Also what "release" has voxl-cpu-monitor? I don't think it's included in the latest image.

## Reply by Chad Sweet (ModalAI staff) · 2021-06-01 14:36:23 UTC

voxl-perfmon is here https://gitlab.com/voxl-public/utilities/voxl-utils/-/blob/master/bin/voxl-perfmon

it will let you know if CPU is maxing out which is the most likely scenario. Do you have a fan on the Snapdragon? that will help maximize CPU performance

If you're launching too much stuff at the same time, can you manually launch incrementally until you get it figured out?

## Reply by benjamin linne · 2021-06-01 21:09:18 UTC

Hi @Chad-Sweet We have a fan and the cpu doesn't appear to be overloaded. Is there any system log capabilities to help identify why the system is locking up and requiring a hard reboot?

## Reply by Eric Katzfey (ModalAI staff) · 2021-06-01 22:09:57 UTC

If you run ```
dmesg -wH``` in a terminal it will show you all kernel messages and will continually update so you can see any last messages.

## Reply by benjamin linne · 2021-06-10 20:28:55 UTC

@Eric-Katzfey We're having trouble identifying the errors in dmesg. Is there a watchdog timer that could automatically restart the voxl? This will help debugging uas in a remote location without needing a hard reset.

## Reply by Eric Katzfey (ModalAI staff) · 2021-06-11 15:27:06 UTC (in reply to benjamin linne)

@benjamin-linne I'm not sure what you mean by a watchdog timer. Usually a watchdog requires a check-in from the threads / processes it is monitoring and will automatically reset the board if the check-in does not happen. You could certainly implement something like that on your own.

## Reply by benjamin linne · 2021-06-11 19:49:19 UTC (in reply to Eric Katzfey)

@Eric-Katzfey Could you point me to how I can setup watchdog timers on the voxl?

## Reply by Eric Katzfey (ModalAI staff) · 2021-06-11 20:14:09 UTC (in reply to benjamin linne)

@benjamin-linne I'm sorry, but I'm really not sure what you are looking for. There are no generic watchdog timers that I am aware of. You would need to make something custom.

## Reply by benjamin linne · 2021-06-11 20:47:23 UTC

@Eric-Katzfey I'm looking for documentation on the hardware watchdog timers onboard the voxl's snapdragon microcontroller

## Reply by Eric Katzfey (ModalAI staff) · 2021-06-13 18:25:16 UTC

Unfortunately, I'm not aware of any such documentation.

## Reply by benjamin linne · 2021-06-14 12:31:34 UTC

@Eric-Katzfey I was referring to the documentation for the specific registers that would need to be modified to enable and configure it since I'm familiar with embedded systems that allow that functionality. Another possibility is to add the /dev/watchdog kernel module, however that would require an update to the operating system. If this is possible, it would greatly improve the robustness since we are encountering many cpu lockups that require human intervention to perform hard reboots. For remote applications this is potentially a critical feature

## Reply by Eric Katzfey (ModalAI staff) · 2021-06-14 14:14:08 UTC

The kernel build is posted at https://gitlab.com/voxl-public/system-image-build/voxl-build