Bots Unlimited Wifi Crash

[John Keller]

We are using the Bots Unlimited wifi board attached directly to our voxl2 board. It seems like the wifi driver will sometimes randomly crash with this message being repeated in dmesg: "Block woal_cfg80211_scan in abnormal driver state". When this happens the drone can no longer connect to wifi. We couldn't find any way in software to reboot the wifi board. To get it working again we have to power cycle the drone. Is there a fix for this?

[John Keller]

Here is the relevant part from dmesg

[ 2130.644295] FW trigger fw dump
[ 2130.644332] =====FW trigger dump====
[ 2130.644347] ==== Start Receive FW dump event  ====
[ 2130.645036] Create directory /data/dump_2130 successfully
[ 2130.645058] Firmware Dump directory name is /data/dump_2130
[ 2130.645078] === START DRIVER INFO DUMP===
[ 2130.645180] DRV dump data in /data/dump_2130/file_drv_info
[ 2130.653905] Drv info total bytes = 205002 (0x320ca)
[ 2130.653933] === DRIVER INFO DUMP END===
[ 2131.373918] ==== FW DUMP END: 1381912 bytes ====
[ 2131.374101] Start to process hanging
[ 2131.374170] Cancel all pending cmd and txrx queue
[ 2131.374381] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.374408] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.374430] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.374451] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.374472] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.374493] Block woal_cfg80211_del_key in abnormal driver state
[ 2131.477115] Block woal_cfg80211_scan in abnormal driver state
[ 2132.479524] Block woal_cfg80211_scan in abnormal driver state
[ 2133.481978] Block woal_cfg80211_scan in abnormal driver state
[ 2134.484164] Block woal_cfg80211_scan in abnormal driver state
[ 2135.486210] Block woal_cfg80211_scan in abnormal driver state
[ 2136.489052] Block woal_cfg80211_scan in abnormal driver state
[ 2137.491936] Block woal_cfg80211_scan in abnormal driver state

[Hector Nevarez]

Hi John,

I can help take a look at this issue. I'll first need to try and reproduce this issue, so give me a bit here to try and do so.

Also just for my understanding can you send the output of voxl-wifi getmode so I can which wifi mode the device was configured in.

[John Keller]

@Hector-Nevarez Hi Hector,

Here is the output of voxl-wifi getmode

WiFi is currently set up as follows:
	Mode: station
	Hardware: Concurrent STA+AP capable (AP:uap0, STA:mlan0)
	Station: Active on mlan0 - "Chiron"
	SoftAP: Disabled

Thank you!

[LukeNow]

Hi John,

We are still looking at the issue and trying to reproduce and will get back to you if we find anything.

If its possible could you also upload that /data/dump file to this ticket? If the size is not supported possibly uploading to google drive or similar file sharing service for us to inspect would be very helpful.

Thanks!

[John Keller]

Thank you for looking into it. Here is a google drive link. [https://drive.google.com/drive/folders/1cRrAo7wXScs-se2wVErIZE_t8HV11EbR?usp=sharing](link url)
containing the /data/dump. dump_958 is from a crash that just happened. The other files in there are things that claude thought might be relevant.

[LukeNow]

Hi John,

Thanks for the data dumps. Could you run this wifi_capture.sh on your board and send us the data when you reproduce the crash along with the dump file again? It should be here along with a simple readme https://drive.google.com/drive/folders/14nMhkvOMeHJ6zLJYMW4scAglTfkRYY8N?usp=sharing

Could you also briefly describe how you are connecting/what devices you are connecting? Also does this happen under any specific workloads or is it sporadic? It seems like a firmware crash so any additional information would be very useful if we end up needing to report it.

Thanks!

• Luke

[John Keller]

Hi Luke,

I uploaded the wifcap and corresponding dump file here https://drive.google.com/drive/folders/11nv9jt6RslnnnDXi7DgAw6W-eMBwbSsG.

The crash is sporadic in general but I was able to make it happen pretty consistently by viewing the hires_small_color image though the web gui. We have the rgb camera, forwards and downward tracking camera, and tof sensor. The only modifications we have done are removing the downward range sensor, adding some neopixel leds, and putting a larger ground plane on the gps. We have seen the crash before removing the range sensor and adding the leds. We usually run a ros2 autonomy stack and stream the hires camera via gstreamer. We see it happen randomly while doing this.

Thanks,
John

[LukeNow]

Hi John,

Thank you for the information it was very helpful for analysis.
It seems as though the firmware crash happens on stream timeouts and causes the firmware to crash.
Here is a potential workaround with the installer, I kept all the claude output in case that helps with any analysis on your end: https://drive.google.com/drive/folders/1sbwS3RZ4wgmC2raH7SnDKZz3OYSOavSg?usp=sharing

If you could try running that and using the device normally, hopefully that stops it from crashing. We also do have routing on that expansion board for the RESET pin but for some reason the firmware does not respect it. If you keep seeing the issue that might be the next place to look, but NXP has marked this chip as EOL and won't support any fixes there, so our best hope is to workaround whatever is happening.

Let us know how that goes and any other findings. Thanks!

• Luke

[John Keller]

Hi Luke,

Sorry for the late response. When I ran the verification after installing and power cycling, I got this:

unit enabled : static
unit state   : failed
[up=112s] driver not present (mlan0 or /proc/mwlan/adapter0 missing)

RESULT: NOT ACTIVE -- ADDBA timeout reads '<unreadable>', expected 0

It seems like it is because my system is missing mlanutl. Claude thinks there is a specific of version of that program that I might need. If you know the version, I can build it or if you wanted to sent the binary that would work too, whichever is easier. Here is claude's full summary in case it helps:

I read the bundle and ran read-only checks against the drone (no changes made). The install is fine — the problem is a missing dependency the script never names.

What's actually wrong

mlanutl does not exist on this board. Nothing else is broken.

On 10.3.1.82 right now:

check	result
mlan0	up, holds 10.3.1.82/16 — you are SSH'd in over it
/proc/mwlan/adapter0	present (config, mlan0/, uap0/)
lsmod	usbxxx + mlan loaded, USB dev 1286:204e (post-firmware, normal)
mlan0 registered at	uptime 7.9 s, associated at 11.4 s
find / -name mlanutl*	nothing — not in /usr/sbin, not in any package

The botsu-sparrow 1.0.8-r0 deb ships 9 files (2 .ko, 6 firmware blobs, sparrow.conf) and no userspace tools; ModalAI's voxl2-wlan doesn't provide mlanutl either. Their bench board has it, yours never did.

Why that produces those two misleading messages

sparrow-ba-fix.sh:53 —

have_driver() {
    [ -d "$ADAP" ] || return 1
    [ -x "$MLANUTL" ] || return 1     # <-- fails here
    ip link show "$IFACE" >/dev/null 2>&1 || return 1
}

MLANUTL falls back to /usr/sbin/mlanutl, which doesn't exist, so have_driver returns false while the driver is perfectly healthy. Every caller then lies:

• status prints driver not present (mlan0 or /proc/mwlan/adapter0 missing) — both of which are present.
• apply-wait spins 15 × 2 s and prints mlan0 did not appear within 30s, hence unit state: failed.
• verify's own mlanutl mlan0 addbapara produces nothing → '<unreadable>'.

The journal confirms the wiring works: udev fired on mlan0 creation and started the unit at up=8 s (matching the 7.9 s dmesg line), and it died at up=38 s — exactly the 30 s loop. (The journal's wall clocks look 13 min apart because the VOXL2 has no RTC and the clock jumps once the network syncs; trust the up= numbers.) unit enabled: static is also expected — the unit has no [Install] section on purpose; udev starts it.

Meanwhile the crash trigger is live and unmitigated — /proc/mwlan/adapter0/mlan0/debug shows BA streams on tid 0 and tid 4, win_size 64, at the default 65535 TU timeout.

What to change

1. Ask ModalAI for an aarch64 mlanutl built against this driver, and ask them to add it to the botsu-sparrow deb. The version string to quote them:
   USB8997---16.197.121.p2-MX5X16505.p7.2-(FP197), firmware 16.197.121.
   (mlanutl lives in NXP's mxm_wifiex tree at mapp/mlanutl — buildable yourself, but it drives private ioctls whose subcommand numbering tracks the driver release, so ask which release the sparrow build came from rather than grabbing an arbitrary tag.)
2. Drop it at /usr/sbin/mlanutl, chmod 755. Then no power cycle is needed — it's a runtime ioctl: sh /usr/sbin/sparrow-ba-fix.sh apply, then sh sparrow-ba-install.sh verify should read ADDBA timeout = 0. The existing udev rule + oneshot will then handle it at every boot with no reinstall.
3. Consider reinstalling as install monitor instead of oneshot — the setting is runtime-only and silently reverts on a module reload or warm reset, which is exactly the situation you're in after a wifi wedge.

Two things worth sending back to ModalAI with the request: the deb ships no mlanutl, and have_driver() conflates "tool missing" with "driver missing" — that's what sent you looking at the power cycle instead of the toolchain. Same as the region_code note, it's a packaging gap on their side.

Also note robot/setup/copy.sh runs this installer on every drone you provision, so all of them will fail verify identically until mlanutl is in the image.