Bots Unlimited Wifi Crash

[Hector Nevarez]

Hi John,

I can help take a look at this issue. I'll first need to try and reproduce this issue, so give me a bit here to try and do so.

Also just for my understanding can you send the output of voxl-wifi getmode so I can which wifi mode the device was configured in.

[John Keller]

@Hector-Nevarez Hi Hector,

Here is the output of voxl-wifi getmode

WiFi is currently set up as follows:
	Mode: station
	Hardware: Concurrent STA+AP capable (AP:uap0, STA:mlan0)
	Station: Active on mlan0 - "Chiron"
	SoftAP: Disabled

Thank you!

[LukeNow]

Hi John,

We are still looking at the issue and trying to reproduce and will get back to you if we find anything.

If its possible could you also upload that /data/dump file to this ticket? If the size is not supported possibly uploading to google drive or similar file sharing service for us to inspect would be very helpful.

Thanks!

[John Keller]

Thank you for looking into it. Here is a google drive link. [https://drive.google.com/drive/folders/1cRrAo7wXScs-se2wVErIZE_t8HV11EbR?usp=sharing](link url)
containing the /data/dump. dump_958 is from a crash that just happened. The other files in there are things that claude thought might be relevant.

[LukeNow]

Hi John,

Thanks for the data dumps. Could you run this wifi_capture.sh on your board and send us the data when you reproduce the crash along with the dump file again? It should be here along with a simple readme https://drive.google.com/drive/folders/14nMhkvOMeHJ6zLJYMW4scAglTfkRYY8N?usp=sharing

Could you also briefly describe how you are connecting/what devices you are connecting? Also does this happen under any specific workloads or is it sporadic? It seems like a firmware crash so any additional information would be very useful if we end up needing to report it.

Thanks!

• Luke

[John Keller]

Hi Luke,

I uploaded the wifcap and corresponding dump file here https://drive.google.com/drive/folders/11nv9jt6RslnnnDXi7DgAw6W-eMBwbSsG.

The crash is sporadic in general but I was able to make it happen pretty consistently by viewing the hires_small_color image though the web gui. We have the rgb camera, forwards and downward tracking camera, and tof sensor. The only modifications we have done are removing the downward range sensor, adding some neopixel leds, and putting a larger ground plane on the gps. We have seen the crash before removing the range sensor and adding the leds. We usually run a ros2 autonomy stack and stream the hires camera via gstreamer. We see it happen randomly while doing this.

Thanks,
John

[LukeNow]

Hi John,

Thank you for the information it was very helpful for analysis.
It seems as though the firmware crash happens on stream timeouts and causes the firmware to crash.
Here is a potential workaround with the installer, I kept all the claude output in case that helps with any analysis on your end: https://drive.google.com/drive/folders/1sbwS3RZ4wgmC2raH7SnDKZz3OYSOavSg?usp=sharing

If you could try running that and using the device normally, hopefully that stops it from crashing. We also do have routing on that expansion board for the RESET pin but for some reason the firmware does not respect it. If you keep seeing the issue that might be the next place to look, but NXP has marked this chip as EOL and won't support any fixes there, so our best hope is to workaround whatever is happening.

Let us know how that goes and any other findings. Thanks!

• Luke

[John Keller]

Hi Luke,

Sorry for the late response. When I ran the verification after installing and power cycling, I got this:

unit enabled : static
unit state   : failed
[up=112s] driver not present (mlan0 or /proc/mwlan/adapter0 missing)

RESULT: NOT ACTIVE -- ADDBA timeout reads '<unreadable>', expected 0

It seems like it is because my system is missing mlanutl. Claude thinks there is a specific of version of that program that I might need. If you know the version, I can build it or if you wanted to sent the binary that would work too, whichever is easier. Here is claude's full summary in case it helps:

I read the bundle and ran read-only checks against the drone (no changes made). The install is fine — the problem is a missing dependency the script never names.

What's actually wrong

mlanutl does not exist on this board. Nothing else is broken.

On 10.3.1.82 right now:

check	result
mlan0	up, holds 10.3.1.82/16 — you are SSH'd in over it
/proc/mwlan/adapter0	present (config, mlan0/, uap0/)
lsmod	usbxxx + mlan loaded, USB dev 1286:204e (post-firmware, normal)
mlan0 registered at	uptime 7.9 s, associated at 11.4 s
find / -name mlanutl*	nothing — not in /usr/sbin, not in any package

The botsu-sparrow 1.0.8-r0 deb ships 9 files (2 .ko, 6 firmware blobs, sparrow.conf) and no userspace tools; ModalAI's voxl2-wlan doesn't provide mlanutl either. Their bench board has it, yours never did.

Why that produces those two misleading messages

sparrow-ba-fix.sh:53 —

have_driver() {
    [ -d "$ADAP" ] || return 1
    [ -x "$MLANUTL" ] || return 1     # <-- fails here
    ip link show "$IFACE" >/dev/null 2>&1 || return 1
}

MLANUTL falls back to /usr/sbin/mlanutl, which doesn't exist, so have_driver returns false while the driver is perfectly healthy. Every caller then lies:

• status prints driver not present (mlan0 or /proc/mwlan/adapter0 missing) — both of which are present.
• apply-wait spins 15 × 2 s and prints mlan0 did not appear within 30s, hence unit state: failed.
• verify's own mlanutl mlan0 addbapara produces nothing → '<unreadable>'.

The journal confirms the wiring works: udev fired on mlan0 creation and started the unit at up=8 s (matching the 7.9 s dmesg line), and it died at up=38 s — exactly the 30 s loop. (The journal's wall clocks look 13 min apart because the VOXL2 has no RTC and the clock jumps once the network syncs; trust the up= numbers.) unit enabled: static is also expected — the unit has no [Install] section on purpose; udev starts it.

Meanwhile the crash trigger is live and unmitigated — /proc/mwlan/adapter0/mlan0/debug shows BA streams on tid 0 and tid 4, win_size 64, at the default 65535 TU timeout.

What to change

1. Ask ModalAI for an aarch64 mlanutl built against this driver, and ask them to add it to the botsu-sparrow deb. The version string to quote them:
   USB8997---16.197.121.p2-MX5X16505.p7.2-(FP197), firmware 16.197.121.
   (mlanutl lives in NXP's mxm_wifiex tree at mapp/mlanutl — buildable yourself, but it drives private ioctls whose subcommand numbering tracks the driver release, so ask which release the sparrow build came from rather than grabbing an arbitrary tag.)
2. Drop it at /usr/sbin/mlanutl, chmod 755. Then no power cycle is needed — it's a runtime ioctl: sh /usr/sbin/sparrow-ba-fix.sh apply, then sh sparrow-ba-install.sh verify should read ADDBA timeout = 0. The existing udev rule + oneshot will then handle it at every boot with no reinstall.
3. Consider reinstalling as install monitor instead of oneshot — the setting is runtime-only and silently reverts on a module reload or warm reset, which is exactly the situation you're in after a wifi wedge.

Two things worth sending back to ModalAI with the request: the deb ships no mlanutl, and have_driver() conflates "tool missing" with "driver missing" — that's what sent you looking at the power cycle instead of the toolchain. Same as the region_code note, it's a packaging gap on their side.

Also note robot/setup/copy.sh runs this installer on every drone you provision, so all of them will fail verify identically until mlanutl is in the image.

[LukeNow]

Hi John,

No worries, also apologies for the delayed response. I should have asked for the SDK version you are on since we started packaging a voxl-sparrow-utils package that has that mlanutl bin that was missing in a later SDK version, I included the deb in the installer which should hopefully install correctly.

Here is the new one to try with the README.MD with the instructions: https://drive.google.com/drive/folders/1SBCSf8_EZkkYV3GnvDhC2poD7ckjDLWq?usp=sharing , Thanks!

• Luke

[DavidTorres]

Hi, I'm running a Bots Unlimited Sparrow (M0213 VOXL 2 Wi-Fi add-on) on a VOXL 2 M0054.

Setup:

• System image: 1.8.09-M0054-14.1a-perf-20261009 (/etc/version)
• voxl-suite 1.8.0
• Kernel: 4.19.125
• Packages: botsu-sparrow 1.0.8-r0, botsu-sparrow-utils 1.0.14, voxl2-wlan 1.0-r0
• Firmware/driver: USB8997---16.197.121.p2-MX5X16505.p7.2-(FP197)
• Station mode, 5 GHz, 80 MHz. Country set manually with iw reg set (does not persist across reboot)
• Mitigation installed with the sparrow-ba bundle, power-cycled. verify reports ADDBA timeout 0, mitigation ACTIVE, one BA stream.

Crash (during an iperf3 UDP test at -b 800M; iperf3 reported about 142 Mb/s delivered):

• Timeout cmd id (...) RSSI_INFO [0xa4]
• tx_pending = 415, wmm_tx_pending[0] = 366
• fw_dump_timer fired, Start to process hanging, then repeated Block woal_cfg80211_scan in abnormal driver state
• dmesg | grep "Timeout cmd id = 0xd0" returns nothing, so this doesn't appear to be the DELBA path the workaround covers.
• No /data/dump_* directory was created.

Questions:

1. Is the RSSI_INFO timeout under TX load a known issue? Is there a fix or firmware update?
2. Is there a recommended maximum offered load to avoid transmit queue buildup?
3. Is botsu-sparrow 1.0.8-r0 the intended driver for this image, alongside botsu-sparrow-utils 1.0.14?
4. How do we persist the regulatory domain on this image?

See the dmesg tail https://drive.google.com/file/d/1hVnhbvKdmFDvvVXIw6YxXD4b4HGkP1hO/view?usp=sharing

[LukeNow]

Hi @davidtorres ,

To answer some of your questions:

1. The RSSI_INFO looks like a symptom of something else and is not a known issue. The firmware for the Bots Sparrow board seems to be EOL so no updates or any fixes as far as I know,
2. There is not an maximum load throughput as it depends on the signal strength, but 800M is very high and would be expected to queue packets, but it still should not be hanging like you are seeing.
3. Yes those versions are shipped together as a part of our SDK and are expected
4. The easiest way would be to set a udev rule that sets it on boot, something like
   ===========
   ACTION=="add", SUBSYSTEM=="net", KERNEL=="mlan0", RUN+="<path to iw> reg set <country>"
   ===========

My questions for you:

1. can you send the full dmesg log when you reproduce this along with the files produced by sparrow-ba-fix.sh collect from the earlier workaround
2. How often do you see this when doing your UDP test at -b 800M and do you see it when its slower than that, 200M 400M?
3. Did you run into this issue without the mitigation workaround from earlier installed? Was the issue earlier in the thread something you have also seen and made you apply the workaround?
4. Are you running iperf3 with the voxl as the client or the server? If you could include the full commands you used on the host and client side that would be very helpful

Thanks!