Trouble with EM7565 startup...

Ed Sutter

Hi,
I've mentioned in other posts that we have a few cases where we need to use the EM7565 because of the LTE bands that it supports. Anyway, I've been working the last few days with the EM7565 plugged into an M2-to-USB adapter so that I can plug it in to the expansion USB connector on the Voxl-Flight-Deck (VFD).
To start off I made some progress with the modem (it registers with my LTE network!) when plugging it into my Ubuntu 18.04 host. That was a sanity check in preparation for plugging it into my VFD. Note that I have already worked with the RC7611 and the LTE-addon board successfully, so I've walked through the process once.

So, I started with no changes thinking that maybe it will "just work". NOPE . To trace the steps, I added a few lines to /etc/systemd/system/voxl-modem.service so I could view the log with journalctl -u voxl-modem. The output of this follows:

-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Wed 2022-09-21 20:31:04 UTC. --
Jan 01 00:00:07 apq8096 systemd[1]: Starting Modem Initialization...
Jan 01 00:00:09 apq8096 systemd[1]: Started Modem Initialization.
Jan 01 00:00:09 apq8096 voxl-modem-start.sh[2846]: VOXL detected...
Jan 01 00:00:09 apq8096 voxl-modem-start.sh[2846]: Initializing v2 modem...
Jan 01 00:00:09 apq8096 voxl-modem-start.sh[2846]: Waiting for ttyUSB2...
Jan 01 00:00:12 apq8096 voxl-modem-start.sh[2846]: ttyUSB2
Jan 01 00:00:13 apq8096 voxl-modem-start.sh[2846]: Initalizing cellular connection...
Sep 21 20:23:12 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:23:13 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:23:28 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 8
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:23:29 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:23:44 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 8
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:23:45 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:24:01 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 8
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:24:02 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:24:17 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 8
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:24:18 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:24:33 apq8096 voxl-modem-start.sh[2846]: Failed to open device
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 is available
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 10
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 9
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Unstick QMI countdown: 8
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Checking for system registration
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Attempt: 1
Sep 21 20:24:34 apq8096 voxl-modem-start.sh[2846]: Got exception when checking registration status
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:25:20 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist

Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:26:06 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist

Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:26:52 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist

Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:27:39 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist

Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:28:25 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist

Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: Configuring GPIO to pull modem out of reset
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: Waiting for the diag serial port(/dev/ttyUSB0) to activate
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/ttyUSB0
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: /dev/ttyUSB0 is available
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: Verifying existence of QMI port(/dev/cdc-wdm0)
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: checking for serial port: /dev/cdc-wdm0
Sep 21 20:29:11 apq8096 voxl-modem-start.sh[2846]: /dev/cdc-wdm0 does not exist
...

This is trace from voxl-modem-start.sh, as well as sierra.py and uqmi. Notice that the last 7 lines just keep repeating (I show that above with the blank line between each block) because for some reason /dev/cdc-wdm0 just goes away. Note that /dev/ttyUSB[1-2] also disappear.
I know I'm now working with a modem that you don't claim support for, but if you have any idea what may be causing this it sure would be appreciated. Also, where is the source for the uqmi tool?

tom

@Ed-Sutter Do you know how much power the m.2 to USB adapter you're plugging in is pulling? Depending on which add-on board you're plugging it into on VOXL, it may not be able to provide the required current. Some of our add-on boards' USB ports max out at 0.5 A (M0030, M0017, M0048).

When we've tested with 3rd party modem USB adapters in the past we've had to use our debug board v2 (M0078): https://www.modalai.com/products/voxl-debug-board-v2?_pos=4&_sid=bc3d50f8c&_ss=r

which allows you to provide a higher current to the USB port.

Source for uqmi lives here: https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-modem/-/tree/master/src/modem/uqmi

Ed Sutter

@tom Well, when I plug the modem into my Ubuntu host using one of these testers it shows a suspiciously low current (10mA), but it also doesn't quite work (yet) on that system. The command mmcli --modem=0 shows (snippets):

  Hardware |         manufacturer: Sierra Wireless, Incorporated
           |                model: EM7565
           |             revision: SWI9X50C_01.08.04.00 dbb5d0 jenkins 2018/08/21 21:40:11
           |         h/w revision: 10001
           |            supported: gsm-umts, lte
           |              current: gsm-umts, lte
           |         equipment id: 353533100239592
  --------------------------------
  System   |               device: /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1
           |              drivers: qcserial, qmi_wwan
           |               plugin: Sierra
           |         primary port: cdc-wdm0
           |                ports: ttyUSB0 (qcdm), ttyUSB2 (at), cdc-wdm0 (qmi),
           |                       wwp0s20f0u1i8 (net)
  --------------------------------
  Numbers  |                  own: 17327793967
  --------------------------------
  Status   |                 lock: sim-pin2
           |       unlock retries: sim-pin (3), sim-pin2 (3), sim-puk (10), sim-puk2 (10)
           |                state: registered
           |          power state: on
           |          access tech: lte
           |       signal quality: 59% (recent)

and the interface only comes up if I manually bring it up with ifconfig wwp0s20fu1i8 up ...

wwp0s20f0u1i8: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet6 fe80::2495:2f9d:cec5:53f  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 528 (528.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Notice, there's no inet address likely because [for some reason I have yet to understand] POINTOPOINT,NOARP flags are set. Any thoughts on that? (meanwhile I will get a powered USB hub for now just to see if it makes any difference).

Eric Katzfey

@Ed-Sutter The other thing you have to check is the USB vendor / device id. The Linux device drivers often have those hard coded into them for support. If those id's aren't in the driver then it won't set it up.

Ed Sutter

@Eric-Katzfey That shows up with lsusb...

Bus 001 Device 002: ID 1199:9091 Sierra Wireless, Inc.

but I assume you're talking about something more specific to wwan? Do you know what I would look for to see if my host meets that requirement? I've obviously got modemmanager, libmbim-utils and libqmi-utils installed, but maybe I need newer versions?

Eric Katzfey

@Ed-Sutter I'm speaking specifically of VOXL, not your host computer, but it's the same idea. It's the actual Linux device driver for that piece of hardware. For example: https://gitlab.com/voxl-public/system-image-build/apq8096-kernel/-/blob/main/drivers/net/usb/qmi_wwan.c#L772

Ed Sutter

@Eric-Katzfey got it... Thanks!

Ed Sutter

This post is deleted!

Ed Sutter

@Eric-Katzfey Update... I don't have your USB expansion board yet (being ordered), but I was able to get things working by just using a powered USB hub; so I'm assuming it will work the same with the new expansion board.

Ed Sutter

@tom I received the new USB add-on (M0078). I just wanna be sure I understand this... All I need to do to get more current is to move the jumper over to the USB-VBUS position? I assume that just bypasses any current-limiting protection that would be provided by the USB driver?
true?

tom

@Ed-Sutter You'll actually have the jumper on the USB PWR side as seen in this photo:
but otherwise yes you've got the right idea

Ed Sutter

@tom Cool! Just tried it with the EM7565...

ifconfig wwan0
wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.234.85.224  P-t-P:10.234.85.224  Mask:255.255.255.192
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:604 (604.0 B)  TX bytes:656 (656.0 B)

The modem came right up!
Thanks

tom

@Ed-Sutter Beautiful! Glad to hear that

Ed Sutter

@tom Do you guys have any plans to support the EM7565 with an alternative adapter?

tom

@Ed-Sutter Not that I know of at the moment. Is there a specific reason why you're choosing to go with this card as opposed to others? I know you mentioned earlier that it supports certain bands you need.

Ed Sutter

@tom Yea, that's it, we need a band that is not supported by any of the modems that fit your add-on board.
Anyway, now that I've had some time to work with it (the EM7565), things aren't as good as I initially thought...
I've been testing this in various ways and just getting more confused... Initially I was using voxl-streamer to push video over LTE to VLC on another machine. It starts up just fine (video is clean), but the video just stops after some random time (usually less than an hour). So I thought -maybe- my LTE service (I'm using a SIM card on my own AT&T account that is supposed to be used with a flip phone) was doing some throttling. I reduced the load to just open a TCP connection and send small messages (over LTE) every second. Same thing happens. Then I tried running the same thing but now just using WiFi while the wwan0 is up and alive... Same thing even on WiFi! Once it gets in this state, I can't ping out at all (on either interface), but all interfaces are up and I am able to ssh into VOXL. If I remove the EM7565, things are fine on WiFi.

Does this make any sense at all?
Can you think of anything I can do to diagnose this?
I have a small (cheap) USB power monitor in line with the modem and it never exceeds 100mA on the display.

tom

@Ed-Sutter Does it work if you only have one of the network interfaces up at once? For example, only having wwan0 up or only having wlan0 up? I'm wondering if some weird routing issue happens

Ed Sutter

@tom Good thought! I'll try turning down wlan0 while using wwan0...

tom

@Ed-Sutter And to be safe if you're in station mode I would set your ssid and pass to dummy values, i.e voxl-wifi station tmp tmp because I've seen in the past where wlan0 will try to bring itself back up after you manually force it down.

Ed Sutter

@tom Ok, I tried what you said (good idea by the way)... I turned down the wlan0 link and did a very minimial TCP/IP data transfer loop (about 25 characters a second) and after about 15 minutes things just got wonky. I terminated the test, and again, I wasn't able to ping out.
So this morning I was going to reverse the interfaces (enable wlan0 and turn down wwan0) and repeat, but didn't get that far cause I noticed this...

$: adb shell
/ # ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:A2:AB:4C:25
          inet addr:----------  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: ----------------/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:216 (216.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 18:47:3D:B1:BE:0F
          inet addr:192.168.1.230  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: ----------------/64 Scope:Link
          inet6 addr: ----------------------------/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:221 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3000
          RX bytes:45620 (44.5 KiB)  TX bytes:4536 (4.4 KiB)

wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:--.--.--.--  P-t-P:--.--.--.--  Mask:255.255.255.192
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:22 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2350 (2.2 KiB)  TX bytes:2164 (2.1 KiB)

/ # ping google.com
PING google.com (142.251.40.238) 56(84) bytes of data.
64 bytes from lga34s39-in-f14.1e100.net (142.251.40.238): icmp_req=1 ttl=55 time=38.8 ms
64 bytes from lga34s39-in-f14.1e100.net (142.251.40.238): icmp_req=2 ttl=55 time=43.0 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 38.858/40.949/43.041/2.101 ms
/ # ip link set dev wwan0 down
/ # ping google.com
ping: unknown host google.com
/ # ping 142.250.204.14
PING 142.250.204.14 (142.250.204.14) 56(84) bytes of data.
64 bytes from 142.250.204.14: icmp_req=1 ttl=117 time=199 ms
64 bytes from 142.250.204.14: icmp_req=2 ttl=117 time=218 ms
^C
--- 142.250.204.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 199.439/208.882/218.325/9.443 ms
/ #

This is just after rebooting VOXL (I replaced some network addresses with dashes). Notice that the initial ping succeeds (using DNS), then I shut down wwan0, then second ping (using dns) fails, but when I use direct IP addess it succeeds.
It seems like the DNS server configuration information in linux gets wonked after bringing wwan0 down.
Any thoughts?