Trouble with EM7565 startup...

Ed Sutter · 28 Sept 2022, 21:21

@tom Cool! Just tried it with the EM7565...

ifconfig wwan0
wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.234.85.224  P-t-P:10.234.85.224  Mask:255.255.255.192
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:604 (604.0 B)  TX bytes:656 (656.0 B)

The modem came right up!
Thanks

tom · 28 Sept 2022, 21:21

@Ed-Sutter Beautiful! Glad to hear that

Ed Sutter · 29 Sept 2022, 16:06

@tom Do you guys have any plans to support the EM7565 with an alternative adapter?

tom · 29 Sept 2022, 16:59

@Ed-Sutter Not that I know of at the moment. Is there a specific reason why you're choosing to go with this card as opposed to others? I know you mentioned earlier that it supports certain bands you need.

Ed Sutter · 29 Sept 2022, 19:27

@tom Yea, that's it, we need a band that is not supported by any of the modems that fit your add-on board.
Anyway, now that I've had some time to work with it (the EM7565), things aren't as good as I initially thought...
I've been testing this in various ways and just getting more confused... Initially I was using voxl-streamer to push video over LTE to VLC on another machine. It starts up just fine (video is clean), but the video just stops after some random time (usually less than an hour). So I thought -maybe- my LTE service (I'm using a SIM card on my own AT&T account that is supposed to be used with a flip phone) was doing some throttling. I reduced the load to just open a TCP connection and send small messages (over LTE) every second. Same thing happens. Then I tried running the same thing but now just using WiFi while the wwan0 is up and alive... Same thing even on WiFi! Once it gets in this state, I can't ping out at all (on either interface), but all interfaces are up and I am able to ssh into VOXL. If I remove the EM7565, things are fine on WiFi.

Does this make any sense at all?
Can you think of anything I can do to diagnose this?
I have a small (cheap) USB power monitor in line with the modem and it never exceeds 100mA on the display.

tom · 29 Sept 2022, 19:53

@Ed-Sutter Does it work if you only have one of the network interfaces up at once? For example, only having wwan0 up or only having wlan0 up? I'm wondering if some weird routing issue happens

Ed Sutter · 29 Sept 2022, 19:55

@tom Good thought! I'll try turning down wlan0 while using wwan0...

tom · 29 Sept 2022, 19:57

@Ed-Sutter And to be safe if you're in station mode I would set your ssid and pass to dummy values, i.e voxl-wifi station tmp tmp because I've seen in the past where wlan0 will try to bring itself back up after you manually force it down.

Ed Sutter · 30 Sept 2022, 12:26

@tom Ok, I tried what you said (good idea by the way)... I turned down the wlan0 link and did a very minimial TCP/IP data transfer loop (about 25 characters a second) and after about 15 minutes things just got wonky. I terminated the test, and again, I wasn't able to ping out.
So this morning I was going to reverse the interfaces (enable wlan0 and turn down wwan0) and repeat, but didn't get that far cause I noticed this...

$: adb shell
/ # ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:A2:AB:4C:25
          inet addr:----------  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: ----------------/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:216 (216.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr 18:47:3D:B1:BE:0F
          inet addr:192.168.1.230  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: ----------------/64 Scope:Link
          inet6 addr: ----------------------------/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:221 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3000
          RX bytes:45620 (44.5 KiB)  TX bytes:4536 (4.4 KiB)

wwan0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:--.--.--.--  P-t-P:--.--.--.--  Mask:255.255.255.192
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:22 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2350 (2.2 KiB)  TX bytes:2164 (2.1 KiB)

/ # ping google.com
PING google.com (142.251.40.238) 56(84) bytes of data.
64 bytes from lga34s39-in-f14.1e100.net (142.251.40.238): icmp_req=1 ttl=55 time=38.8 ms
64 bytes from lga34s39-in-f14.1e100.net (142.251.40.238): icmp_req=2 ttl=55 time=43.0 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 38.858/40.949/43.041/2.101 ms
/ # ip link set dev wwan0 down
/ # ping google.com
ping: unknown host google.com
/ # ping 142.250.204.14
PING 142.250.204.14 (142.250.204.14) 56(84) bytes of data.
64 bytes from 142.250.204.14: icmp_req=1 ttl=117 time=199 ms
64 bytes from 142.250.204.14: icmp_req=2 ttl=117 time=218 ms
^C
--- 142.250.204.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 199.439/208.882/218.325/9.443 ms
/ #

This is just after rebooting VOXL (I replaced some network addresses with dashes). Notice that the initial ping succeeds (using DNS), then I shut down wwan0, then second ping (using dns) fails, but when I use direct IP addess it succeeds.
It seems like the DNS server configuration information in linux gets wonked after bringing wwan0 down.
Any thoughts?

Ed Sutter · 30 Sept 2022, 17:18

@tom One thing I just noticed...
If I boot up without an LTE modem, then /etc/resolv.conf is:

# Generated by dhcpcd from wlan0
# /etc/resolv.conf.head can replace this line
domain myfiosgateway.com
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line

That (above) makes sense. That's my ISP (Fios) and wifi subnet in my home.
If I boot up with the LTE modem, then /etc/resolve.conf is:

nameserver 172.26.38.1

Looks like something in the modem startup overwrites /etc/resolv.conf. The script "enable-dns.sh" is called by voxl-modem-start.sh, but it only writes to /etc/resolve.conf if it doesn't already exist (plus it writes 8.8.8.8, which apparently is Google dns).
Any idea how this is being written?

Ed Sutter · 30 Sept 2022, 19:13

@tom I'm starting a new thread on this...

Ed Sutter · 4 Oct 2022, 18:08

@Eric-Katzfey Somehow this slipped through the cracks... I just looked in that file and I don't see the VID/PID of the EM7565 listed. So a couple of questions:

What does this mean?
Do you know what is involved in getting it listed?

I have the latest driver files from Sierra Wireless, and they do have what I need. Is this a module that I can update on VOXL?
Ed

Ed Sutter · 5 Oct 2022, 12:29

@Eric-Katzfey @tom Digging a bit deeper into this, I see that the voxl-build procedure does patch the qmi_wwan.c file, and after the patch, the VID/PID for EM7565 is in there. On one hand this is good news because it implies that the kernel does have support for the modem I'm trying to use; however, its not-so-good news for me because running with that modem has not reliable on VOXL.
Aside from verifying that I am actually running the same version on my VOXL-Flight-Deck as is being built in voxl-build (see below), I'm at a loss regarding other things to do to debug this. Any thoughts on this?
Following is the output of voxl-version on my board:

/ # voxl-version
cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory
--------------------------------------------------------------------------------
system-image:    ModalAI 3.3.0 BUILDER: ekatzfey BUILD_TIME: 2021-06-06_19:28
kernel:          #1 SMP PREEMPT Sun Jun 6 19:41:01 UTC 2021 3.18.71-perf
factory-bundle:  1.0.1 (Yocto installation)
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.5.0
Depends: libjpeg_turbo (>= 9.0.4), libmodal_exposure (>= 0.0.2), libmodal_json (>= 0.3.6), libmodal_pipe (>= 2.1.1), lib
rc_math (>= 1.1.5), libvoxl_cutils (>= 0.0.2), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.2), mongoose (>= 7.
3.0), opencv (>= 4.5.2-2), openmp (>= 10.0.2), voxl-camera-calibration (>= 0.1.1), voxl-camera-server (>= 0.8.1), voxl-c
pu-monitor (>= 0.2.0), voxl-dfs-server (>= 0.2.2), voxl-docker-support (>= 1.1.3), voxl-gphoto2 (>= 0.0.5), voxl-imu-ser
ver (>= 0.9.1), voxl-mavlink (>= 0.0.2), voxl-modem (>= 0.12.0), voxl-mpa-tools (>= 0.3.6), voxl-nodes (>= 0.2.0), voxl-
portal (>= 0.1.2), voxl-qvio-server (>= 0.3.4), voxl-streamer (>= 0.2.6), voxl-tag-detector (>= 0.0.2), voxl-tflite (>=
2.2.3), voxl-tflite-server (>= 0.1.5), voxl-utils (>= 0.8.4), voxl-vision-px4 (>= 0.9.5), voxl-vpn (>= 0.0.3)
Status: install user installed
Section: base
Architecture: all
Maintainer: james@modalai.com
MD5Sum: 172dd2541a4b01ec415651b5ded3afab
Size: 1956
Filename: voxl-suite_0.5.0_202111041852.ipk
Description: meta-package for voxl-suite software collection
Installed-Time: 144

--------------------------------------------------------------------------------