LTE Modem "Feather is Disabled"

tom

Just to make sure I understand your setup, are you using a v1 modem or v2 modem?

Another question, did you previously run voxl-configure-modem? If so there will be a service running in the background on boot attempting to connect to the LTE network.

You can check the status of this service by running systemctl status -l voxl-modem

scottesicdrone

Thanks for the quick response! We are using a v1 modem.
We ran voxl-configure-modem previously so that definitely could be the case. Here is the output for systemctl status -l voxl-modem:

yocto:/# systemctl status -l voxl-modem
● voxl-modem.service - Modem Initialization
   Loaded: loaded (/etc/systemd/system/voxl-modem.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 1970-01-01 00:00:08 UTC; 51 years 3 months ago
  Process: 2041 ExecStartPre=/bin/sleep 2 (code=exited, status=0/SUCCESS)
 Main PID: 2824 (voxl-modem-star)
   CGroup: /system.slice/voxl-modem.service
           ├─2824 /bin/bash /usr/bin/voxl-modem-start.sh
           └─2907 /usr/bin/python /usr/bin/voxl-modem --feather_configure

Jan 01 00:00:06 apq8096 systemd[1]: Starting Modem Initialization...
Jan 01 00:00:08 apq8096 systemd[1]: Started Modem Initialization.
Jan 01 00:00:09 apq8096 voxl-modem-start.sh[2824]: Initializing v1 modem...

Should I disable this service and try configuring and connecting?

tom

@scottesicdrone Okay great, if you went through the setup using voxl-configure-modem then you shouldn't need to run anything manually. Can I also see what your voxl-modem config file looks like: /etc/modalai/voxl-modem.conf? As a next step I would try rebooting and monitoring the status of the voxl-modem service with watch systemctl status -l voxl-modem.

scottesicdrone

Here is my voxl-modem.conf file:

{
	"modem_type":	"v1",
	"apn":	"broadband",
	"world":	"false",
	"microhard_ip": "192.168.168.100",
	"modallink":	"false",
	"placeholder": "hello world"
}

I'm watching the service right now and here is where it is at currently

● voxl-modem.service - Modem Initialization
   Loaded: loaded (/etc/systemd/system/voxl-modem.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 1970-01-01 00:01:23 UTC; 51 years 3 months ago
  Process: 2129 ExecStartPre=/bin/sleep 2 (code=exited, status=0/SUCCESS)
 Main PID: 2853 (voxl-modem-star)
   CGroup: /system.slice/voxl-modem.service
           ├─2853 /bin/bash /usr/bin/voxl-modem-start.sh
           └─3364 /usr/bin/python /usr/bin/voxl-modem --connect_to_network

Apr 02 17:13:49 apq8096 voxl-modem-start.sh[2853]: Setting network preference to LTE
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Failed to open device
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Finished setting network preference
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Disabling band 5...
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: configband starting
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Band mask 1 to 32: 0x0000080a 0x0a 0x08 0x00 0x00
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Band mask 33 to 64: 0x00000000 0x00 0x00 0x00 0x00
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Band mask 65 to 96: 0x00000000 0x00 0x00 0x00 0x00
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Error opening /dev/ttyUSB0: No such file or directory
Apr 02 17:13:59 apq8096 voxl-modem-start.sh[2853]: Initalizing cellular connection...

tom

@scottesicdrone Okay let's have you go through the steps manually to see if it's a timing issue.

First, disable the voxl-modem service -> systemctl disable voxl-modem and reboot.

Now, run the following:

voxl-modem --feather_configure -> this should bring up /dev/qcqmi0 and /dev/ttyUSB0, make sure these exist before proceeding

Next, run:

uqmi -P -d /dev/qcqmi0 --set-network-preference lte

then

configband write /dev/ttyUSB0 0x80a

Lastly, run

voxl-modem --conect_to_network

scottesicdrone

I disabled the service and ran voxl-modem --feather_configure and got this:

Configuring GPIO pins
    Setting defaults:
        GPIO_PIN_WWAN_WAKEUP_IN  = 1
        GPIO_PIN_WWAN_POWER_ON_N = 1
        GPIO_PIN_WWAN_RESET_N    = 1
    feather is DISABLED
ENABLE WNC feather module
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
WARNING: feather state transtion failed.
   enable failed, try again 0 retries remaining
ENABLE WNC feather module
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
WARNING: feather state transtion failed.
   enable failed, try again 1 retries remaining
ENABLE WNC feather module
    feather is DISABLED
    feather is DISABLED

And this will keep going if I let it, I'm pretty sure. In the /dev folder, /dev/qcqmi0 exists but /dev/ttyUSB0 doesn't exist.

tom

@scottesicdrone Thanks for the info, I will look into this behavior in order to get a solution for you

scottesicdrone

Thanks, Tom, for the help! I really appreciate it! Is it possible that the serial port and the ports got renamed from /dev/ttyACM0 and /dev/ttyUSB0? I'm not too familiar with how these devices are generated or normal behavior for these devices. We have also noticed that this happened on two different VOXLs with the same v1 modem, so this problem seems to be with the modem and not the VOXL (though this is not something I am 100% certain about).

tom

@scottesicdrone I'm leaning toward it being a hardware issue with your modem. Did anything hardware related change from when it was working and wasn't working? Do you have any other v1 modems you can test with?

scottesicdrone

Nothing physically changed that I can recall, and unfortunately, we don't have extra modems though we ordered more modems. Last potential useful bit of info: when I ran dmesg, I saw errors relating to qmi pop up but I don't know what to make of them:

yocto:/# dmesg | grep "qmi"
[    1.186410] usbcore: registered new interface driver qmi_wwan
[    1.187419] msm_sharedmem: sharedmem_register_qmi: qmi init successful
[    5.741357] sysmon_send_event_no_qmi: Message sending failed -19
[    5.890086] type=1130 audit(7.556:61): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0-s15:c0.c1023 msg='unit=qmi_shutdown_modemd comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    5.903844] sysmon_send_event_no_qmi: Message sending failed -19
[    5.903850] sysmon_send_event_no_qmi: Message sending failed -19
[    5.908564] sysmon-qmi: sysmon_clnt_svc_arrive: Connection established between QMI handle and slpi's SSCTL service
[    5.914578] sysmon_send_event_no_qmi: Message sending failed -19
[    6.039513] sysmon-qmi: sysmon_clnt_svc_arrive: Connection established between QMI handle and adsp's SSCTL service
[    6.523379] sysmon-qmi: sysmon_clnt_svc_arrive: Connection established between QMI handle and modem's SSCTL service
[    6.543970] msm_slim_qmi_send_power_request: QMI request failed 0x1 (Unknown error)
[    7.045169] msm_slim_qmi_send_power_request: QMI request failed 0x1 (Unknown error)
[    7.546548] msm_slim_qmi_send_power_request: QMI request failed 0x1 (Unknown error)
[    8.049225] msm_slim_qmi_send_power_request: QMI request failed 0x1 (Unknown error)

tom

Can you also run voxl-version so I can see what system image / package version you're using.

scottesicdrone

Here is what I get when I run voxl-version:

yocto:/# voxl-version
--------------------------------------------------------------------------------
system-image:    ModalAI 3.1.0 BUILDER: ekatzfey BUILD_TIME: 2020-12-23_19:41
kernel:          #1 SMP PREEMPT Wed Dec 23 19:50:23 UTC 2020 3.18.71-perf
factory-bundle:  1.0.1 (Yocto installation)
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.3.4
Depends: voxl-utils (>= 0.6.0), libmodal_json (>= 0.3.4), libmodal_pipe (>= 1.7.8), librc_math (>= 1.1.4), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.1), opencv (>= 4.5.1), openmp (>= 10.0.1), voxl-camera-server (>= 0.5.6), voxl-dfs-server (>= 0.0.7), voxl-docker-support (>= 1.1.1), voxl-gphoto2 (>= 0.0.5), voxl-hal3-tof-cam-ros (>= 0.0.5), voxl-imu-server (>= 0.7.8), voxl-modem (>= 0.11.0), voxl-mpa-tflite-server (>= 0.0.2), voxl-mpa-tools (>= 0.1.6), voxl-nodes (>= 0.1.3), voxl-qvio-server (>= 0.2.1), voxl-rtsp (>= 1.0.3), voxl-streamer (>= 0.2.1), voxl-vision-px4 (>= 0.8.1), voxl-vpn (>= 0.0.3)
Status: unknown ok not-installed
Section: base
Architecture: all
Maintainer: james@modalai.com
MD5Sum: e1e5bf1a125affe5036099f6ce895bed
Size: 1818
Filename: voxl-suite_0.3.4.ipk
Description: meta-package for voxl-suite stable release

Package: voxl-suite
Version: 0.2.0
Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
Status: install user installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
Size: 870
Filename: voxl-suite_0.2.0.ipk
Description: meta-package to install all of the voxl-suite
Installed-Time: 689

--------------------------------------------------------------------------------

tom

Just to show you what a working modem would look like:

/ # voxl-modem --feather_configure
Configuring GPIO pins
    Setting defaults:
        GPIO_PIN_WWAN_WAKEUP_IN  = 1
        GPIO_PIN_WWAN_POWER_ON_N = 1
        GPIO_PIN_WWAN_RESET_N    = 1
    feather is DISABLED
ENABLE WNC feather module
    feather is DISABLED
    feather is DISABLED
    feather is DISABLED
    feather is ENABLED, device detected: /dev/ttyUSB0

It comes up within a few seconds. Good thought about qmi / dmesg, those errors will go away once the modem is powered up correctly by voxl-modem

scottesicdrone

This makes a lot of sense. I have seen this working before but now it just says feather is DISABLED until it power cycles and repeats. So if I understand correctly, after a few seconds, /dev/ttyUSB0 will be created and then the feather is enabled? Also random question: what is a feather exactly? I haven't heard of this before and I tried to look into it but I'm still confused!

tom

@scottesicdrone Haha okay so "feather" is a remnant from the days that the VOXL predecessor was a Qualcomm product. The nickname for the project at the time was "Eagle", therefore a "feather" was the nickname for an add-on board for the Eagle. So any reference to "feather" in voxl-modem is just talking about the modem add-on board. An example of one of the VOXL's predecessors is currently powering NASA's "Ingenuity" Drone on Mars (https://mars.nasa.gov/technology/helicopter/).

A little ModalAI lore for you that you won't find on Google

scottesicdrone

Love the Lore and good thing to note! Last last thing, here is the link to an instance of me running dmesg if that's of any use.
output here
I'm going to keep trying to figure this out because this is a time-sensitive situation but if we will know for sure if it's a hardware issue once we get another modem to test it on. Thanks for the help thus far! Anything else I should try on here?

tom

@scottesicdrone Thanks! I am going to have another resource take a look at the issue and I will update you with our conclusion. I will message you if anything else is needed.

scottesicdrone

Sounds good, thanks!