voxl-tflite-server not working on latest SW

[jaredjohansen]

I am starting my VOXL with the latest code.

From here, I download and extract voxl_platform_3-3-0-0.4.6-b.tar.gz. I follow these steps to flash the system image (and elect to wipe the /data partition).

When that completes, I connect to wifi and install the voxl-suite.

When that completes, I run voxl-configure-mpa --wizard to configure the services. Among other things, I select hires + tracking for the voxl-camera-server and enable the voxl-tflite-server.

When that completes, I install voxl-portal (opkg install voxl-portal), enable that service (systemctl enable voxl-portal), and set the VOXL into softap mode. I now hard reboot the VOXL.

Once it reboots, I can see the hires, tracking, and qvio-overlay images in the voxl-portal on my laptop. I cannot see the tflite-overlay.

voxl-inspect-services shows that it is enabled, but not running. I run journalctl -u voxl-tflite-server.service and see the following messages:

-- Logs begin at Thu 1970-01-01 00:11:17 UTC, end at Thu 1970-01-01 00:13:33 UTC. --
Jan 01 00:11:20 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
Jan 01 00:11:22 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:11:22 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
Jan 01 00:11:22 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Start request repeated too quickly.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: [[1;31mFailed to start voxl-tflite-server.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'start-limit'.[[0m
Jan 01 00:11:22 apq8096 bash[2146]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
pen shared object file: No such file or directory
Jan 01 00:11:22 apq8096 bash[2351]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
pen shared object file: No such file or directory
Jan 01 00:11:22 apq8096 bash[2389]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
pen shared object file: No such file or directory
Jan 01 00:11:22 apq8096 bash[2583]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
pen shared object file: No such file or directory
Jan 01 00:11:22 apq8096 bash[2683]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
pen shared object file: No such file or directory

I found that libEGL_adreno.so exists in the filesystem:

yocto:~$ ls -al /share/qualcomm-proprietary/usr/lib64/libEGL_adreno.so   
-rw-r--r--. 1 root root 119624 May 20  2021 /share/qualcomm-proprietary/usr/lib64/libEGL_adreno.so

I tried adding that path to LD_LIBRARY_PATH and restarting the voxl-tflite-service, but it results in the same error.

Can you please help me know what I need to do to get this working?

[A Former User]

Hi jared,

This is a strange bug, as voxl-tflite-server is expecting those libraries to be under /usr/lib64/. I am looking into this now, but a quick fix would be manually copying those files into the lib64/ directory using this command:

cp -r /share/qualcomm-proprietary/usr/lib64/* /usr/lib64/

[jaredjohansen]

@Matt-Turi, that got me a step further!

After running the command you specified, I ran systemctl restart voxl-tflite-server. Now the service runs (without dying), but no data comes out.

This is the output of journalctl -u voxl-tflite-server.service (immediately after I restarted the service):

Jan 01 00:31:11 apq8096 systemd[1]: Stopped voxl-tflite-server.
Jan 01 00:31:11 apq8096 systemd[1]: Started voxl-tflite-server.
Jan 01 00:31:11 apq8096 bash[3578]: Warning: please export TSAN_OPTIONS='ignore_noninstrumented_modules=1' to avoid false positive reports from the OpenMP runtime.!
Jan 01 00:31:11 apq8096 bash[3578]: Creating new config file: /etc/modalai/voxl-tflite-server.conf
Jan 01 00:31:11 apq8096 bash[3578]: ------VOXL TFLite Server------
Jan 01 00:31:11 apq8096 bash[3578]: INFO: Created TensorFlow Lite delegate for GPU.
Jan 01 00:31:11 apq8096 bash[3578]: GPU acceleration is SUPPORTED on this platform
Jan 01 00:31:11 apq8096 bash[3578]: ERROR: Next operations are not supported by GPU delegate:
Jan 01 00:31:11 apq8096 bash[3578]: CUSTOM TFLite_Detection_PostProcess: Operation is not supported.
Jan 01 00:31:11 apq8096 bash[3578]: First 114 operations will run on the GPU, and the remaining 1 on the CPU.
Jan 01 00:31:14 apq8096 bash[3578]: INFO: Initialized OpenCL-based API.

If I reboot, I get the same (new) behavior.

If I run voxl-tflite-server -d, I get the following output (and then it hangs):

yocto:/$ voxl-tflite-server -d
Enabling debug mode
exising instance of voxl-tflite-server found, attempting to stop it
=================================================================
skip_n_frames:                    5
=================================================================
=================================================================
model:                            /usr/bin/dnn/ssdlite_mobilenet_v2_coco.tflite
=================================================================
=================================================================
input_pipe:                       /run/mpa/hires_preview/
=================================================================

------VOXL TFLite Server------

Loaded model /usr/bin/dnn/ssdlite_mobilenet_v2_coco.tflite
Resolved reporter
INFO: Created TensorFlow Lite delegate for GPU.
GPU acceleration is SUPPORTED on this platform
ERROR: Next operations are not supported by GPU delegate:
CUSTOM TFLite_Detection_PostProcess: Operation is not supported.
First 114 operations will run on the GPU, and the remaining 1 on the CPU.
INFO: Initialized OpenCL-based API.
Applied delegate 

------Setting TFLiteThread to ready!! W: 300 H: 300 C:3

That command shows that it is looking for this input_pipe: /run/mpa/hires_preview/. I looked in that directory and that pipe does not exist (although /run/mpa/hires/ does). I edited /etc/modalai/voxl-tflite-server.conf, changed the input_pipe to /run/mpa/hires/, and restarted the voxl-tflite-server. That didn't work.

I presume voxl-tflite-server is not working because there is no /run/mpa/hires_preview/. Is that true?

If so, any idea why there is no /run/mpa/hires_preview/?

[A Former User]

This post is deleted!

[jaredjohansen]

I ran those commands:

yocto:/$ opkg update
Downloading http://voxl-packages.modalai.com/stable/Packages.gz.
Updated source 'stable'.
yocto:/$ opkg upgrade
Not selecting librc_math 1.1.2 as installing it would break existing dependencies.
Not selecting opencv 4.5.1 as installing it would break existing dependencies.
Not selecting libmodal_pipe 1.2.2 as installing it would break existing dependencies.
Upgrading voxl-utils from 0.8.1 to 0.8.2 on root.
Downloading http://voxl-packages.modalai.com/stable/voxl-utils_0.8.2.ipk.
Configuring voxl-utils.

done installing voxl-utils

After running this, I hard rebooted my VOXL. Same behavior as before.

In regards to opkg upgrade, I'm not sure why it wants libmodal_pipe 1.2.2 since there is a 2.1.1 version. Here's the result of opkg info libmodal_pipe. It looks like there are two versions installed?

yocto:/$ opkg info libmodal_pipe
Package: libmodal_pipe
Version: 2.1.1
Depends: libmodal_json
Conflicts: libvoxl_pipe, modal_pipe_interfaces
Status: install ok installed
Section: base
Architecture: all
Maintainer: james@modalai.com
MD5Sum: 8b7f79e8ef7db663361bc2ce98ae6197
Size: 110208
Filename: libmodal_pipe_2.1.1.ipk
Description: Interface for services to communicate over posix named pipes
Installed-Time: 1635461181

Package: libmodal_pipe
Version: 1.2.2
Status: unknown ok not-installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: ebfe02800708c1813be3bf313aaa23c0
Size: 27426
Filename: libmodal_pipe_1.2.2.ipk
Description: Interface for services to communicate over posix named pipes

[A Former User]

Looks like it was a bug with some of our recent package updates. The latest voxl-tflite-server (v0.1.6) and voxl-utils (v0.8.2) will fix these issues, and can be obtained from the opkg dev or stable repo. Once installed, you can edit the config file and choose either "hires" (not hires_preview) or "tracking", but the default will be hires.

[jaredjohansen]

I already had voxl-utils v0.8.2 installed (see earlier post), so I did not update that.

I downloaded voxl-tflite-server v0.1.6 from http://voxl-packages.modalai.com/dev/ and adb push'ed it to my voxl. I then installed it (opkg install ipk/voxl-tflite-server_0.1.6_202110292056.ipk) and rebooted my voxl.

I saw the same behavior as before until I modified /etc/modalai/voxl-tflite-server.conf. Once I set

        "input_pipe":   "/run/mpa/hires/"

and restarted the voxl-tflite-server, it began to work!

Thanks, @Matt-Turi, for helping me resolve this issue!