voxl-tflite-server not working on latest SW



  • I am starting my VOXL with the latest code.

    From here, I download and extract voxl_platform_3-3-0-0.4.6-b.tar.gz. I follow these steps to flash the system image (and elect to wipe the /data partition).

    When that completes, I connect to wifi and install the voxl-suite.

    When that completes, I run voxl-configure-mpa --wizard to configure the services. Among other things, I select hires + tracking for the voxl-camera-server and enable the voxl-tflite-server.

    When that completes, I install voxl-portal (opkg install voxl-portal), enable that service (systemctl enable voxl-portal), and set the VOXL into softap mode. I now hard reboot the VOXL.

    Once it reboots, I can see the hires, tracking, and qvio-overlay images in the voxl-portal on my laptop. I cannot see the tflite-overlay.

    voxl-inspect-services shows that it is enabled, but not running. I run journalctl -u voxl-tflite-server.service and see the following messages:

    -- Logs begin at Thu 1970-01-01 00:11:17 UTC, end at Thu 1970-01-01 00:13:33 UTC. --
    Jan 01 00:11:20 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
    Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:20 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
    Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
    Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
    Jan 01 00:11:21 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:21 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
    Jan 01 00:11:22 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:11:22 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Main process exited, code=exited, status=127/n/a[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'exit-code'.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart.
    Jan 01 00:11:22 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Start request repeated too quickly.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;31mFailed to start voxl-tflite-server.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Unit entered failed state.[[0m
    Jan 01 00:11:22 apq8096 systemd[1]: [[1;39mvoxl-tflite-server.service: Failed with result 'start-limit'.[[0m
    Jan 01 00:11:22 apq8096 bash[2146]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
    pen shared object file: No such file or directory
    Jan 01 00:11:22 apq8096 bash[2351]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
    pen shared object file: No such file or directory
    Jan 01 00:11:22 apq8096 bash[2389]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
    pen shared object file: No such file or directory
    Jan 01 00:11:22 apq8096 bash[2583]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
    pen shared object file: No such file or directory
    Jan 01 00:11:22 apq8096 bash[2683]: /usr/bin/voxl-tflite-server: error while loading shared libraries: libEGL_adreno.so: cannot o
    pen shared object file: No such file or directory
    

    I found that libEGL_adreno.so exists in the filesystem:

    yocto:~$ ls -al /share/qualcomm-proprietary/usr/lib64/libEGL_adreno.so   
    -rw-r--r--. 1 root root 119624 May 20  2021 /share/qualcomm-proprietary/usr/lib64/libEGL_adreno.so
    

    I tried adding that path to LD_LIBRARY_PATH and restarting the voxl-tflite-service, but it results in the same error.

    Can you please help me know what I need to do to get this working?


  • Dev Team

    Hi jared,

    This is a strange bug, as voxl-tflite-server is expecting those libraries to be under /usr/lib64/. I am looking into this now, but a quick fix would be manually copying those files into the lib64/ directory using this command:

    cp -r /share/qualcomm-proprietary/usr/lib64/* /usr/lib64/
    


  • @Matt-Turi, that got me a step further!

    After running the command you specified, I ran systemctl restart voxl-tflite-server. Now the service runs (without dying), but no data comes out.

    This is the output of journalctl -u voxl-tflite-server.service (immediately after I restarted the service):

    Jan 01 00:31:11 apq8096 systemd[1]: Stopped voxl-tflite-server.
    Jan 01 00:31:11 apq8096 systemd[1]: Started voxl-tflite-server.
    Jan 01 00:31:11 apq8096 bash[3578]: Warning: please export TSAN_OPTIONS='ignore_noninstrumented_modules=1' to avoid false positive reports from the OpenMP runtime.!
    Jan 01 00:31:11 apq8096 bash[3578]: Creating new config file: /etc/modalai/voxl-tflite-server.conf
    Jan 01 00:31:11 apq8096 bash[3578]: ------VOXL TFLite Server------
    Jan 01 00:31:11 apq8096 bash[3578]: INFO: Created TensorFlow Lite delegate for GPU.
    Jan 01 00:31:11 apq8096 bash[3578]: GPU acceleration is SUPPORTED on this platform
    Jan 01 00:31:11 apq8096 bash[3578]: ERROR: Next operations are not supported by GPU delegate:
    Jan 01 00:31:11 apq8096 bash[3578]: CUSTOM TFLite_Detection_PostProcess: Operation is not supported.
    Jan 01 00:31:11 apq8096 bash[3578]: First 114 operations will run on the GPU, and the remaining 1 on the CPU.
    Jan 01 00:31:14 apq8096 bash[3578]: INFO: Initialized OpenCL-based API.
    

    If I reboot, I get the same (new) behavior.

    If I run voxl-tflite-server -d, I get the following output (and then it hangs):

    yocto:/$ voxl-tflite-server -d
    Enabling debug mode
    exising instance of voxl-tflite-server found, attempting to stop it
    =================================================================
    skip_n_frames:                    5
    =================================================================
    =================================================================
    model:                            /usr/bin/dnn/ssdlite_mobilenet_v2_coco.tflite
    =================================================================
    =================================================================
    input_pipe:                       /run/mpa/hires_preview/
    =================================================================
    
    ------VOXL TFLite Server------
    
    Loaded model /usr/bin/dnn/ssdlite_mobilenet_v2_coco.tflite
    Resolved reporter
    INFO: Created TensorFlow Lite delegate for GPU.
    GPU acceleration is SUPPORTED on this platform
    ERROR: Next operations are not supported by GPU delegate:
    CUSTOM TFLite_Detection_PostProcess: Operation is not supported.
    First 114 operations will run on the GPU, and the remaining 1 on the CPU.
    INFO: Initialized OpenCL-based API.
    Applied delegate 
    
    ------Setting TFLiteThread to ready!! W: 300 H: 300 C:3
    

    That command shows that it is looking for this input_pipe: /run/mpa/hires_preview/. I looked in that directory and that pipe does not exist (although /run/mpa/hires/ does). I edited /etc/modalai/voxl-tflite-server.conf, changed the input_pipe to /run/mpa/hires/, and restarted the voxl-tflite-server. That didn't work.

    I presume voxl-tflite-server is not working because there is no /run/mpa/hires_preview/. Is that true?

    If so, any idea why there is no /run/mpa/hires_preview/?


  • Dev Team

    This post is deleted!


  • I ran those commands:

    yocto:/$ opkg update
    Downloading http://voxl-packages.modalai.com/stable/Packages.gz.
    Updated source 'stable'.
    yocto:/$ opkg upgrade
    Not selecting librc_math 1.1.2 as installing it would break existing dependencies.
    Not selecting opencv 4.5.1 as installing it would break existing dependencies.
    Not selecting libmodal_pipe 1.2.2 as installing it would break existing dependencies.
    Upgrading voxl-utils from 0.8.1 to 0.8.2 on root.
    Downloading http://voxl-packages.modalai.com/stable/voxl-utils_0.8.2.ipk.
    Configuring voxl-utils.
    
    done installing voxl-utils
    

    After running this, I hard rebooted my VOXL. Same behavior as before.

    In regards to opkg upgrade, I'm not sure why it wants libmodal_pipe 1.2.2 since there is a 2.1.1 version. Here's the result of opkg info libmodal_pipe. It looks like there are two versions installed?

    yocto:/$ opkg info libmodal_pipe
    Package: libmodal_pipe
    Version: 2.1.1
    Depends: libmodal_json
    Conflicts: libvoxl_pipe, modal_pipe_interfaces
    Status: install ok installed
    Section: base
    Architecture: all
    Maintainer: james@modalai.com
    MD5Sum: 8b7f79e8ef7db663361bc2ce98ae6197
    Size: 110208
    Filename: libmodal_pipe_2.1.1.ipk
    Description: Interface for services to communicate over posix named pipes
    Installed-Time: 1635461181
    
    Package: libmodal_pipe
    Version: 1.2.2
    Status: unknown ok not-installed
    Section: base
    Architecture: armv7a
    Maintainer: james@modalai.com
    MD5Sum: ebfe02800708c1813be3bf313aaa23c0
    Size: 27426
    Filename: libmodal_pipe_1.2.2.ipk
    Description: Interface for services to communicate over posix named pipes
    

  • Dev Team

    Looks like it was a bug with some of our recent package updates. The latest voxl-tflite-server (v0.1.6) and voxl-utils (v0.8.2) will fix these issues, and can be obtained from the opkg dev or stable repo. Once installed, you can edit the config file and choose either "hires" (not hires_preview) or "tracking", but the default will be hires.



  • I already had voxl-utils v0.8.2 installed (see earlier post), so I did not update that.

    I downloaded voxl-tflite-server v0.1.6 from http://voxl-packages.modalai.com/dev/ and adb push'ed it to my voxl. I then installed it (opkg install ipk/voxl-tflite-server_0.1.6_202110292056.ipk) and rebooted my voxl.

    I saw the same behavior as before until I modified /etc/modalai/voxl-tflite-server.conf. Once I set

            "input_pipe":   "/run/mpa/hires/"
    

    and restarted the voxl-tflite-server, it began to work!

    Thanks, @Matt-Turi, for helping me resolve this issue!


Log in to reply