ModalAI Forum

I pulled the latest version of the stable code from the voxl-tflite-server master branch on github and built it directly in voxl-cross. It appeared to build just fine. However, when trying to run voxl-streamer with the tflite overlay it appears that the tflite pipe is not being created.

When running

voxl-streamer -c tflite-overlay

the output simply hangs and doesn't display the usual message of:

Stream available at rtsp://127.0.0.1:8900/live

Any input on why this might be happening? Here is the output of building the ipk file:

voxl-cross:~/voxl-tflite-server$ ./clean.sh 
voxl-cross:~/voxl-tflite-server$ ./install_build_deps.sh stable
using stable repository
Downloading http://voxl-packages.modalai.com/stable/Packages.gz.
Updated source 'stable'.
Installing libmodal_pipe (2.0.10) on root.
Downloading http://voxl-packages.modalai.com/stable/libmodal_pipe_2.0.10.ipk.
Configuring libmodal_pipe.

Done installing libmodal_pipe

Installing libmodal_json (0.3.5) on root.
Downloading http://voxl-packages.modalai.com/stable/libmodal_json_0.3.5.ipk.
Configuring libmodal_json.

Done installing libmodal_json

Installing opencv (4.5.2) on root.
Downloading http://voxl-packages.modalai.com/stable/opencv_4.5.2-2.ipk.
Configuring opencv.

done installing opencv

Installing voxl-tflite (0.0.1) on root.
Downloading http://voxl-packages.modalai.com/stable/voxl-tflite_0.0.1.ipk.
Configuring voxl-tflite.
voxl-cross:~/voxl-tflite-server$ ./build.sh 
+ set -e
+ mkdir -p build
+ cd build
+ cmake -DCMAKE_TOOLCHAIN_FILE=/opt/cross_toolchain/aarch64-gnu-4.9.toolchain.cmake -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_NEON=ON '-DCMAKE_CXX_FLAGS= -std=c++11 -march=armv8-a' ../server
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc-4.9
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc-4.9 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++-4.9
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++-4.9 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    ENABLE_NEON


-- Build files have been written to: /home/root/voxl-tflite-server/build
+ make -j 4 TARGET=aarch64 TARGET_TOOLCHAIN_PREFIX=aarch64-linux-gnu-
Scanning dependencies of target voxl-tflite-server
[ 40%] Building CXX object CMakeFiles/voxl-tflite-server.dir/main.cpp.o
[ 40%] Building C object CMakeFiles/voxl-tflite-server.dir/undistort.c.o
[ 60%] Building CXX object CMakeFiles/voxl-tflite-server.dir/models.cpp.o
[ 80%] Building CXX object CMakeFiles/voxl-tflite-server.dir/threads.cpp.o
In file included from /usr/include/modal_pipe_client.h:42:0,
                 from /usr/include/modal_pipe.h:41,
                 from /home/root/voxl-tflite-server/server/threads.h:40,
                 from /home/root/voxl-tflite-server/server/models.cpp:46:
/usr/include/modal_pipe_common.h:63:2: warning: #warning Building for Android [-Wcpp]
 #warning Building for Android
  ^
[100%] Linking CXX executable voxl-tflite-server
[100%] Built target voxl-tflite-server
voxl-cross:~/voxl-tflite-server$ ./make_package.sh 

Package Name:  voxl-tflite-server
version Number:  0.1.1
[100%] Built target voxl-tflite-server
Install the project...
-- Install configuration: "RELEASE"
-- Installing: ../ipk/data/usr/bin/voxl-tflite-server
/home/root/voxl-tflite-server
ar: creating voxl-tflite-server_0.1.1.ipk

DONE

I am currently modifying the voxl-tflite-server to send custom metadata about the detection, confidence, and bounding box coordinates across a custom-made pipe called tflite_metadata. I have been using voxl_mpa_to_ros to view the data from certain sensors in ROS. I am able to customize what data I view from existing sensors by modifying the voxl_mpa_to_ros.launch script as follows:

yocto:~$ cat custom_voxl_mpa_to_ros.launch 
<?xml version="1.0"?>

<!--
/****************************************************************************
 * Copyright 2021 Modal AI
 * Complete MPA launch file exposes all node params as arguments
 ****************************************************************************/
-->


<launch>

    <!-- This parameter will remove tof points with lower confidence than the given value
         Confidence values range from 0-255 -->
    <arg name="tof_cutoff"     default="100"             />

    <!-- TOF point cloud data is published w.r.t the TOF ros tf frame -->
    <arg name="tof_frameid"    default="tof"             />

    <!-- sample interface variables, specify whether to publish, and the pipe to look for -->
    <arg name="image0_publish" default="false"                  />
    <arg name="image0_pipe"    default="tracking"              />

    <arg name="image1_publish" default="false"                  />
    <arg name="image1_pipe"    default="hires_preview"         />

    <arg name="image2_publish" default="true"                  />
    <arg name="image2_pipe"    default="qvio_overlay"          />

    <arg name="image3_publish" default="true"                  />
    <arg name="image3_pipe"    default="dfs_disparity"         />

    <arg name="image4_publish" default="true"                  />
    <arg name="image4_pipe"    default="tflite"                />

    <arg name="image5_publish" default="false"                 />
    <arg name="image5_pipe"    default=""                      />

    <arg name="image6_publish" default="false"                 />
    <arg name="image6_pipe"    default=""                      />

    <arg name="image7_publish" default="false"                 />
    <arg name="image7_pipe"    default=""                      />

    <arg name="stereo0_publish"   default="false"               />
    <arg name="stereo0_pipe"      default="stereo"             />

    <arg name="stereo1_publish"   default="false"              />
    <arg name="stereo1_pipe"      default=""                   />

    <arg name="tof0_publish"      default="true"               />
    <arg name="tof0_pipe"         default="tof"                />

    <arg name="tof1_publish"      default="false"              />
    <arg name="tof1_pipe"         default=""                   />

    <arg name="imu0_publish"      default="true"               />
    <arg name="imu0_pipe"         default="imu0"               />

    <arg name="imu1_publish"      default="true"               />
    <arg name="imu1_pipe"         default="imu1"               />

    <arg name="vio0_publish"      default="true"               />
    <arg name="vio0_pipe"         default="qvio"               />

    <arg name="vio1_publish"      default="false"              />
    <arg name="vio1_pipe"         default=""                   />

    <arg name="PC0_publish"       default="true"               />
    <arg name="PC0_pipe"          default="dfs_point_cloud"    />

    <arg name="PC1_publish"       default="true"              />
    <arg name="PC1_pipe"          default="tflite_metadata"                   />

    <arg name="PC2_publish"       default="false"              />
    <arg name="PC2_pipe"          default=""                   />

    <arg name="PC3_publish"       default="false"              />
    <arg name="PC3_pipe"          default=""                   />

    <arg name="namespace" default="mpa" />
    <!-- start mpa node with all arguments -->
    <group ns="$(arg namespace)">
        <node name="voxl_mpa_to_ros_node" type="voxl_mpa_to_ros_node" pkg="voxl_mpa_to_ros" output="screen">

            <param name="tof_cutoff" value="$(arg tof_cutoff)" />
            <param name="tof_frameid" value="$(arg tof_frameid)" />

            <param name="image0_publish"    value="$(arg image0_publish)"    />
            <param name="image0_pipe"       value="$(arg image0_pipe)"       />

            <param name="image1_publish"    value="$(arg image1_publish)"    />
            <param name="image1_pipe"       value="$(arg image1_pipe)"       />

            <param name="image2_publish"    value="$(arg image2_publish)"    />
            <param name="image2_pipe"       value="$(arg image2_pipe)"       />

            <param name="image3_publish"    value="$(arg image3_publish)"    />
            <param name="image3_pipe"       value="$(arg image3_pipe)"       />

            <param name="image4_publish"    value="$(arg image4_publish)"    />
            <param name="image4_pipe"       value="$(arg image4_pipe)"       />

            <param name="image5_publish"    value="$(arg image5_publish)"    />
            <param name="image5_pipe"       value="$(arg image5_pipe)"       />

            <param name="image6_publish"    value="$(arg image6_publish)"    />
            <param name="image6_pipe"       value="$(arg image6_pipe)"       />

            <param name="image7_publish"    value="$(arg image7_publish)"    />
            <param name="image7_pipe"       value="$(arg image7_pipe)"       />

            <param name="stereo0_publish"   value="$(arg stereo0_publish)"   />
            <param name="stereo0_pipe"      value="$(arg stereo0_pipe)"      />

            <param name="stereo1_publish"   value="$(arg stereo1_publish)"   />
            <param name="stereo1_pipe"      value="$(arg stereo1_pipe)"      />

            <param name="tof0_publish"       value="$(arg tof0_publish)"     />
            <param name="tof0_pipe"          value="$(arg tof0_pipe)"        />

            <param name="tof1_publish"       value="$(arg tof1_publish)"     />
            <param name="tof1_pipe"          value="$(arg tof1_pipe)"        />

            <param name="imu0_publish"      value="$(arg imu0_publish)"      />
            <param name="imu0_pipe"         value="$(arg imu0_pipe)"         />

            <param name="imu1_publish"      value="$(arg imu1_publish)"      />
            <param name="imu1_pipe"         value="$(arg imu1_pipe)"         />

            <param name="vio0_publish"      value="$(arg vio0_publish)"      />
            <param name="vio0_pipe"         value="$(arg vio0_pipe)"         />

            <param name="vio1_publish"      value="$(arg vio1_publish)"      />
            <param name="vio1_pipe"         value="$(arg vio1_pipe)"         />

            <param name="PC0_publish"      value="$(arg PC0_publish)"        />
            <param name="PC0_pipe"         value="$(arg PC0_pipe)"           />

            <param name="PC1_publish"      value="$(arg PC1_publish)"        />
            <param name="PC1_pipe"         value="$(arg PC1_pipe)"           />

            <param name="PC2_publish"      value="$(arg PC2_publish)"        />
            <param name="PC2_pipe"         value="$(arg PC2_pipe)"           />

            <param name="PC3_publish"      value="$(arg PC3_publish)"        />
            <param name="PC3_pipe"         value="$(arg PC3_pipe)"           />

        </node>

        <group ns="$(arg image0_pipe)/image_raw" >
          <rosparam param="disable_pub_plugins">
            - 'image_transport/compressedDepth'
            - 'image_transport/compressed'
          </rosparam>
        </group>
    </group>

</launch>

I realized after running the above script and running:

rostopic info tflite_metadata

that I had falsely assumed that the PC0_pipe name was an empty slot for custom pipes and that it was actually used for point cloud data.

I would like to be able to simply send JSON strings across custom pipes that will convert to rostopics with the mpa_to_ros service which is currently not possible since only the interfaces found here are supported.

Would it be possible for ModalAI to add a feature to the voxl_mpa_to_ros code to allow for a custom pipe with JSON string data to be sent? It would make sense that the voxl_mpa_to_ros would be able to support converting any JSON string data sent across pipes in the libmodal pipe architecture with functions like int pipe_server_write(int ch, const void* data, int bytes); to be viewed in ROS.

This was all working up until the time libmodal_pipe was updated which seems to be sept 2. How would I use an older version of libmodal_pipe?

Also, here are the available mpa pipes after installing the server.

@Matt-Turi Great thanks! I'll give it a shot now!

Prior to the voxl-tflite-server update to 0.1.3, I was able to stream both the hires and the tflite-overlay using voxl-streamer. Now I am only able to do one or the others. Any suggestions on how to fix this?