I pulled the latest version of the stable code from the voxl-tflite-server master branch on github and built it directly in voxl-cross. It appeared to build just fine. However, when trying to run voxl-streamer with the tflite overlay it appears that the tflite pipe is not being created.

When running

voxl-streamer -c tflite-overlay

the output simply hangs and doesn't display the usual message of:

Stream available at rtsp://127.0.0.1:8900/live

Any input on why this might be happening? Here is the output of building the ipk file:

voxl-cross:~/voxl-tflite-server$ ./clean.sh 
voxl-cross:~/voxl-tflite-server$ ./install_build_deps.sh stable
using stable repository
Downloading http://voxl-packages.modalai.com/stable/Packages.gz.
Updated source 'stable'.
Installing libmodal_pipe (2.0.10) on root.
Downloading http://voxl-packages.modalai.com/stable/libmodal_pipe_2.0.10.ipk.
Configuring libmodal_pipe.

Done installing libmodal_pipe

Installing libmodal_json (0.3.5) on root.
Downloading http://voxl-packages.modalai.com/stable/libmodal_json_0.3.5.ipk.
Configuring libmodal_json.

Done installing libmodal_json

Installing opencv (4.5.2) on root.
Downloading http://voxl-packages.modalai.com/stable/opencv_4.5.2-2.ipk.
Configuring opencv.

done installing opencv

Installing voxl-tflite (0.0.1) on root.
Downloading http://voxl-packages.modalai.com/stable/voxl-tflite_0.0.1.ipk.
Configuring voxl-tflite.
voxl-cross:~/voxl-tflite-server$ ./build.sh 
+ set -e
+ mkdir -p build
+ cd build
+ cmake -DCMAKE_TOOLCHAIN_FILE=/opt/cross_toolchain/aarch64-gnu-4.9.toolchain.cmake -DCMAKE_BUILD_TYPE=RELEASE -DENABLE_NEON=ON '-DCMAKE_CXX_FLAGS= -std=c++11 -march=armv8-a' ../server
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc-4.9
-- Check for working C compiler: /usr/bin/aarch64-linux-gnu-gcc-4.9 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++-4.9
-- Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++-4.9 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    ENABLE_NEON


-- Build files have been written to: /home/root/voxl-tflite-server/build
+ make -j 4 TARGET=aarch64 TARGET_TOOLCHAIN_PREFIX=aarch64-linux-gnu-
Scanning dependencies of target voxl-tflite-server
[ 40%] Building CXX object CMakeFiles/voxl-tflite-server.dir/main.cpp.o
[ 40%] Building C object CMakeFiles/voxl-tflite-server.dir/undistort.c.o
[ 60%] Building CXX object CMakeFiles/voxl-tflite-server.dir/models.cpp.o
[ 80%] Building CXX object CMakeFiles/voxl-tflite-server.dir/threads.cpp.o
In file included from /usr/include/modal_pipe_client.h:42:0,
                 from /usr/include/modal_pipe.h:41,
                 from /home/root/voxl-tflite-server/server/threads.h:40,
                 from /home/root/voxl-tflite-server/server/models.cpp:46:
/usr/include/modal_pipe_common.h:63:2: warning: #warning Building for Android [-Wcpp]
 #warning Building for Android
  ^
[100%] Linking CXX executable voxl-tflite-server
[100%] Built target voxl-tflite-server
voxl-cross:~/voxl-tflite-server$ ./make_package.sh 

Package Name:  voxl-tflite-server
version Number:  0.1.1
[100%] Built target voxl-tflite-server
Install the project...
-- Install configuration: "RELEASE"
-- Installing: ../ipk/data/usr/bin/voxl-tflite-server
/home/root/voxl-tflite-server
ar: creating voxl-tflite-server_0.1.1.ipk

DONE

I am currently modifying the voxl-tflite-server to send custom metadata about the detection, confidence, and bounding box coordinates across a custom-made pipe called tflite_metadata. I have been using voxl_mpa_to_ros to view the data from certain sensors in ROS. I am able to customize what data I view from existing sensors by modifying the voxl_mpa_to_ros.launch script as follows:

yocto:~$ cat custom_voxl_mpa_to_ros.launch 
<?xml version="1.0"?>

<!--
/****************************************************************************
 * Copyright 2021 Modal AI
 * Complete MPA launch file exposes all node params as arguments
 ****************************************************************************/
-->


<launch>

    <!-- This parameter will remove tof points with lower confidence than the given value
         Confidence values range from 0-255 -->
    <arg name="tof_cutoff"     default="100"             />

    <!-- TOF point cloud data is published w.r.t the TOF ros tf frame -->
    <arg name="tof_frameid"    default="tof"             />

    <!-- sample interface variables, specify whether to publish, and the pipe to look for -->
    <arg name="image0_publish" default="false"                  />
    <arg name="image0_pipe"    default="tracking"              />

    <arg name="image1_publish" default="false"                  />
    <arg name="image1_pipe"    default="hires_preview"         />

    <arg name="image2_publish" default="true"                  />
    <arg name="image2_pipe"    default="qvio_overlay"          />

    <arg name="image3_publish" default="true"                  />
    <arg name="image3_pipe"    default="dfs_disparity"         />

    <arg name="image4_publish" default="true"                  />
    <arg name="image4_pipe"    default="tflite"                />

    <arg name="image5_publish" default="false"                 />
    <arg name="image5_pipe"    default=""                      />

    <arg name="image6_publish" default="false"                 />
    <arg name="image6_pipe"    default=""                      />

    <arg name="image7_publish" default="false"                 />
    <arg name="image7_pipe"    default=""                      />

    <arg name="stereo0_publish"   default="false"               />
    <arg name="stereo0_pipe"      default="stereo"             />

    <arg name="stereo1_publish"   default="false"              />
    <arg name="stereo1_pipe"      default=""                   />

    <arg name="tof0_publish"      default="true"               />
    <arg name="tof0_pipe"         default="tof"                />

    <arg name="tof1_publish"      default="false"              />
    <arg name="tof1_pipe"         default=""                   />

    <arg name="imu0_publish"      default="true"               />
    <arg name="imu0_pipe"         default="imu0"               />

    <arg name="imu1_publish"      default="true"               />
    <arg name="imu1_pipe"         default="imu1"               />

    <arg name="vio0_publish"      default="true"               />
    <arg name="vio0_pipe"         default="qvio"               />

    <arg name="vio1_publish"      default="false"              />
    <arg name="vio1_pipe"         default=""                   />

    <arg name="PC0_publish"       default="true"               />
    <arg name="PC0_pipe"          default="dfs_point_cloud"    />

    <arg name="PC1_publish"       default="true"              />
    <arg name="PC1_pipe"          default="tflite_metadata"                   />

    <arg name="PC2_publish"       default="false"              />
    <arg name="PC2_pipe"          default=""                   />

    <arg name="PC3_publish"       default="false"              />
    <arg name="PC3_pipe"          default=""                   />

    <arg name="namespace" default="mpa" />
    <!-- start mpa node with all arguments -->
    <group ns="$(arg namespace)">
        <node name="voxl_mpa_to_ros_node" type="voxl_mpa_to_ros_node" pkg="voxl_mpa_to_ros" output="screen">

            <param name="tof_cutoff" value="$(arg tof_cutoff)" />
            <param name="tof_frameid" value="$(arg tof_frameid)" />

            <param name="image0_publish"    value="$(arg image0_publish)"    />
            <param name="image0_pipe"       value="$(arg image0_pipe)"       />

            <param name="image1_publish"    value="$(arg image1_publish)"    />
            <param name="image1_pipe"       value="$(arg image1_pipe)"       />

            <param name="image2_publish"    value="$(arg image2_publish)"    />
            <param name="image2_pipe"       value="$(arg image2_pipe)"       />

            <param name="image3_publish"    value="$(arg image3_publish)"    />
            <param name="image3_pipe"       value="$(arg image3_pipe)"       />

            <param name="image4_publish"    value="$(arg image4_publish)"    />
            <param name="image4_pipe"       value="$(arg image4_pipe)"       />

            <param name="image5_publish"    value="$(arg image5_publish)"    />
            <param name="image5_pipe"       value="$(arg image5_pipe)"       />

            <param name="image6_publish"    value="$(arg image6_publish)"    />
            <param name="image6_pipe"       value="$(arg image6_pipe)"       />

            <param name="image7_publish"    value="$(arg image7_publish)"    />
            <param name="image7_pipe"       value="$(arg image7_pipe)"       />

            <param name="stereo0_publish"   value="$(arg stereo0_publish)"   />
            <param name="stereo0_pipe"      value="$(arg stereo0_pipe)"      />

            <param name="stereo1_publish"   value="$(arg stereo1_publish)"   />
            <param name="stereo1_pipe"      value="$(arg stereo1_pipe)"      />

            <param name="tof0_publish"       value="$(arg tof0_publish)"     />
            <param name="tof0_pipe"          value="$(arg tof0_pipe)"        />

            <param name="tof1_publish"       value="$(arg tof1_publish)"     />
            <param name="tof1_pipe"          value="$(arg tof1_pipe)"        />

            <param name="imu0_publish"      value="$(arg imu0_publish)"      />
            <param name="imu0_pipe"         value="$(arg imu0_pipe)"         />

            <param name="imu1_publish"      value="$(arg imu1_publish)"      />
            <param name="imu1_pipe"         value="$(arg imu1_pipe)"         />

            <param name="vio0_publish"      value="$(arg vio0_publish)"      />
            <param name="vio0_pipe"         value="$(arg vio0_pipe)"         />

            <param name="vio1_publish"      value="$(arg vio1_publish)"      />
            <param name="vio1_pipe"         value="$(arg vio1_pipe)"         />

            <param name="PC0_publish"      value="$(arg PC0_publish)"        />
            <param name="PC0_pipe"         value="$(arg PC0_pipe)"           />

            <param name="PC1_publish"      value="$(arg PC1_publish)"        />
            <param name="PC1_pipe"         value="$(arg PC1_pipe)"           />

            <param name="PC2_publish"      value="$(arg PC2_publish)"        />
            <param name="PC2_pipe"         value="$(arg PC2_pipe)"           />

            <param name="PC3_publish"      value="$(arg PC3_publish)"        />
            <param name="PC3_pipe"         value="$(arg PC3_pipe)"           />

        </node>

        <group ns="$(arg image0_pipe)/image_raw" >
          <rosparam param="disable_pub_plugins">
            - 'image_transport/compressedDepth'
            - 'image_transport/compressed'
          </rosparam>
        </group>
    </group>

</launch>

I realized after running the above script and running:

rostopic info tflite_metadata

that I had falsely assumed that the PC0_pipe name was an empty slot for custom pipes and that it was actually used for point cloud data.

I would like to be able to simply send JSON strings across custom pipes that will convert to rostopics with the mpa_to_ros service which is currently not possible since only the interfaces found here are supported.

Would it be possible for ModalAI to add a feature to the voxl_mpa_to_ros code to allow for a custom pipe with JSON string data to be sent? It would make sense that the voxl_mpa_to_ros would be able to support converting any JSON string data sent across pipes in the libmodal pipe architecture with functions like int pipe_server_write(int ch, const void* data, int bytes); to be viewed in ROS.

This was all working up until the time libmodal_pipe was updated which seems to be sept 2. How would I use an older version of libmodal_pipe?

Also, here are the available mpa pipes after installing the server.

@Matt-Turi Great thanks! I'll give it a shot now!

Prior to the voxl-tflite-server update to 0.1.3, I was able to stream both the hires and the tflite-overlay using voxl-streamer. Now I am only able to do one or the others. Any suggestions on how to fix this?

@Matt-Turi thanks for letting me know! Does this new version crash less often? Also, will the output pipe work with voxl_mpa_to_ros?

After running:

roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch 

on the VOXL with the latest stable version of voxl-tflite-server running (v0.1.6), I noticed that the tflite pipe would disconnect and reconnect after about 5-10 minutes.

Here is an output of that occuring:

Found new interface: tflite
Found new interface: tracking
Found new interface: cpu_monitor
Found new interface: imu0
Found new interface: imu1
Found new interface: vvpx4_gps_raw_int
Found new interface: vvpx4_mavlink_io
Found new interface: vvpx4_sys_status
Found new interface: vvpx4_vehicle_gps
Found new interface: vvpx4_body_wrt_fixed
Found new interface: vvpx4_body_wrt_local
Found new interface: qvio_extended
Found new interface: vvpx4_shell
Found new interface: qvio
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising

When this happens all of the cameras on voxl-portal are no longer viewable. I can select them but no image shows up. Even if I restart voxl-portal or the voxl-tflite server the cameras don't show up in voxl-portal. The only fix to this is to restart the voxl entirely.

I then tested to see if this happened without voxl-mpa-to-ros running. Unfortunately, the same error still occurs. After about 5-10 minutes the cameras are no longer viewable in voxl-portal with the only fix being to reboot the voxl.

Should I be building the ipk package for my custom version of mpa_to_ros in the voxl-cross container? It appears that voxl-cross doesn't contain /opt/ros/indigo/setup.bash which is sourced by the build script in voxl_mpa_to_ros

Thanks for the update @Alex-Gardner! Can you clarify what you mean when you say the source doesn't need the launch file anymore? What would be the proper way to launch mpa_to_ros without the launch file?

Hello,

I just wanted to follow up on the above post.

Thanks!

I am currently modifying the voxl-tflite-server to send custom metadata about the detection, confidence, and bounding box coordinates across a custom-made pipe called tflite_metadata. I have been using voxl_mpa_to_ros to view the data from certain sensors in ROS. I am able to customize what data I view from existing sensors by modifying the voxl_mpa_to_ros.launch script as follows:

yocto:~$ cat custom_voxl_mpa_to_ros.launch 
<?xml version="1.0"?>

<!--
/****************************************************************************
 * Copyright 2021 Modal AI
 * Complete MPA launch file exposes all node params as arguments
 ****************************************************************************/
-->


<launch>

    <!-- This parameter will remove tof points with lower confidence than the given value
         Confidence values range from 0-255 -->
    <arg name="tof_cutoff"     default="100"             />

    <!-- TOF point cloud data is published w.r.t the TOF ros tf frame -->
    <arg name="tof_frameid"    default="tof"             />

    <!-- sample interface variables, specify whether to publish, and the pipe to look for -->
    <arg name="image0_publish" default="false"                  />
    <arg name="image0_pipe"    default="tracking"              />

    <arg name="image1_publish" default="false"                  />
    <arg name="image1_pipe"    default="hires_preview"         />

    <arg name="image2_publish" default="true"                  />
    <arg name="image2_pipe"    default="qvio_overlay"          />

    <arg name="image3_publish" default="true"                  />
    <arg name="image3_pipe"    default="dfs_disparity"         />

    <arg name="image4_publish" default="true"                  />
    <arg name="image4_pipe"    default="tflite"                />

    <arg name="image5_publish" default="false"                 />
    <arg name="image5_pipe"    default=""                      />

    <arg name="image6_publish" default="false"                 />
    <arg name="image6_pipe"    default=""                      />

    <arg name="image7_publish" default="false"                 />
    <arg name="image7_pipe"    default=""                      />

    <arg name="stereo0_publish"   default="false"               />
    <arg name="stereo0_pipe"      default="stereo"             />

    <arg name="stereo1_publish"   default="false"              />
    <arg name="stereo1_pipe"      default=""                   />

    <arg name="tof0_publish"      default="true"               />
    <arg name="tof0_pipe"         default="tof"                />

    <arg name="tof1_publish"      default="false"              />
    <arg name="tof1_pipe"         default=""                   />

    <arg name="imu0_publish"      default="true"               />
    <arg name="imu0_pipe"         default="imu0"               />

    <arg name="imu1_publish"      default="true"               />
    <arg name="imu1_pipe"         default="imu1"               />

    <arg name="vio0_publish"      default="true"               />
    <arg name="vio0_pipe"         default="qvio"               />

    <arg name="vio1_publish"      default="false"              />
    <arg name="vio1_pipe"         default=""                   />

    <arg name="PC0_publish"       default="true"               />
    <arg name="PC0_pipe"          default="dfs_point_cloud"    />

    <arg name="PC1_publish"       default="true"              />
    <arg name="PC1_pipe"          default="tflite_metadata"                   />

    <arg name="PC2_publish"       default="false"              />
    <arg name="PC2_pipe"          default=""                   />

    <arg name="PC3_publish"       default="false"              />
    <arg name="PC3_pipe"          default=""                   />

    <arg name="namespace" default="mpa" />
    <!-- start mpa node with all arguments -->
    <group ns="$(arg namespace)">
        <node name="voxl_mpa_to_ros_node" type="voxl_mpa_to_ros_node" pkg="voxl_mpa_to_ros" output="screen">

            <param name="tof_cutoff" value="$(arg tof_cutoff)" />
            <param name="tof_frameid" value="$(arg tof_frameid)" />

            <param name="image0_publish"    value="$(arg image0_publish)"    />
            <param name="image0_pipe"       value="$(arg image0_pipe)"       />

            <param name="image1_publish"    value="$(arg image1_publish)"    />
            <param name="image1_pipe"       value="$(arg image1_pipe)"       />

            <param name="image2_publish"    value="$(arg image2_publish)"    />
            <param name="image2_pipe"       value="$(arg image2_pipe)"       />

            <param name="image3_publish"    value="$(arg image3_publish)"    />
            <param name="image3_pipe"       value="$(arg image3_pipe)"       />

            <param name="image4_publish"    value="$(arg image4_publish)"    />
            <param name="image4_pipe"       value="$(arg image4_pipe)"       />

            <param name="image5_publish"    value="$(arg image5_publish)"    />
            <param name="image5_pipe"       value="$(arg image5_pipe)"       />

            <param name="image6_publish"    value="$(arg image6_publish)"    />
            <param name="image6_pipe"       value="$(arg image6_pipe)"       />

            <param name="image7_publish"    value="$(arg image7_publish)"    />
            <param name="image7_pipe"       value="$(arg image7_pipe)"       />

            <param name="stereo0_publish"   value="$(arg stereo0_publish)"   />
            <param name="stereo0_pipe"      value="$(arg stereo0_pipe)"      />

            <param name="stereo1_publish"   value="$(arg stereo1_publish)"   />
            <param name="stereo1_pipe"      value="$(arg stereo1_pipe)"      />

            <param name="tof0_publish"       value="$(arg tof0_publish)"     />
            <param name="tof0_pipe"          value="$(arg tof0_pipe)"        />

            <param name="tof1_publish"       value="$(arg tof1_publish)"     />
            <param name="tof1_pipe"          value="$(arg tof1_pipe)"        />

            <param name="imu0_publish"      value="$(arg imu0_publish)"      />
            <param name="imu0_pipe"         value="$(arg imu0_pipe)"         />

            <param name="imu1_publish"      value="$(arg imu1_publish)"      />
            <param name="imu1_pipe"         value="$(arg imu1_pipe)"         />

            <param name="vio0_publish"      value="$(arg vio0_publish)"      />
            <param name="vio0_pipe"         value="$(arg vio0_pipe)"         />

            <param name="vio1_publish"      value="$(arg vio1_publish)"      />
            <param name="vio1_pipe"         value="$(arg vio1_pipe)"         />

            <param name="PC0_publish"      value="$(arg PC0_publish)"        />
            <param name="PC0_pipe"         value="$(arg PC0_pipe)"           />

            <param name="PC1_publish"      value="$(arg PC1_publish)"        />
            <param name="PC1_pipe"         value="$(arg PC1_pipe)"           />

            <param name="PC2_publish"      value="$(arg PC2_publish)"        />
            <param name="PC2_pipe"         value="$(arg PC2_pipe)"           />

            <param name="PC3_publish"      value="$(arg PC3_publish)"        />
            <param name="PC3_pipe"         value="$(arg PC3_pipe)"           />

        </node>

        <group ns="$(arg image0_pipe)/image_raw" >
          <rosparam param="disable_pub_plugins">
            - 'image_transport/compressedDepth'
            - 'image_transport/compressed'
          </rosparam>
        </group>
    </group>

</launch>

I realized after running the above script and running:

rostopic info tflite_metadata

that I had falsely assumed that the PC0_pipe name was an empty slot for custom pipes and that it was actually used for point cloud data.

I would like to be able to simply send JSON strings across custom pipes that will convert to rostopics with the mpa_to_ros service which is currently not possible since only the interfaces found here are supported.

Would it be possible for ModalAI to add a feature to the voxl_mpa_to_ros code to allow for a custom pipe with JSON string data to be sent? It would make sense that the voxl_mpa_to_ros would be able to support converting any JSON string data sent across pipes in the libmodal pipe architecture with functions like int pipe_server_write(int ch, const void* data, int bytes); to be viewed in ROS.

It appears that the VOXL Time of Flight Sensor has different Field of View specs based on distance according to this datasheet:
https://docs.modalai.com/voxl-tof-sensor-datasheet/

Specifically, it appears that the FOV changes once the TOF sensor is "near field":

FOV (H x V)	117° x 117° (near field 100° x 80°)

I'm curious what defines near field and if this switch to 100x80 is instant or transforming from 117x117 with time and/or distance?

It appears that there are two imu's (imu0 and imu1) available on the VOXL. Out of curiousity, what is the motivation behind having two imu's on the VOXL. Do they serve different functions?

I'm curious what screws are used to screw in the hires cam on the VOXL flight deck (top center camera)? I'd like to buy some extras.

Re: Auto-Exposure for Hires Sensors?

I am using voxl-camera-server with three sensors:

1. The 4k hires sensor included with the voxl flight deck
2. the tracking sensor included with the voxl flight deck
3. An added TOF sensor from ModalAl in place of the stereo cameras

I am curious if voxl-camera-server automatically enables auto-exposure and if there is a way to verify that it is enabled?