ModalAI Forum

@Matt-Turi thanks for letting me know! Does this new version crash less often? Also, will the output pipe work with voxl_mpa_to_ros?

After running:

roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch 

on the VOXL with the latest stable version of voxl-tflite-server running (v0.1.6), I noticed that the tflite pipe would disconnect and reconnect after about 5-10 minutes.

Here is an output of that occuring:

Found new interface: tflite
Found new interface: tracking
Found new interface: cpu_monitor
Found new interface: imu0
Found new interface: imu1
Found new interface: vvpx4_gps_raw_int
Found new interface: vvpx4_mavlink_io
Found new interface: vvpx4_sys_status
Found new interface: vvpx4_vehicle_gps
Found new interface: vvpx4_body_wrt_fixed
Found new interface: vvpx4_body_wrt_local
Found new interface: qvio_extended
Found new interface: vvpx4_shell
Found new interface: qvio
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising
Interface: tflite's data pipe disconnected, closing until it returns
Found pipe for interface: tflite, now advertising

When this happens all of the cameras on voxl-portal are no longer viewable. I can select them but no image shows up. Even if I restart voxl-portal or the voxl-tflite server the cameras don't show up in voxl-portal. The only fix to this is to restart the voxl entirely.

I then tested to see if this happened without voxl-mpa-to-ros running. Unfortunately, the same error still occurs. After about 5-10 minutes the cameras are no longer viewable in voxl-portal with the only fix being to reboot the voxl.

Should I be building the ipk package for my custom version of mpa_to_ros in the voxl-cross container? It appears that voxl-cross doesn't contain /opt/ros/indigo/setup.bash which is sourced by the build script in voxl_mpa_to_ros

Thanks for the update @Alex-Gardner! Can you clarify what you mean when you say the source doesn't need the launch file anymore? What would be the proper way to launch mpa_to_ros without the launch file?

Hello,

I just wanted to follow up on the above post.

Thanks!

I am currently modifying the voxl-tflite-server to send custom metadata about the detection, confidence, and bounding box coordinates across a custom-made pipe called tflite_metadata. I have been using voxl_mpa_to_ros to view the data from certain sensors in ROS. I am able to customize what data I view from existing sensors by modifying the voxl_mpa_to_ros.launch script as follows:

yocto:~$ cat custom_voxl_mpa_to_ros.launch 
<?xml version="1.0"?>

<!--
/****************************************************************************
 * Copyright 2021 Modal AI
 * Complete MPA launch file exposes all node params as arguments
 ****************************************************************************/
-->


<launch>

    <!-- This parameter will remove tof points with lower confidence than the given value
         Confidence values range from 0-255 -->
    <arg name="tof_cutoff"     default="100"             />

    <!-- TOF point cloud data is published w.r.t the TOF ros tf frame -->
    <arg name="tof_frameid"    default="tof"             />

    <!-- sample interface variables, specify whether to publish, and the pipe to look for -->
    <arg name="image0_publish" default="false"                  />
    <arg name="image0_pipe"    default="tracking"              />

    <arg name="image1_publish" default="false"                  />
    <arg name="image1_pipe"    default="hires_preview"         />

    <arg name="image2_publish" default="true"                  />
    <arg name="image2_pipe"    default="qvio_overlay"          />

    <arg name="image3_publish" default="true"                  />
    <arg name="image3_pipe"    default="dfs_disparity"         />

    <arg name="image4_publish" default="true"                  />
    <arg name="image4_pipe"    default="tflite"                />

    <arg name="image5_publish" default="false"                 />
    <arg name="image5_pipe"    default=""                      />

    <arg name="image6_publish" default="false"                 />
    <arg name="image6_pipe"    default=""                      />

    <arg name="image7_publish" default="false"                 />
    <arg name="image7_pipe"    default=""                      />

    <arg name="stereo0_publish"   default="false"               />
    <arg name="stereo0_pipe"      default="stereo"             />

    <arg name="stereo1_publish"   default="false"              />
    <arg name="stereo1_pipe"      default=""                   />

    <arg name="tof0_publish"      default="true"               />
    <arg name="tof0_pipe"         default="tof"                />

    <arg name="tof1_publish"      default="false"              />
    <arg name="tof1_pipe"         default=""                   />

    <arg name="imu0_publish"      default="true"               />
    <arg name="imu0_pipe"         default="imu0"               />

    <arg name="imu1_publish"      default="true"               />
    <arg name="imu1_pipe"         default="imu1"               />

    <arg name="vio0_publish"      default="true"               />
    <arg name="vio0_pipe"         default="qvio"               />

    <arg name="vio1_publish"      default="false"              />
    <arg name="vio1_pipe"         default=""                   />

    <arg name="PC0_publish"       default="true"               />
    <arg name="PC0_pipe"          default="dfs_point_cloud"    />

    <arg name="PC1_publish"       default="true"              />
    <arg name="PC1_pipe"          default="tflite_metadata"                   />

    <arg name="PC2_publish"       default="false"              />
    <arg name="PC2_pipe"          default=""                   />

    <arg name="PC3_publish"       default="false"              />
    <arg name="PC3_pipe"          default=""                   />

    <arg name="namespace" default="mpa" />
    <!-- start mpa node with all arguments -->
    <group ns="$(arg namespace)">
        <node name="voxl_mpa_to_ros_node" type="voxl_mpa_to_ros_node" pkg="voxl_mpa_to_ros" output="screen">

            <param name="tof_cutoff" value="$(arg tof_cutoff)" />
            <param name="tof_frameid" value="$(arg tof_frameid)" />

            <param name="image0_publish"    value="$(arg image0_publish)"    />
            <param name="image0_pipe"       value="$(arg image0_pipe)"       />

            <param name="image1_publish"    value="$(arg image1_publish)"    />
            <param name="image1_pipe"       value="$(arg image1_pipe)"       />

            <param name="image2_publish"    value="$(arg image2_publish)"    />
            <param name="image2_pipe"       value="$(arg image2_pipe)"       />

            <param name="image3_publish"    value="$(arg image3_publish)"    />
            <param name="image3_pipe"       value="$(arg image3_pipe)"       />

            <param name="image4_publish"    value="$(arg image4_publish)"    />
            <param name="image4_pipe"       value="$(arg image4_pipe)"       />

            <param name="image5_publish"    value="$(arg image5_publish)"    />
            <param name="image5_pipe"       value="$(arg image5_pipe)"       />

            <param name="image6_publish"    value="$(arg image6_publish)"    />
            <param name="image6_pipe"       value="$(arg image6_pipe)"       />

            <param name="image7_publish"    value="$(arg image7_publish)"    />
            <param name="image7_pipe"       value="$(arg image7_pipe)"       />

            <param name="stereo0_publish"   value="$(arg stereo0_publish)"   />
            <param name="stereo0_pipe"      value="$(arg stereo0_pipe)"      />

            <param name="stereo1_publish"   value="$(arg stereo1_publish)"   />
            <param name="stereo1_pipe"      value="$(arg stereo1_pipe)"      />

            <param name="tof0_publish"       value="$(arg tof0_publish)"     />
            <param name="tof0_pipe"          value="$(arg tof0_pipe)"        />

            <param name="tof1_publish"       value="$(arg tof1_publish)"     />
            <param name="tof1_pipe"          value="$(arg tof1_pipe)"        />

            <param name="imu0_publish"      value="$(arg imu0_publish)"      />
            <param name="imu0_pipe"         value="$(arg imu0_pipe)"         />

            <param name="imu1_publish"      value="$(arg imu1_publish)"      />
            <param name="imu1_pipe"         value="$(arg imu1_pipe)"         />

            <param name="vio0_publish"      value="$(arg vio0_publish)"      />
            <param name="vio0_pipe"         value="$(arg vio0_pipe)"         />

            <param name="vio1_publish"      value="$(arg vio1_publish)"      />
            <param name="vio1_pipe"         value="$(arg vio1_pipe)"         />

            <param name="PC0_publish"      value="$(arg PC0_publish)"        />
            <param name="PC0_pipe"         value="$(arg PC0_pipe)"           />

            <param name="PC1_publish"      value="$(arg PC1_publish)"        />
            <param name="PC1_pipe"         value="$(arg PC1_pipe)"           />

            <param name="PC2_publish"      value="$(arg PC2_publish)"        />
            <param name="PC2_pipe"         value="$(arg PC2_pipe)"           />

            <param name="PC3_publish"      value="$(arg PC3_publish)"        />
            <param name="PC3_pipe"         value="$(arg PC3_pipe)"           />

        </node>

        <group ns="$(arg image0_pipe)/image_raw" >
          <rosparam param="disable_pub_plugins">
            - 'image_transport/compressedDepth'
            - 'image_transport/compressed'
          </rosparam>
        </group>
    </group>

</launch>

I realized after running the above script and running:

rostopic info tflite_metadata

that I had falsely assumed that the PC0_pipe name was an empty slot for custom pipes and that it was actually used for point cloud data.

I would like to be able to simply send JSON strings across custom pipes that will convert to rostopics with the mpa_to_ros service which is currently not possible since only the interfaces found here are supported.

Would it be possible for ModalAI to add a feature to the voxl_mpa_to_ros code to allow for a custom pipe with JSON string data to be sent? It would make sense that the voxl_mpa_to_ros would be able to support converting any JSON string data sent across pipes in the libmodal pipe architecture with functions like int pipe_server_write(int ch, const void* data, int bytes); to be viewed in ROS.

It appears that the VOXL Time of Flight Sensor has different Field of View specs based on distance according to this datasheet:
https://docs.modalai.com/voxl-tof-sensor-datasheet/

Specifically, it appears that the FOV changes once the TOF sensor is "near field":

FOV (H x V)	117° x 117° (near field 100° x 80°)

I'm curious what defines near field and if this switch to 100x80 is instant or transforming from 117x117 with time and/or distance?

It appears that there are two imu's (imu0 and imu1) available on the VOXL. Out of curiousity, what is the motivation behind having two imu's on the VOXL. Do they serve different functions?

I'm curious what screws are used to screw in the hires cam on the VOXL flight deck (top center camera)? I'd like to buy some extras.

Re: Auto-Exposure for Hires Sensors?

I am using voxl-camera-server with three sensors:

1. The 4k hires sensor included with the voxl flight deck
2. the tracking sensor included with the voxl flight deck
3. An added TOF sensor from ModalAl in place of the stereo cameras

I am curious if voxl-camera-server automatically enables auto-exposure and if there is a way to verify that it is enabled?

I'm following the instructions given here to use voxl_mpa_to_ros. Everything runs fine so long as I run the command:

roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch

However, when I run the command ```

voxl-mpa-to-ros

I get the following:

yocto:~$ voxl-mpa-to-ros
bash: voxl-mpa-to-ros: command not found

This would not be a big deal except for the fact that I would like to choose which services to publish to ros by providing a custom .launch file. I tried to use roslaunch with a custom .launch file
and I got the following error:

yocto:~$ roslaunch voxl_mpa_to_ros custom_voxl_mpa_to_ros.launch
[custom_voxl_mpa_to_ros.launch] is neither a launch file in package [voxl_mpa_to_ros] nor is [voxl_mpa_to_ros] a launch file name
The traceback for the exception was written to the log file

How would I go about providing a custom launch file to choose which services to publish?

@Matt-Turi this is great to know, thanks Matt! Just curious, any reason why mobilenetv2 was chosen from the tf1/tf2 zoo when it seems that there are faster and more accurate models available?

@Matt-Turi did you try the full fledged version of yolo or was it tiny-yolo?

I have some yolo weights on some custom data that I collected for yolo that I wanted to test with voxl-tflite-server. Additionally, I noticed that in there are some classes missing in the coco_labels.txt file included with the code for voxl-tflite-server in master. Some classes such as 'desk' are replaced with question marks:

0  person
1  bicycle
2  car
3  motorcycle
4  airplane
5  bus
6  train
7  truck
8  boat
9  traffic light
10  fire hydrant
11  ???-11
12  stop sign
13  parking meter
14  bench
15  bird
16  cat
17  dog
18  horse
19  sheep
20  cow
21  elephant
22  bear
23  zebra
24  giraffe
25  ???
26  backpack
27  umbrella
28  ???-28
29  ???-29
30  handbag
31  tie
32  suitcase
33  frisbee
34  skis
35  snowboard
36  sports ball
37  kite
38  baseball bat
39  baseball glove
40  skateboard
41  surfboard
42  tennis racket
43  bottle
44  ???-44
45  wine glass
46  cup
47  fork
48  knife
49  spoon
50  bowl
51  banana
52  apple
53  sandwich
54  orange
55  broccoli
56  carrot
57  hot dog
58  pizza
59  donut
60  cake
61  chair
62  couch
63  potted plant
64  bed
65  ???-65
66  dining table
67  ???-67
68  ???-68
69  toilet
70  ???-70
71  tv
72  laptop
73  mouse
74  remote
75  keyboard
76  cell phone
77  microwave
78  oven
79  toaster
80  sink
81  refrigerator
82  ???-82
83  book
84  clock
85  vase
86  scissors
87  teddy bear
88  hair drier
89  toothbrush

@Matt-Turi thanks Matt. I ended up getting a segfault with it. I believe it was too memory intensive because it wasn't supported by the GPU. Is there a chance that tiny-yolo could be supported by ModalAI anytime soon?

I am trying to create a custom version of the voxl-tflite-server using a tflite file of the literature version of the yolo-v4-tiny model. However, when I change the code as well as the the voxl-tflite-server.conf file to include my new model I get the error "FATAL: Unsupported model provided!!" when trying to run the server. I can't find where this message is printed in the code. How can I turn this error off?

Prior to the voxl-tflite-server update to 0.1.3, I was able to stream both the hires and the tflite-overlay using voxl-streamer. Now I am only able to do one or the others. Any suggestions on how to fix this?

@Matt-Turi Great thanks! I'll give it a shot now!

Thanks @Matt-Turi! Can you please reply to this thread to let me know when it’s up?

To be more concrete, my team and I have tried to build the dev version from master, the stable version from master, as well as many other previous versions that we were able to get working before Sept 2 (on multiple drones). The builds all prior worked fine with all of these versions (on multiple drones). After then, we haven't been able to get a single version that will create the tflite pipe to allow voxl-streamer to use the tflite overlay (on any drone). However, all of them show that the tflite server is working fine in debug mode on its own.