ModalAI Forum

sansoy

@Denver-Bennett said in How do I connect MAVROS to PX4 with UDP ports?:

voxl-vision-hub.conf

did you ever get this resolved? having same issue.
S

sansoy

/uas_20/mission_control_server/mission_filepath: /home/root/catkin...
/uas_20/mission_control_server/use_navsatfix_launch: False

NODES
/uas_20/
mavros (mavros/mavros_node)
mission_control_server (demins_mavros/mission_control_server.py)

ROS_MASTER_URI=http://10.10.0.50:11311/

process[uas_20/mavros-1]: started with pid [2835]
process[uas_20/mission_control_server-2]: started with pid [2836]
[INFO] [1767215095.413547]: Waiting for MAVROS services...
[INFO] [1767215095.511182]: Waiting for mavros/mission/push service...
[ WARN] [1767215095.880973783]: DS: plugin not configured!
[INFO] [1767215097.183459]: Waiting for mavros/mission/pull service...
[INFO] [1767215097.254429]: Waiting for mavros/mission/clear service...
[INFO] [1767215097.313131]: Waiting for mavros/mission/set_current service...
[INFO] [1767215097.362325]: Waiting for mavros/set_mode service...
[INFO] [1767215097.412324]: Waiting for mavros/cmd/arming service...
[INFO] [1767215097.469221]: Waiting for mavros/param/set service...
[INFO] [1767215097.520002]: All MAVROS services are available!
[INFO] [1767215097.541382]: Clearing all waypoints from FCU...
[ WARN] [1767215098.608210241]: WP: timeout, retries left 2
[ WARN] [1767215099.609752740]: WP: timeout, retries left 1
[ WARN] [1767215100.610535292]: WP: timeout, retries left 0
[ERROR] [1767215101.611777063]: WP: timed out.
[ERROR] [1767215101.616417]: Failed to clear waypoints
[INFO] [1767215101.670286]: Waypoint Mission Manager Initialized.
[INFO] [1767215101.864318]: Mission Control Server Initialized.

i'm at a loss here. tried everything with ports etc

here's my launch script
roslaunch projectx voxl_end.launch mav_sys_id:=20 fcu_url:="udp://127.0.0.1:14540@127.0.0.1:14550"

sansoy

@tom thanks tom!

with voxl-configure-sku -w
I assumed the option to select is
6) px4-autonomy-dev-kit

i then ran voxl-configure-mpa
saw a few errors but the qvio is working now with no errors.

Sabri

sansoy

@tom i ran
journalctl -u voxl-qvio-server
and saw this message as the extrinsics.conf file doesnot exist

failed to find extrinsics from imu_px4 to tracking in /etc/modalai/extrinsics.conf

sansoy

@tom Screenshot from 2024-02-14 15-48-11.png i did that and tried restarting the voxl-qvio-server but it wont start.

sansoy

Screenshot from 2024-02-14 08-21-10.png

i've followed all instructions from calibrating the tracking camera
https://docs.modalai.com/calibrate-cameras

to
https://docs.modalai.com/voxl-m500-user-guide-using-vio

i've configured every setting with voxl-configure-qvio and still get the following error

voxl2:/$ voxl-inspect-qvio
waiting for server

any ideas?

sansoy

@bgaudel Did you ever get this resolved. I'm experiencing the same thing with IMU_MISSING STALLED

sansoy

@thomas I already wrote a c helper function that uses libmodal-pipe to pipe tflite_data that my python script reads in realtime. it works well
per your message " small C++ helper program to take in images from the camera". how would i modify my c helper to use libmodal-pipe to stream data from one of the cameras?

sansoy

@thomas i'm not using voxl-tflite-server. On my voxl2 i installed the latest version of YOLOv8 from Ultralytics https://docs.ultralytics.com/ and it works well when i run inference on local images. i wanted to test just using a local python script that streams video frames from the hires_small_color camera and runs inference.

sansoy

@thomas Hey thomas, we've got the handshaking between your C helper program and our python script working well. i successfully loaded Yolov8 on the voxl and it works really well on images. Was wondering if we could open the camera pipe the same as the tflite_data pipe to run inference on the frames in realtime. i noticed in your /data/web_root/index.html that you are using websockets for serving the videos in the browsers. is this the better route?

sansoy

@thomas its all working now.

had to reinstall the latest sdk. I'm guessing make install is a no-no. so just did a make and ran the executable.

voxl2:/libmodal-pipe/build/examples(master)$ ./modal-tflite_data-client
channel 0 connected to server
Bounding Box Coordinates:
score: 0.558105, x_min: 406.000000, y_min: 417.000000, x_max: 535.000000, y_max: 472.000000
Bounding Box Coordinates:
score: 0.429199, x_min: 427.000000, y_min: 370.000000, x_max: 573.000000, y_max: 426.000000
Bounding Box Coordinates:
score: 0.617676, x_min: 498.000000, y_min: 300.000000, x_max: 655.000000, y_max: 358.000000
Bounding Box Coordinates:
score: 0.416748, x_min: 505.000000, y_min: 302.000000, x_max: 659.000000, y_max: 359.000000
Bounding Box Coordinates:
score: 0.739258, x_min: 506.000000, y_min: 304.000000, x_max: 667.000000, y_max: 364.000000
Bounding Box Coordinates:
score: 0.493652, x_min: 4.000000, y_min: -1.000000, x_max: 781.000000, y_max: 109.000000
Bounding Box Coordinates:
score: 0.547852, x_min: 509.000000, y_min: 306.000000, x_max: 664.000000, y_max: 363.000000
Bounding Box Coordinates:
score: 0.640625, x_min: 505.000000, y_min: 310.000000, x_max: 660.000000, y_max: 369.000000

sansoy

@thomas from my syslog

Jan 23 00:46:51 m0054 voxl-camera-server[4202]: /usr/bin/voxl-camera-server: symbol lookup error: /usr/bin/voxl-camera-server: undefined symbol: pipe_validate_mavlin
k_message_t
Jan 23 00:46:51 m0054 systemd[1]: voxl-camera-server.service: Main process exited, code=exited, status=127/n/a
Jan 23 00:46:51 m0054 systemd[1]: voxl-camera-server.service: Failed with result 'exit-code'.
Jan 23 00:46:51 m0054 voxl-px4[1656]: ERROR [muorb] SLPI: Failed to parse version request response packet!
Jan 23 00:46:53 m0054 voxl-px4[1656]: message repeated 534 times: [ ERROR [muorb] SLPI: Failed to parse version request response packet!]
Jan 23 00:46:53 m0054 systemd[1]: voxl-camera-server.service: Service hold-off time over, scheduling restart.
Jan 23 00:46:53 m0054 systemd[1]: voxl-camera-server.service: Scheduled restart job, restart counter is at 1.
Jan 23 00:46:53 m0054 systemd[1]: Stopped voxl-camera-server.
Jan 23 00:46:53 m0054 systemd[1]: Started voxl-camera-server.
Jan 23 00:46:53 m0054 voxl-camera-server[4226]: /usr/bin/voxl-camera-server: symbol lookup error: /usr/bin/voxl-camera-server: undefined symbol: pipe_validate_mavlin
k_message_t
Jan 23 00:46:53 m0054 systemd[1]: voxl-camera-server.service: Main process exited, code=exited, status=127/n/a
Jan 23 00:46:53 m0054 systemd[1]: voxl-camera-server.service: Failed with result 'exit-code'.
Jan 23 00:46:53 m0054 voxl-px4[1656]: ERROR [muorb] SLPI: Failed to parse version request response packet!

Actually several enabled services wont restart
Service Name | Enabled | Running | CPU Usage

all i did was compile and make install my modal-tflite_data-client.c file

sansoy

@thomas so i included ai_detection.h and cast the char* data variable and compiled successfully and several of my services died ...lol

Service Name | Enabled | Running | CPU Usage

I've tried everything to restart both camera server and cpu monitor and no luck. Rebooted, Cycled Power,
Any thoughts on how to recover from this?

sansoy

@sansoy so i updated my copy named modal-tflite_data-client.c with fields listed in tflite_data/info
{
"name": "tflite_data",
"location": "/run/mpa/tflite_data/",
"type": "ai_detection_t",
"server_name": "voxl-tflite-server",
"size_bytes": 16384,
"server_pid": 25994
}

Recompiled the script and ran
voxl2:/run/mpa/tflite_data$ modal-tflite_data-client
channel 0 connected to server
sending tflite_data to server control pipe
failed to send control command to server
ERROR: control pipe not available

but when i put a picture in front of it and the object detection is working i'm getting!

voxl2:/run/mpa/tflite_data$ modal-tflite_data-client
channel 0 connected to server
sending tflite_data to server control pipe
failed to send control command to server
ERROR: control pipe not available
received 172 bytes on channel 0: LXOV��O6
received 172 bytes on channel 0: LXOV��F[6
received 172 bytes on channel 0: LXOV7?_�6
received 172 bytes on channel 0: LXOV! ��6
received 172 bytes on channel 0: LXOV
��6
received 172 bytes on channel 0: LXOV��6

sansoy

@thomas ok massive confusion.

i was able to download the example hello scripts and compile them and they work fine.
modal-hello-client.c
modal-hello-pause.c
modal-hello-server.c
modal-hello-sink.c
modal-kill-pipe.c
modal-pipe-ping.c

i then made copies of these and replaced hello with tflite_data and compiled successfuly
however, i cant seem to tie into tflite_data.

so when i'm running voxl-tflite-server two pipes show up, ie tflite and tflite_data

Doesnt that mean these pipes open already?

i tried my modal-tflite_data-client and got the following error

channel 0 connected to server
sending tflite_data to server control pipe
failed to send control command to server
ERROR: control pipe not available
ERROR, pipe is not of type "text"
closing

i just want to get the box coordinates from the pipe. Help me Obi Wan Kenobi!

sansoy

@thomas Thanks Thomas! will get back to you....i'm sure!!

sansoy

The following is a python script generated by bard.google.com which also runs but is not printing any data.

import ctypes
import os

Define structures with packing

class ObjectDetectionMsg(ctypes.Structure):
fields = [
("timestamp_ns", ctypes.c_int64),
("class_id", ctypes.c_uint32),
("class_name", ctypes.c_char * 64),
("class_confidence", ctypes.c_float),
("detection_confidence", ctypes.c_float),
("x_min", ctypes.c_float),
("y_min", ctypes.c_float),
("x_max", ctypes.c_float),
("y_max", ctypes.c_float),
]
pack = 1

class DetectionsArray(ctypes.Structure):
fields = [
("num_detections", ctypes.c_int32),
("detections", ObjectDetectionMsg * 64),
]
pack = 1

Open the pipe for reading

pipe_path = "/run/mpa/tflite_data/request"
pipe_fd = os.open(pipe_path, os.O_RDONLY)

Read data continuously

while True:
# Read a detections_array structure from the pipe
data = os.read(pipe_fd, ctypes.sizeof(DetectionsArray))

# Parse the data into a DetectionsArray object
detections_array = DetectionsArray.from_buffer_copy(data)

# Process the detections
for i in range(detections_array.num_detections):
    detection = detections_array.detections[i]
    # Access detection fields as needed, e.g.,
    print(f"Timestamp: {detection.timestamp_ns}")
    print(f"Class ID: {detection.class_id}")
    print(f"Class name: {detection.class_name.decode()}")
    print(f"Class confidence: {detection.class_confidence}")
    print(f"Detection confidence: {detection.detection_confidence}")
    print(f"Bounding box: ({detection.x_min}, {detection.y_min}) - ({detection.x_max}, {detection.y_max})")

sansoy

I'm not getting any data or errors except for the one error when i restart voxl-tflight-server which seems logical.

the metadata format is
// struct containing all relevant metadata to a tflite object detection
typedef struct ai_detection_t {
uint32_t magic_number;
int64_t timestamp_ns;
uint32_t class_id;
int32_t frame_id;
char class_name[BUF_LEN];
char cam[BUF_LEN];
float class_confidence;
float detection_confidence;
float x_min;
float y_min;
float x_max;
float y_max;
} attribute((packed)) ai_detection_t;

here's my code:

import struct
import os

Define the format strings for the structures

Note: This assumes a specific byte order, size, and alignment which might need adjustment

object_detection_msg_format = 'qI64sfff4f'
detections_array_format = 'i'

Size of the object_detection_msg structure

object_detection_msg_size = struct.calcsize(object_detection_msg_format)

Function to unpack a single object_detection_msg

def unpack_object_detection_msg(data):
unpacked_data = struct.unpack(object_detection_msg_format, data)
return {
'timestamp_ns': unpacked_data[0],
'class_id': unpacked_data[1],
'class_name': unpacked_data[2].decode('utf-8').rstrip('\x00'),
'class_confidence': unpacked_data[3],
'detection_confidence': unpacked_data[4],
'x_min': unpacked_data[5],
'y_min': unpacked_data[6],
'x_max': unpacked_data[7],
'y_max': unpacked_data[8]
}

Function to read and unpack the data from the pipe

def read_detections(pipe_fd):
# Read the number of detections
num_detections_data = os.read(pipe_fd, struct.calcsize(detections_array_format))

if len(num_detections_data) != struct.calcsize(detections_array_format):
    raise ValueError("Did not read the expected number of bytes for num_detections")

num_detections = struct.unpack(detections_array_format, num_detections_data)[0]

# Read and unpack each detection
detections = []
for _ in range(num_detections):
    detection_data = os.read(pipe_fd, object_detection_msg_size)
    detection = unpack_object_detection_msg(detection_data)
    detections.append(detection)

return detections

pipe = open('/run/mpa/tflite_data/request', 'rb')
pipe_fd = pipe.fileno()

try:
detections = read_detections(pipe_fd)
print(detections)
finally:
pipe.close()

Any suggestions/solutions/help would be awesomely appreciated!

sansoy

@thomas said in Can anyone recommend a Tflite Colab Notebook for VOXL2 Training:

It's important to note that voxl-tflite-server isn't intended to be a high-grade tool for running any ML model with great configurations and optimizations; it's just an example of how you can use the VOXL ecosystem to run models. As much as I would enjoy building such a tool and implementing your request (and the others I get), there just isn't enough of a business demand for it right now.

Thanks for the info. how can i get access to the box coordinates in the image?

sansoy

@thomas WORKED! i did most of what you stated except for renaming my file to your filename. !

in a future release would be able to use the original filename vs renaming it to something that is hardcoded?