ModalAI Forum

@Eric-Katzfey ,
Please excuse me for the confusion caused by me.

Goal -
my goal is to achieve continuous video stream from lepton camera connected through USB cable. the USB port I am using is Microhard.

Issues I faced -

1. During my initial testing, I was able to stream the lepton video through VLC, but the camera stream kept on getting interrupted (the camera stream ended), once I restart the uvc-service and the stream-service , it starts again. This observation was in loop.
   I wanted your help to solve this issue.
2. Once you asked for the log file, I tried to replicate this test, but the stream was not working through VLC.
   I attached the log file.
3. To verify the camera is working fine I tried with ffplay, the stream was working fine.
4. I verified the MIPI camera, I was able to see the video on the voxl-portal, but while using the stream-service, it was not working on VLC.

Now, I would like your help to solve the following problems

1. Get the video stream on VLC
2. Get the video stream to work continuously with out any interruption.
3. Also it would be great if you clear one more query of mine, whether I can setup the USB port with any camera(which supports USB protocol), If I make necessary changes to the codes or UVC-service and Stream-service.
   Thank you for your patience and looking forward to response.

@Eric-Katzfey
sure I can summarize,

Camera 1 - Lepton,
QGC - Did not test
VLC - Did not stream, error message.
ffplay - stream worked

Camera 2 - logitech c270 camera
QGC - Did not test
VLC - Did not stream, error message.
ffplay - stream worked

I did not understand what you meant by mipi camera, if it is normal hires camera connected to the yellow board, it is streaming fine, slightly slow through the microhard connection, but I can see the video stream on the portal.

the following message showed up when i tried to trigged the rtsp stream through vlc

A new client rtsp://192.168.168.150:58656(null) has connected, total clients: 2
rtsp client disconnected, total clients: 1
rtsp client disconnected, total clients: 0
no more rtsp clients, closing source pipe intentionally
A new client rtsp://192.168.168.150:45852(null) has connected, total clients: 1
Camera server Connected
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm

@eric-katzfey
I tried with c270 camera, the observation was similar was not able to stream in vlc, but was able to stream through ffplay.

@Eric-Katzfey
I did not try streaming over wifi, i only try using Microhard.

The following arr log files of lepton.

This is the log file while trying to recreate the scenario. I am not able to connect to the VLC through rtsp but the rtsp stream is visible through ffplay, I attaching that log too.

the following are the log files of rtsp stream and connecting through vlc only.

voxl2:/$ voxl-streamer --standalone --port 8904 -i uvc

Waiting for pipe uvc to appear

Found Pipe

detected following stats from pipe:

w: 160 h: 120 fps: 9 format: YUV422_UYVY

Stream available at rtsp://127.0.0.1:8904/live

A new client rtsp://192.168.168.150:46960(null) has connected, total clients: 1

Camera server Connected

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

gbm_create_device(156): Info: backend name is: msm_drm

A new client rtsp://192.168.168.150:46976(null) has connected, total clients: 2

rtsp client disconnected, total clients: 1

rtsp client disconnected, total clients: 0

no more rtsp clients, closing source pipe intentionally 

this is the log file of when I ran first using ffplay, then tried with vlc. that when the error again was observed.

voxl2:/$ voxl-streamer --standalone --port 8901 -i uvc
Waiting for pipe uvc to appear
Found Pipe
detected following stats from pipe:
w: 160 h: 120 fps: 9 format: YUV422_UYVY
Stream available at rtsp://127.0.0.1:8901/live
A new client rtsp://192.168.168.150:60200(null) has connected, total clients: 1
Camera server Connected
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
A new client rtsp://192.168.168.150:60210(null) has connected, total clients: 2
rtsp client disconnected, total clients: 1
ERROR:   New frame rejected, status = -2
rtsp client disconnected, total clients: 0
no more rtsp clients, closing source pipe intentionally
Removed 1 sessions
A new client rtsp://192.168.168.150:49562(null) has connected, total clients: 1
Camera server Connected
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
A new client rtsp://192.168.168.150:49570(null) has connected, total clients: 2
rtsp client disconnected, total clients: 1

(voxl-streamer:28035): GLib-GObject-WARNING **: 16:00:31.725: instance with invalid (NULL) class pointer

(voxl-streamer:28035): GLib-GObject-CRITICAL **: 16:00:31.726: g_signal_emit_by_name: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
rtsp client disconnected, total clients: 0
no more rtsp clients, closing source pipe intentionally
A new client rtsp://192.168.168.150:58654(null) has connected, total clients: 1
Camera server Connected
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
A new client rtsp://192.168.168.150:58656(null) has connected, total clients: 2
rtsp client disconnected, total clients: 1
rtsp client disconnected, total clients: 0
no more rtsp clients, closing source pipe intentionally
A new client rtsp://192.168.168.150:45852(null) has connected, total clients: 1
Camera server Connected
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
Removed 1 sessions

t@t:~$ ffplay -rtsp_transport tcp rtsp://192.168.168.100:8901/live 
ffplay version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://192.168.168.100:8901/live': 0B f=0/0   
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.592211, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 160x120, 9 fps, 8.75 tbr, 90k tbn, 18 tbc
t@t:~$ d=   4 aq=    0KB vq=    0KB sq=    0B f=1/1   

this is running without any interruptions

voxl2:/$ voxl-uvc-server  -d
loading config file
Enabling debug messages
=================================================================
width:                            160
height:                           120
fps:                              9
pipe_name:                        uvc
=================================================================
voxl-uvc-server starting
Image resolution 160x120, 9 fps chosen
UVC initialized
Device found
Device opened
uvc_get_stream_ctrl_format_size succeeded for format UYVY
Streaming starting
Got frame callback! frame_format = 4, width = 160, height = 120, length = 640, ptr = (nil)
 * got image 30
 * got image 60
 * got image 90
 * got image 120
 * got image 150
 * got image 180
 * got image 210
 * got image 240
 * got image 270
 * got image 300
 * got image 330
 * got image 360
 * got image 390
 * got image 420
 * got image 450
 * got image 480
 * got image 510
 * got image 540
 * got image 570
 * got image 600
 * got image 630
 * got image 660
 * got image 690
 * got image 720
 * got image 750
 * got image 780
 * got image 810

Hi @alex-kushleyev @vinny @eric-katzfey
Hardware: (voxl 2 , microhard board, with lepton connected through usb port on the carrier board)

Software version: (voxl sdk 1.5.0)

What I am trying to do: to stream video through usb connection, using the UVC stream service on voxl 2 . The lepton stream through SPI was low quality compared to the usb connection.

What actually happens: The stream was successful , with good quality video stream, which I was able to see in the QGC. The problem i was facing was the irregular stopping of the streaming. I had to restart the service multiple times to see the video.

If there is something I am missing to get a regular continuous live streaming through microhard board , please do share. I want to experiment with different cameras before finalising the components, the USB connection helps me to do that.
Please help to stream live video from the lepton camera through the microhard board.

Steps to reproduce: I followed normal Microhard connection steps , along with setting up the UVC stream service for the Lepton camera, I used the groups gets usb board. The camera was detected and the stream started, after a continuous stream of 25 to 30 seconds the streaming stops, and behavior was observed mulitple times.

@Vinny Thank you for reverting back. I will try to externally power the Doodle labs module.

I wanted a clarity on the ADB Debug port, How can I connect the sensor or any extra module(for my use case) and get it working. what will be the challenges that I might face.

Hi @eric-katzfey @vinny @alex-kushleyev
https://www.modalai.com/products/m0062-2
will this expander board help me connect to connect the USB based Radar and connect Doodle labs to another.

Or else can I connect the USB based Radar to the Debug ADB (USBC HOST MODE) port?

@Eric-Katzfey Thank you for reverting back, I will use j18 or 19 for the flight controller connection.
So I am assuming the UART on J3 is capable of accepting the radar data(if the radar supports UART communication).

I have some more queries that I have mentioned above, Please help me with those. Especially how to install the drivers, where to install it, and also how to access the data in ROS.

Than you for the responses.

@vinny @alex-kushleyev

@eric-katzfey Thank you for getting back to me.

To clarify my setup:
I am replicating the RADVIO pipeline from NTNU using their ROS 1 codebase. The software messaging between PX4 and ROS is already sorted; my main challenge is finalizing the physical hardware interfaces on the VOXL 2.

Current Hardware Configuration:

J3 Port: Attached to a USB 3.0 / UART expansion board.

Expansion UART: Connected to FlightCore v2.

Expansion USB: Connected to a Doodle Labs mini radio.

Unused/Free Ports: J10, J18, and J19.

Key Questions:

1. Physical Port Mapping: Depending on how the radar needs to be interfaced, which specific headers/ports on VOXL 2 should be allocated for:

• list itemUSB: (If connecting via a USB interface)

• list itemUART: (If connecting via serial)

• list itemI2C: (If connecting via I2C)

2. Alternate UART Setup: If I disconnect the J3 expansion UART from the FlightCore v2 and instead route direct UART via J18 (for an FPV ESC), can the freed-up UART port on the J3 expansion board be safely reassigned to the radar?

3. Drivers & ROS 1 Integration: What is the recommended workflow or driver stack on VOXL 2 for bringing up radar serial/USB streams and exposing them to ROS 1 nodes?

Any advice on the optimal pinout and hardware topology for this build would be greatly appreciated!

Hello @eric-katzfey @vinny,

I am planning to integrate a 60 GHz mmWave radar into my current VOXL 2-based system and would appreciate some guidance regarding the recommended interface, hardware compatibility, and software integration.

I am currently evaluating the following two radar options:

1. DesignCore RS-6843AOPU USB 60 GHz mmWave Radar Sensor — D3 Engineering
D3 Engineering RS-6843AOPU

2. Texas Instruments IWR6843AOPEVM — IWR6843AOP Evaluation Module
TI IWR6843AOPEVM

Both are based on the IWR6843AOP 60–64 GHz mmWave radar platform.

Current VOXL 2 configuration

My current system uses the J3 connector for my existing Doodle Labs and Flight Controller connections.

Since I am currently using the USB connection available on J3 for these existing connections, the UART and I²C interfaces are still available for my radar integration.

Therefore, I am considering using the J19 External Sensors connector, which provides UART and I²C interfaces, to connect the external radar while keeping my existing J3 configuration unchanged.

I would like to confirm whether this approach is feasible and which interface would be recommended for the radar.

UART

• Can the RS-6843AOPU or IWR6843AOPEVM be connected to VOXL 2 through one of the J19 UART interfaces?

• Which J19 UART would you recommend for a dedicated radar application?

• Are the J19 UARTs already assigned to any default VOXL 2 functionality, and can one be reassigned to a custom radar application?

• Are the UART voltage levels of both radar options compatible with the 3.3 V CMOS UART on VOXL 2?

• What baud rate and data protocol would you recommend for receiving processed radar data?

I²C

I am also considering using I²C instead of UART, particularly if I only need processed radar information such as range, radial velocity, angle, target information, or obstacle information.

• Can I use the VOXL 2 J19 I²C interface to connect either of these radar modules?

• Which J19 I²C bus would you recommend?

• Are either of the J19 I²C buses already reserved for other VOXL 2 peripherals/functions?

• Can a custom VOXL application directly access the J19 I²C bus?

• Is there an existing VOXL driver/API that can be used for a custom I²C radar device?

• Between UART and I²C, which would you recommend for radar integration considering bandwidth, latency, reliability, and software support?

USB

I would also like to understand the USB option.

• For the RS-6843AOPU, is USB the preferred interface for accessing its processed radar/algorithm output?

• For the IWR6843AOPEVM, can VOXL 2 receive the radar point-cloud data directly over USB?

• Given that my J3 USB connection is already being used, is there another recommended USB host interface/add-on for connecting the radar to VOXL 2?

Final recommendation

Before I start the integration, I would especially appreciate your recommendation on:

RS-6843AOPU vs. IWR6843AOPEVM

and

J19 UART vs. J19 I²C vs. USB

for the most reliable and maintainable VOXL 2 integration.

My main requirement is to keep my existing Doodle Labs and Flight Controller connections on J3 unchanged and use the remaining available VOXL 2 interfaces for the radar.

Any guidance on the recommended hardware connection, J19 pin/interface selection, voltage levels, and software integration path would be greatly appreciated.

Thank you!

@moderator Any reply will be much appreciated.

@Eric-Katzfey thank you for reverting back, I am using an Optical flow along with FC for position hold, for the scenarios where the VIO fails.
My current architecture has voxl 2 + flight core v2 + optical flow attached to the Flight core + voxl FPV esc.

Please could you give me update whether a new FC is coming up. It would help me reconsider the architecture.

Hey @Eric-Katzfey ,
I have similar query. Not exactly the same, but I want to run ardupilot along with Voxl fpv 4 in 1 ESC.
Ardupilot on 3rd party FC.

The architecture that I would like to try out is :

Voxl 2 ---> 3 rd Party FC (with ardupilot)----> voxl FPV 4 in 1 ESC

Especially since flight core v2 has reached EOL. Is any new FC being release?

Please help me out here.

@ModalAI-Team Help me out here . I want to purchase some new components, but with out having a clear idea on the architecture I won't be able to do it.
Please guide me here.

@ModalAI-Team
If anyone could help me understand this behaviour and help me solve the boot issue, if there was any way, would mean a lot.

I have 2 VOXL 2 which are not booting. which gave successful result in unbricking but did not detect ADB earlier. That was until last day when while trying unbricking the same boards, I got a positive result in one of the boards.

If this issue can solved by anyway, please help me out.

Hey @Moderator , @ModalAI-Team

I had few more questions regarding the same.

1. Does the 3D map generated have the same quality has PMD TOF?
2. Can the 3D map be used for P-to-P navigation from voxl-Portal?
3. Can the TOF data be used for collision prevention?
4. Can multiple maps be used together?
5. Does Multiple TOF give collision prevention in multi direction?
6. If you could provide a image of map generated by the new TOF, it would be great and help us to make a decision for our architecture.

Thank you for your help and time, looking forward to your response.

Re: voxl2 is not booting

@alex-kushleyev @vinny @eric-katzfey @modaltb

I was trying to unbrick the same voxl 2 again, which was successful and I was even able to flash the 1.4 SDK, and both showed successful. As I completed the reboot after flashing, the board went back to the boot mode, it is not able to pass the boot loop, the LED is constantly ON, ADB is unresponsive again.

Since the log data is huge, I am sharing the link to the log below. Please help me understand this behaviour.

link text

@moderator

Thank you very much for your detailed explanation of the VOXL 2 Mini ToF sensor.

This information will greatly assist us in finalizing our approach for integrating the sensor into our UAV platform.

Thank you once again for your continued support and guidance.

@Robbie-McCue

Thank you for your feedback I will look into this.

@Alex-Kushleyev @Eric-Katzfey

I would appreciate any guidance from the ModalAI team or community regarding third-party radio and VTX integration with VOXL2, as this information will help us finalize our system architecture.

Thank you for your support and guidance.