# Seeking Reference Code for MPA Integration with RTSP Video Streams for TFLite Server Source: https://forum.modalai.com/topic/3220/seeking-reference-code-for-mpa-integration-with-rtsp-video-streams-for-tflite-server Category: General Questions (https://forum.modalai.com/category/2/general-questions) Posted: 2024-03-15 09:14:40 UTC by Ethan Wu Replies: 36 · Views: 12986 ## Ethan Wu · 2024-03-15 09:14:40 UTC Hi, I'd like to read a video stream from an RTSP source and then pass it through MPA to connect to a tflite server. Where can I find good reference code for handling video streams with MPA? Thanks. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-15 14:56:54 UTC @Ethan-Wu , I think one of the easier ways to do this would be using python3. Opencv (python3) has capability of capturing rtsp stream into separate frames. We are currently prototyping python3 bindings for some of the mpa pub/sub (specifically images). So, essentially you would need a python3 script that captures frames using cv2, does any format conversion if needed, and publishes via python mpa bindings. If you are interested in testing out this approach, I can provide an example for you within a few days. What format would you need to publish the image to the tflite server? mono / rgb / yuv etc. Alex ## Reply by Ethan Wu · 2024-03-18 01:48:33 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Sure, a few examples would be great. I think the format is yuv, thank you very much. ## Reply by Ethan Wu · 2024-03-18 01:54:16 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Originally, I was thinking of achieving simple image transmission by modifying the example code under [libmodal-pipe](https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/examples/modal-hello-server.c), and then connecting it to both MPA and the tflite server. Would you advise against this approach? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-18 12:49:20 UTC (in reply to Ethan Wu) @Ethan-Wu , publishing an image using MPA is easy. How were you planning to handle the RTSP video stream and converting the output to YUV? An example of publishing various image types (gray, rgb) can be found in `voxl-camera-server` source code. The example you referenced is a generic one, there is more specific API for publishing images. - create the publisher: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/hal3_camera_mgr.cpp?ref_type=heads#L2161 , you just need to call `pipe_server_create`.. in short... ``` pipe_info_t info; strcpy(info.name , "my_new_camera"); strcpy(info.type , "camera_image_metadata_t"); strcpy(info.server_name, "my_test_app"); info.size_bytes = 64*1024*1024; //pipe buffer size in bytes int pipe_id = pipe_server_get_next_available_channel(); #pipe id to use for publishing int flags = 0; pipe_server_create(pipe_id, info, flags); ``` - publish : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/hal3_camera_mgr.cpp?ref_type=heads#L934 (please take a look at the `camera_image_metadata_t` in `libmodal_pipe` headers. there is a timestamp and other fields that are not directly filled in this example (they are filled elsewhere). short example (assuming image is just raw8): ``` int publish_image(uint8_t * image_data, uint32_t width, uint32_t height) { camera_image_metadata_t meta; meta.magic_number = CAMERA_MAGIC_NUMBER; meta.frame_id = 0; //TODO fill in the frame id / counter meta.width = width; meta.stride = width; meta.height = height; meta.size_bytes = meta.width * meta.height; meta.format = IMAGE_FORMAT_RAW8; meta.timestamp_ns = 0; //TODO: fill in correct timestamp meta.exposure_ns = 0; //TODO: fill in exposure info if available meta.gain = 0; //TODO: fill in gain info if available return pipe_server_write_camera_frame(pipe_id, meta, image_data); } ``` The full definition of `camera_image_metadata_t` can be found here : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/library/include/modal_pipe_interfaces.h#L190 ## Reply by Ethan Wu · 2024-03-19 06:05:37 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, thanks for replying, that really helps. I was intended to use opencv C++ to read rtsp stream, at least to read a local video for testing. I looked into voxl-mpa-tools which makes use of voxl-opencv as one of its dependency, and wrote a simple program to read video source like below : ```C++ #include #include using namespace cv; using namespace std; int main() { string filename; cout << "Enter the file name of the video: "; cin >> filename; VideoCapture cap(filename); if (!cap.isOpened()) { cerr << "Error: Unable to open video file." << endl; return -1; } // play video on local host } ``` While it can played both mp4 and RTSP source on my PC, isOpened always returns False on voxl. I also install python3 and opencv-python on voxl and test it out, but it's able to open mp4 but failed to read RTSP. That goes same on my PC, so I might get something wrong. I have used ffmpeg to confirm that the RTSP URL is correct, so I don't think it's a problem with the URL being incorrect. ## Reply by Ethan Wu · 2024-03-19 06:08:06 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev As for converting video to YUV, I haven't reach that part, but I think opencv might be used for that conversion. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-19 12:46:48 UTC (in reply to Ethan Wu) @Ethan-Wu , I have not tested using opecv in c++ to receive the RTSP streams, but i have tested the same thing in python3 with opencv on VOXL2. I also found that video capture did not work if i just provided the rtsp stream location as the "filename". It turns out that on VOXL2, the opencv video capture uses gstreamer (not sure about PC), so i had to tell opencv what gstreamer pipeline to use. Specifically, here is the the snippet of the python code that i tested to receive h264 stream that was actually generated by `voxl-camera-server` on VOXL2 (i have not tested h265 this way). You can test out a similar approach in C++ ``` import cv2 stream_url = 'rtsp://127.0.0.1:8900/live' #using software decoder (i think) stream = 'gst-launch-1.0 rtspsrc location=' + stream_url + ' latency=0 ! queue ! rtph264depay ! h264parse ! avdec_h264 ! autovideoconvert ! appsink' #using hardware decoder #stream = 'gst-launch-1.0 rtspsrc location=' + stream_url + ' latency=0 ! queue ! rtph264depay ! h264parse ! qtivdec turbo=true ! autovideoconvert ! appsink' vcap = cv2.VideoCapture(stream,cv2.CAP_GSTREAMER) while(1): ret, frame = vcap.read() #check ret and process image... ``` Please note that the current version of opencv installed on voxl2 does not have python3 bindings, but i enabled that in order to use opencv in Python3. We will probably update the opencv voxl2 deb package soon to include the python3 bindings. ## Reply by Ethan Wu · 2024-03-20 07:44:46 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, I'm now able to stream rtsp with opencv-python on my PC, but fail on voxl, and I think that might be some environmental issue. I'm using conda and following [this script](https://github.com/opencv/opencv-python/issues/530#issuecomment-1006343643) to install gstreamer support opencv, and the program just stuck at VideoCapture() whenever start running. Here are some information after I hit ctrl+C to stop it : ``` plaintext ^C (python:21276): GStreamer-CRITICAL **: 07:27:40.646: Trying to dispose element queue0, but it is in PAUSED instead of the NULL state. You need to explicitly set elements to the NULL state before dropping the final reference, to allow them to clean up. This problem may also be caused by a refcounting bug in the application or some element. (python:21276): GStreamer-CRITICAL **: 07:27:40.646: Trying to dispose element pipeline0, but it is in READY instead of the NULL state. You need to explicitly set elements to the NULL state before dropping the final reference, to allow them to clean up. This problem may also be caused by a refcounting bug in the application or some element. [ WARN:0@1.170] global /tmp/tmp.zudKvFNc4N/opencv-python-master/opencv/modules/videoio/src/cap_gstreamer.cpp (1356) open OpenCV | GStreamer warning: unable to start pipeline [ WARN:0@1.170] global /tmp/tmp.zudKvFNc4N/opencv-python-master/opencv/modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created (python:21276): GStreamer-CRITICAL **: 07:27:40.646: Trying to dispose element appsink0, but it is in READY instead of the NULL state. You need to explicitly set elements to the NULL state before dropping the final reference, to allow them to clean up. This problem may also be caused by a refcounting bug in the application or some element. Traceback (most recent call last): File "/playground/play.py", line 17, in vcap = cv2.VideoCapture(stream,cv2.CAP_GSTREAMER) KeyboardInterrupt ``` I'm currently dealing with it and not sure whether it's a problem from voxl or not. Also, since MPA API is written in C++, I'm curious that even if I success to stream RTSP in python, how should I connect them? ## Reply by Ethan Wu · 2024-03-20 09:23:21 UTC (in reply to Ethan Wu) @Ethan-Wu Or perhaps my gstreamer installation is not completely right? Here's the document I read for installation : https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-20 12:30:00 UTC (in reply to Ethan Wu) @Ethan-Wu, Gstreamer on voxl2 has customizations to make use of voxl2 hardware (video encoder, decoder), so i am not sure if standard gstreamer installation would work. Perhaps that is why you are seeing issues with the standard installation of gstreamer. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-20 20:20:33 UTC (in reply to Alex Kushleyev) I wrote some examples how to use use python3 to get external rtsp stream into python using opencv (which uses gstreamer) and then publish RGB image via MPA. - https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/pympa-experimental/tools/python/rtsp_rx.py - https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/pympa-experimental/tools/python/rtsp_rx_mpa_pub.py . This requires updated version of the opencv package for VOXL2 with python3 bindings (i have not published that yet). there is also an experimental interface to get and send images via MPA (using `pympa` python bindings). If you are interested trying out this option, we can explore it ## Reply by Ethan Wu · 2024-03-21 02:45:23 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thank you very much, that really works! And it's REALLY COOL! Now, the remaining issue is with GStreamer support. When streaming RTSP, GStreamer often freezes without any indication or error messages, making it difficult to identify the problem. Although this may not be the most suitable question to ask here, I still want to seek advice and see if there are other ways besides blindly testing different combinations. Thank you for your assistance. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-21 06:12:16 UTC (in reply to Ethan Wu) @Ethan-Wu , can you clarify the gstreamer freezing issue? Are you saying that the client side freezes? what source are you using as the rtsp server? In my testing the above python scripts, i found that if the rtsp stream is generated using a combination of `voxl-camera-server` and `voxl-streamer` then the gstreamer client does freeze occasionally. I will investigate this at some point soon. However, there is a 3rd party app that allows you to use gstreamer on the server side and i did not experience client freezing with that option. In this case, gstreamer server side is also on voxl2 and it is using a `qtiqmmfsrc` plugin to get video from the camera pipeline. ``` #start rtsp server using gst-rtsp-launch ( https://github.com/sfalexrog/gst-rtsp-launch ) gst-rtsp-launch -p 8900 -e live "( qtiqmmfsrc camera=0 name=hires ! video/x-h264,format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! h264parse ! rtph264pay name=pay0 pt=96 )" ``` ## Reply by Ethan Wu · 2024-03-21 09:27:50 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I think the client side is freezing, and I noticed that VideoCapture takes some time to complete, but ultimately hangs on cap.read(). Below is the relevant debug information: ``` created output pipe rtsp-debug, channel 0, flags 0 gst-launch-1.0 rtspsrc location=rtsp://169.254.4.201:554/live0 latency=0 ! queue ! rtph264depay ! h264parse config-interval=-1 ! qtivdec ! qtivtransform ! video/x-raw,format=BGR,width=1280,height=720 ! autovideoconvert ! appsink 0:00:00.000749593 14685 0x55c069b780 WARN GST_REGISTRY gstregistrybinary.c:489:gst_registry_binary_check_magic: Binary registry magic version is different : 1.3.0 != 1.12.0 0:00:00.057125316 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:876:priv_gst_parse_yyparse: unexpected reference "gst-launch-1" - ignoring 0:00:00.057160626 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:882:priv_gst_parse_yyparse: unexpected pad-reference "0" - ignoring 0:00:00.057168959 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "rtspsrc"! 0:00:00.057173647 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "rtspsrc" 0:00:00.057890325 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@0x55c070f730] 0:00:00.057908970 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "rtph264depay"! 0:00:00.057914021 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "rtph264depay" 0:00:00.057918969 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no sink [source=@0x55c070f730] 0:00:00.057923292 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "h264parse"! 0:00:00.057927354 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "h264parse" 0:00:00.057932250 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@(nil)] 0:00:00.057936260 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "qtivdec"! 0:00:00.057939437 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "qtivdec" 0:00:00.057944072 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@(nil)] 0:00:00.057947874 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "qtivtransform"! 0:00:00.057951467 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "qtivtransform" 0:00:00.057956884 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@(nil)] 0:00:00.057971726 14685 0x55c069b780 WARN GST_ELEMENT_FACTORY gstelementfactory.c:456:gst_element_factory_make: no such element factory "autovideoconvert"! 0:00:00.057979851 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:816:priv_gst_parse_yyparse: no element "autovideoconvert" 0:00:00.057985215 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@(nil)] 0:00:00.058182496 14685 0x55c069b780 ERROR GST_PIPELINE grammar.y:901:priv_gst_parse_yyparse: link has no source [sink=@0x55c06f51b0] (python:14685): GStreamer-CRITICAL **: 08:49:15.037: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed (python:14685): GStreamer-CRITICAL **: 08:49:15.038: gst_structure_get_int: assertion 'structure != NULL' failed [ WARN:0@30.090] global cap_gstreamer.cpp:1714 open OpenCV | GStreamer warning: cannot query video width/height (python:14685): GStreamer-CRITICAL **: 08:49:15.038: gst_structure_get_fraction: assertion 'structure != NULL' failed [ WARN:0@30.090] global cap_gstreamer.cpp:1722 open OpenCV | GStreamer warning: cannot query video fps [ WARN:0@30.090] global cap_gstreamer.cpp:1777 open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1 (python:14685): GStreamer-CRITICAL **: 08:50:20.041: gst_sample_get_caps: assertion 'GST_IS_SAMPLE (sample)' failed [ERROR:0@95.092] global cap_gstreamer.cpp:934 retrieveVideoFrame GStreamer: gst_sample_get_caps() returns NULL (python:14685): GStreamer-CRITICAL **: 08:51:25.042: gst_sample_get_caps: assertion 'GST_IS_SAMPLE (sample)' failed [ERROR:0@160.094] global cap_gstreamer.cpp:934 retrieveVideoFrame GStreamer: gst_sample_get_caps() returns NULL ^C (python:14685): GStreamer-CRITICAL **: 08:52:30.044: gst_sample_get_caps: assertion 'GST_IS_SAMPLE (sample)' failed [ERROR:0@225.096] global cap_gstreamer.cpp:934 retrieveVideoFrame GStreamer: gst_sample_get_caps() returns NULL Traceback (most recent call last): File "/voxl-mpa-tools/tools/python/rtsp_rx_mpa_pub.py", line 62, in ret, frame = vcap.read() KeyboardInterrupt ``` Today, I referenced the [official website](https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html?gi-language=c) and tested the situation of OpenCV, gstreamer, and MPA with other example commands. The result is that the image produced by **gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink** can ultimately be seen on the portal. So, I don't think it's an issue with gstreamer installation? My RTSP source is from an external camera (edge computing device) acting as a server, connected to VOXL via Ethernet for transmission. Currently, I am not using VOXL's built-in camera or UVC camera, and I am not generating an RTSP stream through VOXL's services. ## Reply by Ethan Wu · 2024-03-21 09:36:41 UTC (in reply to Ethan Wu) @Ethan-Wu About the error 'unexpected reference "gst-launch-1" ', I do get the same error when I replace the [gstreamer command](https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/pympa-experimental/tools/python/rtsp_rx_mpa_pub.py#L23) with **gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink**, so I think that's not a problem. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-21 11:47:55 UTC (in reply to Ethan Wu) @Ethan-Wu , Hmm. When i tested with a local rtsp stream generated on the same voxl2, the video capture started immediately. The freezing would occur in the python client side sometime after i got a bunch of frames (when using voxl-streamer to produce the rtsp stream). When the freezing happened, it looked like voxl-streamer dropped the client connection for some reason. I will test with a local ip cam that is producing rtsp today. It seems gstreamer rtsp receiving end may not be running correctly on your end. How about i share with you updated native opencv package with python bindings and you could try running this natively on voxl2 without your environment? This whay you can test what I am testing. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-21 12:50:33 UTC (in reply to Alex Kushleyev) I just tried playing a local rtsp stream from an ip cam (using `rtsp_rx_mpa_pub.py`) and it initially starts playing but gets stuck after a few seconds. However, changing the pipeline to use the following (using `avdec_h264` instead of `qtivdec`: ``` .... rtph264depay ! h264parse config-interval=-1 ! avdec_h264 ! autovideoconvert ! appsink ``` Does not get stuck. So there must be something about the hw decoder that causes it to get stuck, I will investigate further.. ## Reply by Ethan Wu · 2024-03-22 01:36:17 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Sharing opencv package would be great, and I also want to share how I built opencv that supports gstreamer. I looked into [this issue](https://github.com/opencv/opencv-python/issues/530) and followed instructions [here](https://discuss.bluerobotics.com/t/opencv-python-with-gstreamer-backend/8842/1), and made sure I saw the followings information to get gstreamer working with opencv by calling cv2.getBuildInformation() : ``` Video I/O: DC1394: NO FFMPEG: YES avcodec: YES (57.107.100) avformat: YES (57.83.100) avutil: YES (55.78.100) swscale: YES (4.8.100) avresample: NO GStreamer: YES (1.14.5) v4l/v4l2: YES (linux/videodev2.h) ``` I was using conda for virtual environment and run opencv with python 3.10 (3.9 and 3.8 are also tested), and my opencv version is 4.9.0.80. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-22 02:04:18 UTC (in reply to Ethan Wu) @Ethan-Wu , I uploaded the opencv deb here : https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-opencv_4.5.5-3_arm64.deb it was built using this branch of voxl-opencv project : https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv/-/tree/add-python3-bindings/ You can just install that deb to overwrite your existing opencv installation and try the python scripts for rtsp natively on voxl2 By the way, how do you get gstreamer to output that debug info? does it automatically print it if there is an issue? ## Reply by Ethan Wu · 2024-03-22 02:29:37 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev You can set the environment variable GST_DEBUG to the number that fits the debug level you prefer. I made it 2 so that I can see warning and error output from gst-launch-1.0, here's the reference documentation : https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c ## Reply by Ethan Wu · 2024-03-22 10:00:23 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, Do I just need to upload it to voxl and dpkg -i opencv.deb so that it will overwrite my opencv-python? I did so and failed to start the steam as expected. Also, I searched for some other replacement for opencv and got the code below : ``` import gi gi.require_version('Gst', '1.0') from gi.repository import Gst, GObject, GLib import numpy as np import time import cv2 import sys # Initialize GStreamer Gst.init(None) # Define the RTSP stream URL stream_url = 'rtsp://192.168.0.201:554/live0' # Create a GStreamer pipeline pipeline = Gst.parse_launch(f"rtspsrc location={stream_url} latency=0 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink") # Start the pipeline pipeline.set_state(Gst.State.PLAYING) # Main loop to read frames from pipeline and publish them to MPA frame_cntr = 0 while True: # Retrieve a frame from the pipeline sample = pipeline.get_by_name('appsink0').emit('pull-sample') buffer = sample.get_buffer() result, info = buffer.map(Gst.MapFlags.READ) if result: # Convert the frame to numpy array data = np.ndarray((info.size,), dtype=np.uint8, buffer=info.data) frame = np.reshape(data, (640, 720, 3)) # Increment frame counter frame_cntr += 1 sys.stdout.write("\r") sys.stdout.write(f'got frame {frame_cntr} with dims {frame.shape}') sys.stdout.flush() # Publish the frame to MPA cv2.imshow("frame", frame) else: print("Error mapping buffer") # Delay for a short period to control frame rate if cv2.waitKey(30) == ord('q'): break cv2.destroyAllWindows() # Stop the pipeline pipeline.set_state(Gst.State.NULL) ``` And I saw image like this: ![Screenshot from 2024-03-22 17-55-03.png](https://forum.modalai.com/assets/uploads/files/1711101569814-screenshot-from-2024-03-22-17-55-03.png) Which looks like a decode error, and I will look into it next week. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-22 14:38:17 UTC (in reply to Ethan Wu) @Ethan-Wu , yes the installation of opencv .deb should be done use `dpkg -i `. When you installed the newer version of Gstreamer, did you overwrite the existing one or did the new version go into another location? it seems the new version of Gstreamer is not functioning correctly. You should be able to test the gstreamer using command line pipeline like this (without python) and just dump the frames to stdout: ``` gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8900/live latency=0 ! queue ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! filesink location=/dev/stdout ``` If that is not working, then you may want to reflash VOXL2 back to original state, i am not sure how to recover the original gstreamer. ## Reply by Ethan Wu · 2024-03-25 09:22:17 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, thank you for your assistance over the past few days. I am now able to successfully stream videos from Trip to Voxl and QGC using the code I provided earlier, combined with pympa. The decoding issue I mentioned occurred because the video streamed from Trip2 was in YUV420 format, but I reshaped the data in RGB format. As for the original method using OpenCV + GStreamer, I am still unsure where the problem lies. It could be a decoding issue or an installation problem with OpenCV and GStreamer. Additionally, there were many unexpected situations related to GStreamer during the research process that have not yet been resolved. If I manage to identify the source of the problem, I will provide further information. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-25 15:13:23 UTC (in reply to Ethan Wu) @Ethan-Wu , thanks for the update! I am glad you got it working. The best way to diagnose the gstreamer issue would be to re-install VOXL2 SDK and see if the original gstreamer has any issues. As I mentioned, in my testing, i have not seen your errors, and i was able to receive and decode streams, the only issue i saw was the stream sometimes stops, depending on the rtsp source. This must be a gstreamer run-time / pipeline configuration issue , i suspect that something just times out and connection is dropped. If i find how to resolve it, will follow up here. ## Reply by anghung · 2024-06-21 09:56:55 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev said in [Seeking Reference Code for MPA Integration with RTSP Video Streams for TFLite Server](/post/15593): > @Ethan-Wu , I uploaded the opencv deb here : https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-opencv_4.5.5-3_arm64.deb > > it was built using this branch of voxl-opencv project : https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv/-/tree/add-python3-bindings/ > > You can just install that deb to overwrite your existing opencv installation and try the python scripts for rtsp natively on voxl2 > > By the way, how do you get gstreamer to output that debug info? does it automatically print it if there is an issue? @Alex-Kushleyev I see the add-python3-bindings has new commit. could u rebuilt the deb? I cant build the package with gstreamer. Thanks! ![a680d020-8a1f-4679-bdd0-bfa2ab01013a-image.png](https://forum.modalai.com/assets/uploads/files/1718963715561-a680d020-8a1f-4679-bdd0-bfa2ab01013a-image.png) ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-21 14:26:59 UTC (in reply to anghung) @anghung , sure, I can re-build it. I will talk to the team about making this updated opencv build included into main SDK. Meanwhile, the only change was creating the directory, so you can just manually do this before installing the current deb (from the link above) on voxl2: ``` mkdir -p /usr/lib/python3.6/dist-packages/ ``` ## Reply by anghung · 2024-06-22 14:53:56 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev yes, this is same as i install that package. ![05d5b1ba-7c83-43d2-a259-3da50d81c201-image.png](https://forum.modalai.com/assets/uploads/files/1719068045851-05d5b1ba-7c83-43d2-a259-3da50d81c201-image.png) ## Reply by anghung · 2024-06-22 15:10:33 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev For the original topic, integrate RTSP stream for tflite server. I found that the problem is the sample code u provide, in metadata, format is RGB. However, in tflite, https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L305 it only handle NV12 NV21 YUV422 RAW8. Thus, i need to make the RTSP stream NV12, than tflie can take the rest of the job. ``` ! ... ! ... ! videoconvert ! video/x-raw,format=NV12 ! appsink ``` Are there proper way to handle such format problem? Because NV12 got into tflite, it still need to resize to RGB format. Thanks! ## Reply by anghung · 2024-06-24 09:28:12 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, I try to use HW based decoder with HW resize. I also try SW decoding, cv2.VideoCapture(stream,cv2.CAP_GSTREAMER) doesn't workout. ![211180f6-5ea1-497f-879e-6c63c289f4ef-image.png](https://forum.modalai.com/assets/uploads/files/1719219593612-211180f6-5ea1-497f-879e-6c63c289f4ef-image.png) But in protal, it cannot show anything. ![a0e90a4f-a02b-4638-801d-03c9097c2dbc-image.png](https://forum.modalai.com/assets/uploads/files/1719219694969-a0e90a4f-a02b-4638-801d-03c9097c2dbc-image.png) The following is using gi to launch gstreamer and upscale 720 to 1080. ![c1186682-a7d0-43e4-81c1-9e4f366cfca7-image.png](https://forum.modalai.com/assets/uploads/files/1719221200452-c1186682-a7d0-43e4-81c1-9e4f366cfca7-image.png) ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-24 15:27:01 UTC (in reply to anghung) @anghung , I believe the issue is with telling `appsink` the correct format that you need. Looking at opencv source code (https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_gstreamer.cpp#L1000), you can see the supported formats. Please try inserting `! video/x-raw, format=NV12 !` between `autovideoconvert` and `appsink`. Let me know if that works! Edit: it seems you may have already tried it, if so, did it work? I could not understand the exact problem - do you need both NV12 and RGB? Alex ## Reply by anghung · 2024-06-25 04:34:55 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev The code in voxl-tflite-server inference-helper, it only handle NV12 YUV422 NV21 and RAW8. But when rtsp got in to opencv frame, original it is BGR format, than I use video/x-raw,format=NV12 videoconvert. So the tflite-server can handle the mpa. However in tflite-server, the NV12 will be convert to RGB. Is there something that can make this process more efficient? Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-25 15:52:26 UTC (in reply to anghung) @anghung, if you would like to try, you can add a handler for RGB type image here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L282 and publish RGB from the python script. It should be simple enough to try it, based on other examples of using YUV and RAW8 images in the same file. However, then you will be sending more data via MPA (RGB is 3 bytes per pixel, YUV is 1.5 bytes per pixel). Sending more data via MPA should be more efficient than doing unnecessary conversions between YUV and RGB. So, if you can get the video capture to return RGB (not BGR), you could publish the RGB from the python script and use it directly with modified `inference_helper.cpp`. Alex ## Reply by anghung · 2024-06-26 01:33:53 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Okay, I would try that. Also I'm thinking about implement rtsp to mpa in cpp. Is there some guidance? Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-26 16:38:04 UTC (in reply to anghung) @anghung , you want to implement rtsp directly in cpp, you don't need mpa. You can still use opencv to receive the rtsp stream . You should be able to use `cv::VideoCapture` class to get the rtsp frames decoded into your cpp application, just like you did in the python example. Then, you can use opencv to change the format of the image, if needed, or just request the correct format using the gstreamer pipeline when you create the `VideoCapture` instance. Finally, use the resulting image for processing. You can start by modifying the `voxl-tflite-server` to subscribe to the rtsp stream (as an option, instead of mpa) or just make your new application based on `voxl-tflite-server`. This would be a nice feature, if you get it working and would like to contribute it back :) Alex ## Reply by anghung · 2024-06-27 13:14:24 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Of course, I have been use `cv::VideoCapture` in my laptop for testing. Because I am on a business trip, so I asked a colleague to make modifications in ` voxl-tflite-server ` and conduct some tests. However, the results did not match our expectations. As you mention, this function https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L282 should be able to handle RGB format directly. So we just add another condition in `switch` to prevent the function `return false`. ![2df0259f-7fbf-4e38-8487-b07e7afda545-image.png](https://forum.modalai.com/assets/uploads/files/1719493172891-2df0259f-7fbf-4e38-8487-b07e7afda545-image.png) We send RGB format to MPA and the video is fine in Portal, but tflite cant handle it properly. The journal message show that the tflite service restart repearly. ![34b022d3-cfd3-439f-adaf-82cf67b8a15d-image.png](https://forum.modalai.com/assets/uploads/files/1719493372839-34b022d3-cfd3-439f-adaf-82cf67b8a15d-image.png) The new `voxl-tflite-server` we built can handle NV12 correctly as the old one, regarding this situation, what methods should be used to provide more detailed information? We are happy to contribute our successfully tested code. However, we have a limited number of VOXL2 boards available for development, as some are currently in the R&A process. We have already planned to purchase more VOXL2 boards to meet our development needs. Thanks. ## Reply by anghung · 2024-07-01 07:16:37 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I have checked `inference_helper` and modify to handle RGB format successfully. ![Screenshot 2024-07-01 113145.png](https://forum.modalai.com/assets/uploads/files/1719806154374-screenshot-2024-07-01-113145.png)