Seeking Reference Code for MPA Integration with RTSP Video Streams for TFLite Server
-
About the error 'unexpected reference "gst-launch-1" ', I do get the same error when I replace the gstreamer command with gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink, so I think that's not a problem.
-
Hmm. When i tested with a local rtsp stream generated on the same voxl2, the video capture started immediately. The freezing would occur in the python client side sometime after i got a bunch of frames (when using voxl-streamer to produce the rtsp stream). When the freezing happened, it looked like voxl-streamer dropped the client connection for some reason. I will test with a local ip cam that is producing rtsp today.
It seems gstreamer rtsp receiving end may not be running correctly on your end. How about i share with you updated native opencv package with python bindings and you could try running this natively on voxl2 without your environment? This whay you can test what I am testing.
-
I just tried playing a local rtsp stream from an ip cam (using
rtsp_rx_mpa_pub.py
) and it initially starts playing but gets stuck after a few seconds. However, changing the pipeline to use the following (usingavdec_h264
instead ofqtivdec
:.... rtph264depay ! h264parse config-interval=-1 ! avdec_h264 ! autovideoconvert ! appsink
Does not get stuck.
So there must be something about the hw decoder that causes it to get stuck, I will investigate further..
-
@Alex-Kushleyev
Sharing opencv package would be great, and I also want to share how I built opencv that supports gstreamer. I looked into this issue and followed instructions here, and made sure I saw the followings information to get gstreamer working with opencv by calling cv2.getBuildInformation() :Video I/O: DC1394: NO FFMPEG: YES avcodec: YES (57.107.100) avformat: YES (57.83.100) avutil: YES (55.78.100) swscale: YES (4.8.100) avresample: NO GStreamer: YES (1.14.5) v4l/v4l2: YES (linux/videodev2.h)
I was using conda for virtual environment and run opencv with python 3.10 (3.9 and 3.8 are also tested), and my opencv version is 4.9.0.80.
-
@Ethan-Wu , I uploaded the opencv deb here : https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-opencv_4.5.5-3_arm64.deb
it was built using this branch of voxl-opencv project : https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv/-/tree/add-python3-bindings/
You can just install that deb to overwrite your existing opencv installation and try the python scripts for rtsp natively on voxl2
By the way, how do you get gstreamer to output that debug info? does it automatically print it if there is an issue?
-
@Alex-Kushleyev You can set the environment variable GST_DEBUG to the number that fits the debug level you prefer. I made it 2 so that I can see warning and error output from gst-launch-1.0, here's the reference documentation : https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c
-
@Alex-Kushleyev
Hi,
Do I just need to upload it to voxl and dpkg -i opencv.deb so that it will overwrite my opencv-python? I did so and failed to start the steam as expected.Also, I searched for some other replacement for opencv and got the code below :
import gi gi.require_version('Gst', '1.0') from gi.repository import Gst, GObject, GLib import numpy as np import time import cv2 import sys # Initialize GStreamer Gst.init(None) # Define the RTSP stream URL stream_url = 'rtsp://192.168.0.201:554/live0' # Create a GStreamer pipeline pipeline = Gst.parse_launch(f"rtspsrc location={stream_url} latency=0 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink") # Start the pipeline pipeline.set_state(Gst.State.PLAYING) # Main loop to read frames from pipeline and publish them to MPA frame_cntr = 0 while True: # Retrieve a frame from the pipeline sample = pipeline.get_by_name('appsink0').emit('pull-sample') buffer = sample.get_buffer() result, info = buffer.map(Gst.MapFlags.READ) if result: # Convert the frame to numpy array data = np.ndarray((info.size,), dtype=np.uint8, buffer=info.data) frame = np.reshape(data, (640, 720, 3)) # Increment frame counter frame_cntr += 1 sys.stdout.write("\r") sys.stdout.write(f'got frame {frame_cntr} with dims {frame.shape}') sys.stdout.flush() # Publish the frame to MPA cv2.imshow("frame", frame) else: print("Error mapping buffer") # Delay for a short period to control frame rate if cv2.waitKey(30) == ord('q'): break cv2.destroyAllWindows() # Stop the pipeline pipeline.set_state(Gst.State.NULL)
And I saw image like this:
Which looks like a decode error, and I will look into it next week.
-
@Ethan-Wu , yes the installation of opencv .deb should be done use
dpkg -i <package.deb>
.When you installed the newer version of Gstreamer, did you overwrite the existing one or did the new version go into another location?
it seems the new version of Gstreamer is not functioning correctly.
You should be able to test the gstreamer using command line pipeline like this (without python) and just dump the frames to stdout:
gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8900/live latency=0 ! queue ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! filesink location=/dev/stdout
If that is not working, then you may want to reflash VOXL2 back to original state, i am not sure how to recover the original gstreamer.
-
@Alex-Kushleyev
Hi, thank you for your assistance over the past few days. I am now able to successfully stream videos from Trip to Voxl and QGC using the code I provided earlier, combined with pympa. The decoding issue I mentioned occurred because the video streamed from Trip2 was in YUV420 format, but I reshaped the data in RGB format.As for the original method using OpenCV + GStreamer, I am still unsure where the problem lies. It could be a decoding issue or an installation problem with OpenCV and GStreamer. Additionally, there were many unexpected situations related to GStreamer during the research process that have not yet been resolved. If I manage to identify the source of the problem, I will provide further information.
-
@Ethan-Wu , thanks for the update! I am glad you got it working. The best way to diagnose the gstreamer issue would be to re-install VOXL2 SDK and see if the original gstreamer has any issues. As I mentioned, in my testing, i have not seen your errors, and i was able to receive and decode streams, the only issue i saw was the stream sometimes stops, depending on the rtsp source. This must be a gstreamer run-time / pipeline configuration issue , i suspect that something just times out and connection is dropped. If i find how to resolve it, will follow up here.
-
-