MP4 video read on VOXl2 to MPA

ravi

I want a utility to video to tflite server.
To test the Tflite models.

For transfering video frames, irefereed modal-pipe examples.

But for reading video files. it seems there are 2 posible methods.
using ffmpeg and opencv.

first i tried to find ffmpeg on voxl2, it is not installed so i tried to compile 'voxl-ffmpeg' as per readme for qrb5165.
But it was getting failed.

then i amd tring opencv to read "mp4" format video.
I am getting error in reading mp4 file using python and opencv.
but same code working on my ubuntu perfectly.

Can help me find the issue.
please find the code and error below.

Thank you.

on voxl2, i am getting below error.

voxl2:~/tests$ python3 video-test2.py
gbm_create_device(156): Info: backend name is: msm_drm
[ WARN:0@3.094] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1374) open OpenCV | GStreamer warning: unable to query duration of stream

(python3:18838): GStreamer-CRITICAL **: 13:10:33.515: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(python3:18838): GStreamer-CRITICAL **: 13:10:33.515: gst_structure_get_int: assertion 'structure != NULL' failed
[ WARN:0@3.095] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1384) open OpenCV | GStreamer warning: cannot query video width/height

(python3:18838): GStreamer-CRITICAL **: 13:10:33.515: gst_structure_get_fraction: assertion 'structure != NULL' failed
[ WARN:0@3.095] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1390) open OpenCV | GStreamer warning: cannot query video fps
[ WARN:0@3.095] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1405) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
Traceback (most recent call last):
  File "video-test2.py", line 33, in <module>
    cv2.destroyAllWindows()
cv2.error: OpenCV(4.5.5) /opt/code/opencv/modules/highgui/src/window.cpp:1262: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvDestroyAllWindows'

voxl2:~/tests$

This is my python test code.

# importing libraries
import cv2

# Create a VideoCapture object and read from input file
cap = cv2.VideoCapture('0122_part_030.MP4')

# Check if camera opened successfully
if (cap.isOpened()== False):
    print("Error opening video file")

# Read until video is completed
while(cap.isOpened()):
    
# Capture frame-by-frame
    ret, frame = cap.read()
    if ret == True:
    # Display the resulting frame
        cv2.imshow('Frame', frame)
        
    # Press Q on keyboard to exit
        if cv2.waitKey(25) & 0xFF == ord('q'):
            break

# Break the loop
    else:
        break

# When everything done, release
# the video capture object
cap.release()

# Closes all the frames
cv2.destroyAllWindows()

Alex Kushleyev

Hi @ravi ,

You can install ffmpeg using apt update && apt install ffmpeg

In your opencv example, when running on voxl2 you need to comment out the following, because display support is not compiled into opencv for voxl2:

cv2.imshow
cv2.waitKey
cv2.destroyAllWindows()

Instead of cv2.imshow, you could put

print('got a frame with dims: ',frame.shape)

I just tried this and it worked (ignore the weird dimensions of the video that i used):

voxl2-mini:~$ python3 video_test.py 
gbm_create_device(156): Info: backend name is: msm_drm
gbm_create_device(156): Info: backend name is: msm_drm
[ WARN:0@0.382] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1374) open OpenCV | GStreamer warning: unable to query duration of stream
[ WARN:0@0.382] global /opt/code/opencv/modules/videoio/src/cap_gstreamer.cpp (1405) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=0, duration=-1
got a frame with dims:  (1088, 3360, 3)
got a frame with dims:  (1088, 3360, 3)

It seems you did have more errors than i did, so perhaps something is missing in opencv.

https://github.com/opencv/opencv/issues/25108 this issue is similar.

How did you build opencv with python3 support? Perhaps you did not include the video support.

Alex

Alex Kushleyev

Also, take a look at the following experimental tool that does almost exactly what you need, except it reads the video from an rtsp stream (and then publishes it using MPA):

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/pympa-experimental/tools/python/rtsp_rx_mpa_pub.py?ref_type=heads

In order to run this, you will need to build voxl-mpa-tools from this branch, install the deb on voxl2, and then copy the scripts you need and run them from /usr/share/modalai/voxl-mpa-tools/.

For testing this, i built voxl-opencv library with python3 support (the official version does not have python3 support).

https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv/-/tree/add-python3-bindings

Building this package takes a very long time on a x86-64 machine using voxl emulator (1hr+ on a 12-core machine). Typically we use voxl cross compiler, but it did not work with python3 support enabled, so it only works using voxl-emulator.

So I actually built it the following way:

use ARM-based Mac (M1)
run Parallels Desktop
create Ubunutu 20.04 ARM64 virtual machine
install Docker in the ubuntu virtual machine
download qrb5165-emulator docker image (from https://developer.modalai.com/asset/)
install the docker image in your ubuntu VM
- docker load -i <docker image archive>
check out the voxl-opencv repo (add-python3-bindings branch)
- git clone https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv.git -b add-python3-bindings
go to the voxl-opencv directory
start emulator docker

docker run -it --rm -v `pwd`:/opt/code -w /opt/code qrb5165-emulator:v1.5 bash

now you should be inside a docker container and your opencv source code mapped to the container, ready to build
./build.sh native
./make_package.sh

(I know you did not ask all of this, but maybe it will be useful)

If you do not want to go through all this trouble :), here is the package built from the above source with python3 bindings enabled : https://storage.googleapis.com/modalai_public/temp/voxl-opencv_4.5.5-3_arm64.deb

Alex

ravi

@Alex-Kushleyev

I appreciate your support and providing corrections.

yes, I will remove cv2.imshow, cv2.waitKey ,cv2.destroyAllWindows().
When I tried I was getting "Error opening video file".

I installed voxl-opencv_4.5.5-3_arm64.deb. you linked in the thread below.

https://forum.modalai.com/topic/3220/seeking-reference-code-for-mpa-integration-with-rtsp-video-streams-for-tflite-server/20

Thanks for the instructions for FFmpeg and OpenCV build.

I will try to install it both and test it again.

Thank you.

Alex Kushleyev

@ravi , sounds good.

The updated opencv deb is the same that i posted a while ago, but i fixed a small issue related to installation of a symlink as part of deb install (which was mentioned in the other thread, i believe)

Alex