# Onboard Image Processing using ROS + OpenCV (+ possibly some ML library in the future) Source: https://forum.modalai.com/topic/3540/onboard-image-processing-using-ros-opencv-possibly-some-ml-library-in-the-future Category: VOXL SDK and Software (https://forum.modalai.com/category/47/voxl-sdk-and-software) Tags: ros Posted: 2024-06-11 19:07:16 UTC by Prabhav Gupta Replies: 18 · Views: 4798 ## Prabhav Gupta · 2024-06-11 19:07:16 UTC Hello, I am working with a Flight RB5 Drone and I am faced with a challenge. I have installed ROS Melodic on the drone, and wrote a basic script that subscribes to the camera feed published by the drone when we ececute ``` roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch ``` The issue I am facing comes up at the time of building the package using ```catkin_make``` I am getting the following error: ``` rb5:~/catkin_ws$ catkin_make Base path: /home/root/catkin_ws Source space: /home/root/catkin_ws/src Build space: /home/root/catkin_ws/build Devel space: /home/root/catkin_ws/devel Install space: /home/root/catkin_ws/install #### #### Running command: "make cmake_check_build_system" in "/home/root/catkin_ws/build" #### -- Using CATKIN_DEVEL_PREFIX: /home/root/catkin_ws/devel -- Using CMAKE_PREFIX_PATH: /opt/ros/melodic -- This workspace overlays: /opt/ros/melodic -- Found PythonInterp: /usr/bin/python2 (found suitable version "2.7.17", minimum required is "2") -- Using PYTHON_EXECUTABLE: /usr/bin/python2 -- Using Debian Python package layout -- Using empy: /usr/bin/empy -- Using CATKIN_ENABLE_TESTING: ON -- Call enable_testing() -- Using CATKIN_TEST_RESULTS_DIR: /home/root/catkin_ws/build/test_results -- Forcing gtest/gmock from source, though one was otherwise available. -- Found gtest sources under '/usr/src/googletest': gtests will be built -- Found gmock sources under '/usr/src/googletest': gmock will be built -- Found PythonInterp: /usr/bin/python2 (found version "2.7.17") -- Using Python nosetests: /usr/bin/nosetests-2.7 -- catkin 0.7.29 -- BUILD_SHARED_LIBS is on -- BUILD_SHARED_LIBS is on -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- ~~ traversing 2 packages in topological order: -- ~~ - test_pkg -- ~~ - obj_det -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- +++ processing catkin package: 'test_pkg' -- ==> add_subdirectory(test_pkg) -- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy CMake Warning at /home/root/catkin_ws/build/test_pkg/cmake/test_pkg-genmsg.cmake:3 (message): Invoking generate_messages() without having added any message or service file before. You should either add add_message_files() and/or add_service_files() calls or remove the invocation of generate_messages(). Call Stack (most recent call first): /opt/ros/melodic/share/genmsg/cmake/genmsg-extras.cmake:307 (include) test_pkg/CMakeLists.txt:74 (generate_messages) -- test_pkg: 0 messages, 0 services -- +++ processing catkin package: 'obj_det' -- ==> add_subdirectory(obj_det) -- Could NOT find cv_bridge (missing: cv_bridge_DIR) -- Could not find the required component 'cv_bridge'. The following CMake error indicates that you either need to install the package with the same name or change your environment so that it can be found. CMake Error at /opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:83 (find_package): Could not find a package configuration file provided by "cv_bridge" with any of the following names: cv_bridgeConfig.cmake cv_bridge-config.cmake Add the installation prefix of "cv_bridge" to CMAKE_PREFIX_PATH or set "cv_bridge_DIR" to a directory containing one of the above files. If "cv_bridge" provides a separate development package or SDK, be sure it has been installed. Call Stack (most recent call first): obj_det/CMakeLists.txt:10 (find_package) -- Configuring incomplete, errors occurred! See also "/home/root/catkin_ws/build/CMakeFiles/CMakeOutput.log". See also "/home/root/catkin_ws/build/CMakeFiles/CMakeError.log". Makefile:698: recipe for target 'cmake_check_build_system' failed make: *** [cmake_check_build_system] Error 1 Invoking "make cmake_check_build_system" failed ``` Going through this error, I can identify that the cv_bridge package is missing from my ROS installation. On attempting to fix that using ```sudo apt-get install ros-melodic-cv-bridge```, I get the following error: ``` Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: ros-melodic-cv-bridge : Depends: python-opencv but it is not going to be installed E: Unable to correct problems, you have held broken packages. ``` BUT, when I check for cv2 in the python interpretor, it is installed: ``` rb5:~/catkin_ws$ python3 Python 3.6.9 (default, Mar 10 2023, 16:46:00) [GCC 8.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import cv2 >>> cv2.__version__ '4.10.0' >>> ``` This is the part that is most confusiong to me. Is it a version mismatch? Is it a completely different package that the apt is looking for when it says 'python-opencv'? How do I fix this. My aim while doing this is as follows: I want to access the Drone's camera(s) via ROS, run image processing on the video data received, and subsquently publish meaningful results obtained from the image processing to another ROS topic. If there is another alternate approach that I should probably try, I am open to suggestions. Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-11 21:49:29 UTC @Prabhav-Gupta , there is a custom `voxl-opencv` package that is probably installed and may conflict with your opencv for ROS installation. https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv In order to avoid conflicts like that you may want to consider running your ROS stack inside a standard ubuntu docker image, so that you have full control of the packages that you use. uninstalling `voxl-opencv` may break dependencies for other `voxl-` packages. Alex ## Reply by Prabhav Gupta · 2024-06-12 17:38:56 UTC (in reply to Alex Kushleyev) Hello @Alex-Kushleyev, unfortunately, i did end up uninstalling the ```voxl-opencv``` and some other ```voxl-``` packages in an attempt at installing opencv on the native OS. Haven't faced any issues up until now. Will ask them here in case I need some help. (Just to be clear, ROS+OpenCV still isn't working as expected because of differeing Python version. As it turns out, Melodic uses python2 and I cv_bridge in ROS uses python3). Thanks a lot! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-12 19:57:31 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta , if you are still having issues with ROS+OpenCV, then as I previously suggested, it may be better to go the Docker route and start fresh. You can even grab the official ROS docker images which already have everything installed, including opencv. Alex ## Reply by Prabhav Gupta · 2024-06-20 16:37:20 UTC (in reply to Alex Kushleyev) Hello @Alex-Kushleyev, I have an update on this (and of course an issue as well). Current setup: RB5 Drone running ROS Melodic connected to a wifi network. I am runnnig ```roslaunch voxl_mpa_to_ros voxl_mpa_to_ros.launch``` to make the topics visible on the network. My laptop (ROS Noetic) connected to the same network with ROS networking set properly. I am able to list out the topics as well on the laptop with ```rostopic list```. I wrote a simple script that subscribes to the camera topics and simply displays them on my laptop using opencv. ``` #!/usr/bin/env python3 # Import the necessary libraries import rospy # Python library for ROS from sensor_msgs.msg import Image # Image is the message type from cv_bridge import CvBridge # Package to convert between ROS and OpenCV Images import cv2 # OpenCV library from ultralytics import YOLO import torch print("Imports done, loading model...") model = YOLO('/home/prabhav/Desktop/IISc/yolov8m.pt') print("Model loaded!") def callback(data): # Used to convert between ROS and OpenCV images #print("Bridge Created") br = CvBridge() # Output debugging information to the terminal rospy.loginfo("receiving video frame") # Convert ROS Image message to OpenCV image #print("Converting imgmsg to cv2") current_frame = br.imgmsg_to_cv2(data) #print("Running model on the image") #results = model.predict(current_frame, conf=0.5) #print(results) # Display image cv2.imshow("camera", current_frame) cv2.waitKey(1) def receive_message(): # Tells rospy the name of the node. # Anonymous = True makes sure the node has a unique name. Random # numbers are added to the end of the name. print("Initialization done") rospy.init_node('video_sub_py', anonymous=True) # Node is subscribing to the video_frames topic print("Subscriber Created") rospy.Subscriber('/hires_small_color', Image, callback) # spin() simply keeps python from exiting until this node is stopped print("Spinning") rospy.spin() # Close down the video stream when done cv2.destroyAllWindows() if __name__ == '__main__': receive_message() ``` Here is the issue I am facing: When I subscribe to a color image, say ```/hires_small_color```, and try to display it using the above code, i get the followingn error: ``` [INFO] [1718900460.696906]: receiving video frame [ERROR] [1718900460.710019]: bad callback: Traceback (most recent call last): File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/home/prabhav/catkin_ws/src/image_subscriber/scripts/image_subscriber.py", line 37, in callback cv2.imshow("camera", current_frame) cv2.error: OpenCV(4.10.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::::NONE; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]' > Invalid number of channels in input image: > 'VScn::contains(scn)' > where > 'scn' is 2 ``` On going through the error, I understand that for some reason, only two color channels (out of the three required) are present. What I don't get is why this may be happening. I am able to view the video feed form that camera on the web portal as well normally. (http://) At the same time, subscribing to a monochrome image simply to display it does not come up with any errors. The video feed also runs properly with opencv. Any help would be greatly appreciated. Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-20 17:05:59 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta , When you run `voxl-inspect-cam -a`, what format is your `hires_small_color` ? it is probably YUV (NV12), which has two planes (Y plane and UV plane, not 3 interleaved channels like RGB). If you look at the source code of voxl mpa to ros : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-to-ros/-/blob/master/catkin_ws/src/src/interfaces/camera_interface.cpp?ref_type=heads#L100, the YUV NV12 format is published as is, not converted to RGB or BGR. You should look into whether it is possible to convert from NV12 to RGB either in ROS or opencv. Alternatively you could use the compressed stream (h264/h265) which voxl-mpa-to-ros also publishes and it will save you a lot of network bandwidth. You could adjust the compression quality to get the bit rate you need. Alex ## Reply by Prabhav Gupta · 2024-06-20 17:22:33 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev, Yes, it is is infact NV12. I will look into conversion from NV12 to RGB/BGR. When I try to subscribe to the compressed stream, the subscriber (on the laptop) hangs up and the publisher (drone) gives me this error message: ``` Interface hires_small_color now publishing Interface hires_small_color ceasing to publish Interface hires_small_encoded now publishing terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc [voxl_mpa_to_ros_node-2] process has died [pid 2872, exit code -6, cmd /opt/ros/melodic/lib/voxl_mpa_to_ros/voxl_mpa_to_ros_node __name:=voxl_mpa_to_ros_node __log:=/home/root/.ros/log/391f97e8-2f20-11ef-bc4c-00037f125fa6/voxl_mpa_to_ros_node-2.log]. log file: /home/root/.ros/log/391f97e8-2f20-11ef-bc4c-00037f125fa6/voxl_mpa_to_ros_node-2*.log ``` after which I have to restart the publisher on the drone. In the meantime, I will look into the NV12 to RGB conversion. Thanks a lot for the quick response! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-20 17:29:36 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta Hmm, interesting regarding the compressed publishing issue. I have not tried it myself. Perhaps there is a bug there.. Yes, please check if you can do the conversion, but if you are using high resolution images, just keep in mind that streaming them across the network is very slow. You can do some sanity checking to calculate the bandwidth requirements: - lets assume 1920x1080 YUV (NV12), 30 fps - 1920 * 1080 * 1.5 = 3110400 (3.1 MB per frame) . `1.5` because there Y plane is full size and UV plane is half size - 30fps will be 93MB/s, which is almost the maximum of a 1Gbit network link. - if you try to stream in RGB format, the bandwidth requirement will be double that (3 bytes per pixel instead of 1.5) If you still want to use uncompressed format, you may want to throttle the publishing on the VOXL side (in voxl mpa to ros tools) ## Reply by Prabhav Gupta · 2024-06-20 17:41:14 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thanks a lot! I will look into this. ## Reply by Prabhav Gupta · 2024-06-20 19:19:04 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I tried converting the color space as well, but it turns up (essentially) the same error: ``` [INFO] [1718910969.334643]: receiving video frame [ERROR] [1718910969.365757]: bad callback: Traceback (most recent call last): File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback cb(msg) File "/home/prabhav/catkin_ws/src/image_subscriber/scripts/image_subscriber.py", line 31, in callback rgb_frame = cv2.cvtColor(current_frame, cv2.COLOR_YUV2RGB) cv2.error: OpenCV(4.10.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = cv::impl::::NONE; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]' > Invalid number of channels in input image: > 'VScn::contains(scn)' > where > 'scn' is 2 ``` Apart from cv2.COLOR_YUV2RGB, I tried YUV2RGB_NV12 as well, and that too comes up with the same error. Any ideas? Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-20 20:37:56 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta , Hmm, it is possible that the yuv image is not sent out correctly from mpa to ros. I will need to check with my colleagues. BTW are you able to view the yuv image using ros tools (using `image_view`) or something like that? just display the image in a window on your remote machine. Alex ## Reply by Prabhav Gupta · 2024-06-21 15:57:50 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Yes, I am able to display it on my remote machine using ```rosrun rqt_image_view rqt_image_view```, and then selecting the corrosponding topic. I checked OpenCV's documentation yesterday. As far as I can understand, it expects the image in a 2 channel format itself when it is in YUV and "should" be able to convert it to RGB. Seems like this is more of an issue from OpenCV's side. Any suggestions on what I could possibly do? I refered these links: https://docs.opencv.org/4.x/de/d25/imgproc_color_conversions.html https://docs.opencv.org/4.x/d8/d01/group__imgproc__color__conversions.html#gga4e0972be5de079fed4e3a10e24ef5ef0a86027a131a58ef888873fafd4eb603d0 Thanks! ## Reply by Prabhav Gupta · 2024-06-24 16:23:08 UTC (in reply to Alex Kushleyev) Hi @Alex-Kushleyev, Any ideas on what I can do to get the desired output. I did end up asking this question on Stack Overflow as this seemed to be a more OpenCV related question than an issue with the RB5 drone. You may refer to it [here](https://stackoverflow.com/questions/78654180/unable-to-convert-yuv-to-rgb-using-opencv-and-python). Any help would be greatly appreciated. Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-24 21:21:44 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta , yes it seems like OpenCV and ROS YUV_NV12 formats do not match up. I will take a look at it.. it seems the ROS YUV is packed (interleaved) while standard for storing YUV NV12 is having two planes : plane 1 : Y (size: width*height), plane 2 : UV (size: width*height/2) In the meantime.. you can stream a rtsp h264/h265 from VOXL (use decent quality so that image looks good) and use opencv to receive the stream and get decompressed images: https://stackoverflow.com/questions/40875846/capturing-rtsp-camera-using-opencv-python Would that work for you ? (unfortunately with rtsp stream, you will not get the full image metadata, like exposure, gain, timestamp, etc). RTSP streaming can be done using `voxl-streamer`, which can accept either a YUV (and encode it) or already encoded h264/5 stream from `voxl-camera-server`. Alex ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-24 21:35:50 UTC (in reply to Alex Kushleyev) Here is a link that may provide a clue about the array dimensions for the YUV image : https://answers.opencv.org/question/218407/is-yuv2bgr_nv12-conversion-necessary-to-imshow-an-yuv-image/ ``` YUV = Mat(H+H/2, W, CV_8UC1, pointerToData); // YUV Mat initialization ``` Please check the dimensions of the image of the data in the ROS callback. Alex ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-25 15:46:11 UTC (in reply to Alex Kushleyev) Here is another example of converting YUV_NV12 (published via mpa) into RGB. https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L309 However, i should mention, that when published via MPA, the YUV image has two planes (Y plane, which has dimensions of the whole image and UV plane, which has dimensions of width,height/2). The Y and UV planes are not interleaved. Based on the following code, it seems ROS expects the YUV_NV12 to be interleaved Y and UV : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-to-ros/-/blob/master/catkin_ws/src/src/interfaces/camera_interface.cpp?ref_type=heads#L132 So if you are using ROS transport, it seems, you may need to separate the Y and UV planes before converting to RGB / BGR using opencv.. Alex ## Reply by Prabhav Gupta · 2024-06-25 16:28:05 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I tried using the ```voxl-streamer``` and it works with OpenCV imshow atleast. I am able to view the camera feed on my remote machine. Thanks a lot for this suggestion. I believe I can work around this. Thanks! ## Reply by Prabhav Gupta · 2024-06-25 16:29:01 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thanks a lot for these resources. I will look into the links mentioned and get back. Thanks again! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-26 16:33:17 UTC (in reply to Prabhav Gupta) @Prabhav-Gupta , you are welcome!