crop image for voxl-tflite-server
-
I'd like to crop my 1024x768 image so that voxl-tflite-server only sees a square 256x256 at the center, is this possible? Either in voxl-camera-server or voxl-tflite-server?
-
@KnightHawk06 There may be a way to do it using MISP, since it supports zoom, so you could set up output resolution of 256x256 and zoom in to the desired level. I would need to know more details about the camera that is being used, etc and SDK version you are using. Can you share the camera entry in the
voxl-camera-server.confthat you are currently using? Also please let me know which streams from that camera you are actually using (preview, encoded, etc). Some may be enabled but not used.Alex
-
I'm using 'hires_front_small_color' as the input_pipe for voxl-tflite-server.
/** * voxl-camera-server Configuration File * * Each camera has configurations for up to 4 HAL3 streams: * - `preview` stream for raw unprocessed images from CV cameras * - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming * - `large_video` 4k (ish) h264/h265 for onboard video recording to disk * - `snapshot` ISP-processed JPG snapshots that get saved to disk * * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled * * This file is generated from default values by voxl-configure-cameras. * Do not expect arbitrary resolutions to work, the ISP and video compression * pipelines only support very specific resolutions. * * The default video compression mode is cqp or Constant Quantization Parameter * * * */ { "version": 0.1, "fsync_en": true, "fsync_gpio": 109, "cameras": [{ "type": "ar0144", "name": "tracking_front", "enabled": true, "camera_id": 0, "fps": 5, "en_rotate": true, "en_preview": true, "preview_width": 1280, "preview_height": 800, "en_raw_preview": true, "en_misp": true, "misp_width": 1280, "misp_height": 800, "misp_venc_enable": false, "misp_venc_mode": "h264", "misp_venc_br_ctrl": "cqp", "misp_venc_Qfixed": 30, "misp_venc_Qmin": 15, "misp_venc_Qmax": 50, "misp_venc_nPframes": 29, "misp_venc_mbps": 2, "misp_venc_osd": false, "misp_awb": "auto", "ae_mode": "lme_msv", "gain_min": 54, "gain_max": 8000, "ae_desired_msv": 60, "exposure_min_us": 20, "exposure_max_us": 12000, "exposure_soft_min_us": 5000, "ae_filter_alpha": 0.600000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 1, "misp_gamma": 1 }, { "type": "imx412", "name": "hires_front", "enabled": true, "camera_id": 1, "fps": 30, "en_preview": false, "preview_width": 640, "preview_height": 480, "en_raw_preview": false, "en_small_video": true, "small_video_width": 1024, "small_video_height": 768, "small_venc_mode": "h264", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "small_venc_osd": false, "en_large_video": false, "large_video_width": 4056, "large_video_height": 3040, "large_venc_mode": "h264", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 40, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 40, "large_venc_osd": false, "en_snapshot": true, "en_snapshot_width": 4056, "en_snapshot_height": 3040, "exif_focal_length": 3.0999999046325684, "exif_focal_length_in_35mm_format": 17, "exif_fnumber": 1.2400000095367432, "ae_mode": "isp", "gain_min": 54, "gain_max": 8000, "snapshot_jpeg_quality": 75 }, { "type": "imx412", "name": "hires_down", "enabled": false, "camera_id": 2, "fps": 30, "en_preview": false, "preview_width": 640, "preview_height": 480, "en_raw_preview": false, "en_small_video": true, "small_video_width": 1024, "small_video_height": 768, "small_venc_mode": "h264", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "small_venc_osd": false, "en_large_video": false, "large_video_width": 4056, "large_video_height": 3040, "large_venc_mode": "h264", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 40, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 40, "large_venc_osd": false, "en_snapshot": true, "en_snapshot_width": 4056, "en_snapshot_height": 3040, "exif_focal_length": 3.0999999046325684, "exif_focal_length_in_35mm_format": 17, "exif_fnumber": 1.2400000095367432, "ae_mode": "isp", "gain_min": 54, "gain_max": 8000, "snapshot_jpeg_quality": 75 }, { "type": "ar0144", "name": "tracking_down", "enabled": true, "camera_id": 3, "fps": 30, "en_rotate": false, "en_preview": true, "preview_width": 1280, "preview_height": 800, "en_raw_preview": true, "en_misp": true, "misp_width": 1280, "misp_height": 800, "misp_venc_enable": false, "misp_venc_mode": "h264", "misp_venc_br_ctrl": "cqp", "misp_venc_Qfixed": 30, "misp_venc_Qmin": 15, "misp_venc_Qmax": 50, "misp_venc_nPframes": 29, "misp_venc_mbps": 2, "misp_venc_osd": false, "misp_awb": "auto", "ae_mode": "lme_msv", "gain_min": 54, "gain_max": 8000, "ae_desired_msv": 60, "exposure_min_us": 20, "exposure_max_us": 12000, "exposure_soft_min_us": 5000, "ae_filter_alpha": 0.600000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 1, "misp_gamma": 1 }] }/** * This file contains configuration that's specific to voxl-tflite-server. * * skip_n_frames - how many frames to skip between processed frames. For 30Hz * input frame rate, we recommend skipping 5 frame resulting * in 5hz model output. For 30Hz/maximum output, set to 0. * model - which model to use. Currently support mobilenet, fastdepth, * posenet, deeplab, and yolov5. * input_pipe - which camera to use (tracking, hires, or stereo). * delegate - optional hardware acceleration: gpu, cpu, or nnapi. If * the selection is invalid for the current model/hardware, * will silently fall back to base cpu delegate. * allow_multiple - remove process handling and allow multiple instances * of voxl-tflite-server to run. Enables the ability * to run multiples models simultaneously. * output_pipe_prefix - if allow_multiple is set, create output pipes using default * names (tflite, tflite_data) with added prefix. * ONLY USED IF allow_multiple is set to true. */ { "skip_n_frames": 0, "model": "/usr/bin/dnn/yolov5_float16_quant.tflite", "input_pipe": "/run/mpa/hires_front_small_color/", "delegate": "gpu", "requires_labels": true, "labels": "/usr/bin/dnn/yolov5_labels.txt", "allow_multiple": false, "output_pipe_prefix": "mobilenet" } -
@KnightHawk06, if you just wanted to crop, perhaps the most flexible solution would be a simple mpa client that subscribes to an image stream from camera server and outputs the cropped stream , which you can feed into voxl-tflite-server. If you are ok with running an extra process for this, i can help with the example app. The size of images is small enough that overhead for image transport will be minimal.
I can make an app that will crop / resize the input image to produce a new image.
Alex
-
@Alex-Kushleyev Yeah, I think that would work just fine. The other option I was considering is modifying voxl-mavlink-server so that the user could specify a region of the incoming image to consider before detection. Which path do you recommend? Appreciate your help.
-
Lets start with a standalone app doing a crop and after that I can look into adding an option to camera server to output a cropped image directly (in addition to the full one).
I should be able to put together the crop app pretty quickly, will follow up later today.
Alex
-
I made an app that subscribes to an input pipe, crops the specified region and publishes the output image (works for RAW8 and YUV that is coming out of camera server).
It is currently on a branch with some other tools, but you can use that branch to build the app. Please let me know if you need help doing that. You can also compile the app directly on voxl2 using the following command:
g++ -O2 -Wall voxl-image-repub.cpp -o voxl-image-repub -L/usr/lib64 -lmodal_pipe -lmodal_jsonusage:
./voxl-image-repub ERROR: Pipe name not specified Re-publish cropped camera frames (RAW8 or YUV) Options are: -x, --crop-offset-x crop offset in horizontal dimension -y, --crop-offset-y crop offset in vertical dimension -w, --crop-size-x crop size in horizontal dimension (width) -h, --crop-size-y crop size in vertical dimension (height) -u, --usage print this help message The cropped image will be centered if the crop offsets are not provided. typical usage: /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 --crop-offset-x 128 --crop-offset-y 128In the near future, I think i will extend the app the do the following things:
- change FPS (drop every n'th frame)
- resize with interpolation (probably using OpenCV)
Let me know if this app works for you.
Alex
-
@Alex-Kushleyev this is exactly what I need. Thanks! I was able to reconfigure voxl-tflite-server to use the new cropped image data.
