# RAW10 output for ov9782 Source: https://forum.modalai.com/topic/3457/raw10-output-for-ov9782 Category: ESCs, Sensors and Accessories (https://forum.modalai.com/category/15/escs-sensors-and-accessories) Tags: image-sensor Posted: 2024-05-20 22:13:09 UTC by mkriesel Replies: 10 · Views: 2084 ## mkriesel · 2024-05-20 22:13:09 UTC Hello, I have a voxl 2 mini with 1 ov9782 on j7 on sdk 1.1.2 and I was wondering if is it possible to get RAW 10 bit image from the sensor over the modal pipe through the voxl-camera-server? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-05-21 04:52:49 UTC @mkriesel , Not exactly, but close to that. if you already have the camera set up, add an entry `"en_raw_preview": true` ,such as: ``` "cameras": [ { "type": "ov9782", ... "en_preview": true, "preview_width": 1280, "preview_height": 800, "en_raw_preview": true, ... ``` When you run `voxl-camera-server` with this configuration, you will see a image sent out `_bayer`, which is almost what you want, but it is RAW8 format, with the 2 LSB bits stripped off. If you REALLY need to the 10 bit RAW image, there are a few changes that would need be made in the camera server (mainly disabling 10->8bit conversion and sending out the image via MPA with correct type). Let me know.. Keep in mind that OV9782 is a color camera, which means the image is in the Bayer format, and it will need to be debayered to get RGB, but camera server has an example implementation for doing that and it already publishes the color image after software debayering (if you enable the raw preview mode). Here is a sample output from `voxl-inspect-cam -a` (in my case the camera is called `stereo_left`). `bayer` is the raw (converted 10->8bit), `grey` is debayered into mono, `color` is debayered into a rgb image. ``` | Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 20.5 | 30.0 | 245.6 | RAW8 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 120 | 23.0 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 21.0 | 30.0 | 245.6 | RAW8 ``` Alex ## Reply by mkriesel · 2024-05-21 17:28:09 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev , I do need the actual 10 bit RAW image .so I would appreciate your help in editing the camera server to get that working. Thanks. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-05-21 20:10:52 UTC (in reply to mkriesel) OK, let me check how we should handle this and get back to you. Currently the MPA interface does not have a 10 bit raw type. ## Reply by mkriesel · 2024-05-30 19:38:17 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi Alex, wondering if you have an update for me on this, thanks. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-04 02:09:05 UTC (in reply to mkriesel) @mkriesel , i am still discussing with the team how to integrate this long term, however for now i have made a test branch for you to try. You will need to update two packages: - libmodal_pipe : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/add-raw10-raw12-image-formats - voxl-camera-server: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/test-raw10-bayer-pub build instructions: - First, you need to check out and build the libmodal_pipe and make deb package (`libmodal-pipe_2.10.1_arm64.deb`). For your convenience i pre-built it and uploaded [here](https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/libmodal-pipe_2.10.1-raw10-raw12_arm64.deb) - check out `voxl-camera-server` from my branch, enter `voxl-cross 2.7`) and `./install_build_deps.sh qrb5165 dev` - place the new `libmodal-pipe_2.10.1-raw10-raw12_arm64.deb` into `voxl-camera-server/deps` folder - install the new `libmodal-pipe` deb inside the `voxl-cross` docker : `dpkg -i deps/libmodal-pipe_2.10.1-raw10-raw12_arm64.deb` - build `voxl-camera-server` inside `voxl-cross` docker : `./build.sh qrb5165` and make package `./make_package.sh` . For your convenience, I put the deb [here](https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-camera-server_1.9.5-raw10_arm64.deb) - install the new `libmodal-pipe` deb (first) on VOXL2 and the new `voxl-camera-server` deb second. start / restart `voxl-camera-server` test using `voxl-inspect-cam -a`: ``` | Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1280000 | 1280 | 800 | 5.26 | 782 | 21556 | 21.5 | 30.0 | 307.0 | RAW10 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 21556 | 26.9 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 21556 | 25.1 | 30.0 | 245.6 | RAW8 ``` dump some raw frames (make sure to run `voxl-inspect-cam` first for a while, so that auto exposure kicks in because the frames don't get processed if there are no clients and if you just grab one first frame, auto exposure won't have time to apply: ``` voxl-record-raw-image stereo_left_bayer -d /home/root/ -n 1 ``` The output image is in the MIPI10 bayer format, that is for every 4 pixels, there are 5 bytes in the following format: ``` [pix0][pix1][pix2][pix3][LSBs_for_pix0-3][pix4][pix5][pix6][pix7][LSBs_for_pix4-7]... ``` That is to say, every 5th byte has 2 LSB bits for each of the previous 4 pixel bytes. Each Row is 1280 pixels, 1600 bytes (1280/4 *5=1600) The pixels are in Bayer pattern (RGGB or BGGR depending on the rotation, you should be able to figure it out based on color, once you de-bayer). Please note that I cannot provide any information (yet) about when this will be integrated into the main release - please test at your own risk :) . Hopefully we will be able to add it in the future. I tested only with OV9782 at 1280x800 resolution. Please let me know if this works for you. Alex ## Reply by mkriesel · 2024-06-07 18:54:58 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi, I got the chance to test this over the past few days. It working and I haven't run into any issues so far. Thank you very much for helping out this this. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-11 19:58:17 UTC (in reply to mkriesel) @mkriesel , You are welcome! Right now we have not planned to add this feature to the official build (not high priority). However, if you later need to update to latest voxl-camera-server and still maintain this functionality, you should be able to just merge the latest changes from camera server into this branch (may resolve a simple conflict if it occurs). Alex ## Reply by mkriesel · 2024-07-10 22:02:16 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Hi again, I'm now testing similar capability for the OnSemi AR0144 https://docs.modalai.com/M0149/ but looking at the debug logs I see DEBUG: setting thread affinity for cam hires preview DEBUG: hires checking to see if it is raw8 or raw 10 DEBUG: Frame was actually 8 bit, sending as is DEBUG: Value of standby_active is: 0 so does this mean that I cannot get 10 bit imagery from this sensor? I see messages like 
 Allocated BO with width=1280 height=800 stride=1600 aligned_w=1600 aligned_h=800 size=1282048 flags=0x20900 format=GBM_FORMAT_RAW10
as well. I'm just trying to figure what options I have available for this sensor right now. Thanks! ## Reply by Alex Kushleyev (ModalAI staff) · 2024-07-11 01:49:35 UTC (in reply to mkriesel) @mkriesel , currently our AR0144 driver is set up to send the image in 8 bit mode, so we don't have to convert 10->8 bit on the cpu. The Sensor is capable of sending 10 bit, but we do not have that mode available right now. Alex ## Reply by mkriesel · 2024-07-11 21:47:05 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Alright, thanks for the info