RAW10 output for ov9782
-
Hello, I have a voxl 2 mini with 1 ov9782 on j7 on sdk 1.1.2 and I was wondering if is it possible to get RAW 10 bit image from the sensor over the modal pipe through the voxl-camera-server?
Not exactly, but close to that. if you already have the camera set up, add an entry
"en_raw_preview": true,such as:"cameras": [ { "type": "ov9782", ... "en_preview": true, "preview_width": 1280, "preview_height": 800, "en_raw_preview": true, ...When you run
voxl-camera-serverwith this configuration, you will see a image sent out<camera_name>_bayer, which is almost what you want, but it is RAW8 format, with the 2 LSB bits stripped off.If you REALLY need to the 10 bit RAW image, there are a few changes that would need be made in the camera server (mainly disabling 10->8bit conversion and sending out the image via MPA with correct type). Let me know..
Keep in mind that OV9782 is a color camera, which means the image is in the Bayer format, and it will need to be debayered to get RGB, but camera server has an example implementation for doing that and it already publishes the color image after software debayering (if you enable the raw preview mode).
Here is a sample output from
voxl-inspect-cam -a(in my case the camera is calledstereo_left).bayeris the raw (converted 10->8bit),greyis debayered into mono,coloris debayered into a rgb image.| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 20.5 | 30.0 | 245.6 | RAW8 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 120 | 23.0 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 21.0 | 30.0 | 245.6 | RAW8Alex
-
Not exactly, but close to that. if you already have the camera set up, add an entry
"en_raw_preview": true,such as:"cameras": [ { "type": "ov9782", ... "en_preview": true, "preview_width": 1280, "preview_height": 800, "en_raw_preview": true, ...When you run
voxl-camera-serverwith this configuration, you will see a image sent out<camera_name>_bayer, which is almost what you want, but it is RAW8 format, with the 2 LSB bits stripped off.If you REALLY need to the 10 bit RAW image, there are a few changes that would need be made in the camera server (mainly disabling 10->8bit conversion and sending out the image via MPA with correct type). Let me know..
Keep in mind that OV9782 is a color camera, which means the image is in the Bayer format, and it will need to be debayered to get RGB, but camera server has an example implementation for doing that and it already publishes the color image after software debayering (if you enable the raw preview mode).
Here is a sample output from
voxl-inspect-cam -a(in my case the camera is calledstereo_left).bayeris the raw (converted 10->8bit),greyis debayered into mono,coloris debayered into a rgb image.| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 20.5 | 30.0 | 245.6 | RAW8 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 120 | 23.0 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 120 | 21.0 | 30.0 | 245.6 | RAW8Alex
-
I do need the actual 10 bit RAW image .so I would appreciate your help in editing the camera server to get that working. Thanks.
OK, let me check how we should handle this and get back to you. Currently the MPA interface does not have a 10 bit raw type.
-
OK, let me check how we should handle this and get back to you. Currently the MPA interface does not have a 10 bit raw type.
-
@Alex-Kushleyev Hi Alex, wondering if you have an update for me on this, thanks.
@mkriesel , i am still discussing with the team how to integrate this long term, however for now i have made a test branch for you to try. You will need to update two packages:
- libmodal_pipe : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/add-raw10-raw12-image-formats
- voxl-camera-server: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/test-raw10-bayer-pub
build instructions:
- First, you need to check out and build the libmodal_pipe and make deb package (
libmodal-pipe_2.10.1_arm64.deb). For your convenience i pre-built it and uploaded here - check out
voxl-camera-serverfrom my branch, entervoxl-cross 2.7) and./install_build_deps.sh qrb5165 dev - place the new
libmodal-pipe_2.10.1-raw10-raw12_arm64.debintovoxl-camera-server/depsfolder - install the new
libmodal-pipedeb inside thevoxl-crossdocker :dpkg -i deps/libmodal-pipe_2.10.1-raw10-raw12_arm64.deb - build
voxl-camera-serverinsidevoxl-crossdocker :./build.sh qrb5165and make package./make_package.sh. For your convenience, I put the deb here - install the new
libmodal-pipedeb (first) on VOXL2 and the newvoxl-camera-serverdeb second.
start / restart
voxl-camera-servertest usingvoxl-inspect-cam -a:| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1280000 | 1280 | 800 | 5.26 | 782 | 21556 | 21.5 | 30.0 | 307.0 | RAW10 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 21556 | 26.9 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 21556 | 25.1 | 30.0 | 245.6 | RAW8dump some raw frames (make sure to run
voxl-inspect-camfirst for a while, so that auto exposure kicks in because the frames don't get processed if there are no clients and if you just grab one first frame, auto exposure won't have time to apply:voxl-record-raw-image stereo_left_bayer -d /home/root/ -n 1The output image is in the MIPI10 bayer format, that is for every 4 pixels, there are 5 bytes in the following format:
[pix0][pix1][pix2][pix3][LSBs_for_pix0-3][pix4][pix5][pix6][pix7][LSBs_for_pix4-7]...That is to say, every 5th byte has 2 LSB bits for each of the previous 4 pixel bytes.
Each Row is 1280 pixels, 1600 bytes (1280/4 *5=1600)
The pixels are in Bayer pattern (RGGB or BGGR depending on the rotation, you should be able to figure it out based on color, once you de-bayer).
Please note that I cannot provide any information (yet) about when this will be integrated into the main release - please test at your own risk
. Hopefully we will be able to add it in the future.I tested only with OV9782 at 1280x800 resolution. Please let me know if this works for you.
Alex
-
@mkriesel , i am still discussing with the team how to integrate this long term, however for now i have made a test branch for you to try. You will need to update two packages:
- libmodal_pipe : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/add-raw10-raw12-image-formats
- voxl-camera-server: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/test-raw10-bayer-pub
build instructions:
- First, you need to check out and build the libmodal_pipe and make deb package (
libmodal-pipe_2.10.1_arm64.deb). For your convenience i pre-built it and uploaded here - check out
voxl-camera-serverfrom my branch, entervoxl-cross 2.7) and./install_build_deps.sh qrb5165 dev - place the new
libmodal-pipe_2.10.1-raw10-raw12_arm64.debintovoxl-camera-server/depsfolder - install the new
libmodal-pipedeb inside thevoxl-crossdocker :dpkg -i deps/libmodal-pipe_2.10.1-raw10-raw12_arm64.deb - build
voxl-camera-serverinsidevoxl-crossdocker :./build.sh qrb5165and make package./make_package.sh. For your convenience, I put the deb here - install the new
libmodal-pipedeb (first) on VOXL2 and the newvoxl-camera-serverdeb second.
start / restart
voxl-camera-servertest usingvoxl-inspect-cam -a:| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | stereo_left_bayer | 1280000 | 1280 | 800 | 5.26 | 782 | 21556 | 21.5 | 30.0 | 307.0 | RAW10 | stereo_left_color | 3072000 | 1280 | 800 | 5.26 | 782 | 21556 | 26.9 | 30.0 | 736.7 | RGB | stereo_left_grey | 1024000 | 1280 | 800 | 5.26 | 782 | 21556 | 25.1 | 30.0 | 245.6 | RAW8dump some raw frames (make sure to run
voxl-inspect-camfirst for a while, so that auto exposure kicks in because the frames don't get processed if there are no clients and if you just grab one first frame, auto exposure won't have time to apply:voxl-record-raw-image stereo_left_bayer -d /home/root/ -n 1The output image is in the MIPI10 bayer format, that is for every 4 pixels, there are 5 bytes in the following format:
[pix0][pix1][pix2][pix3][LSBs_for_pix0-3][pix4][pix5][pix6][pix7][LSBs_for_pix4-7]...That is to say, every 5th byte has 2 LSB bits for each of the previous 4 pixel bytes.
Each Row is 1280 pixels, 1600 bytes (1280/4 *5=1600)
The pixels are in Bayer pattern (RGGB or BGGR depending on the rotation, you should be able to figure it out based on color, once you de-bayer).
Please note that I cannot provide any information (yet) about when this will be integrated into the main release - please test at your own risk
. Hopefully we will be able to add it in the future.I tested only with OV9782 at 1280x800 resolution. Please let me know if this works for you.
Alex
-
@Alex-Kushleyev Hi, I got the chance to test this over the past few days. It working and I haven't run into any issues so far. Thank you very much for helping out this this.
@mkriesel , You are welcome!
Right now we have not planned to add this feature to the official build (not high priority). However, if you later need to update to latest voxl-camera-server and still maintain this functionality, you should be able to just merge the latest changes from camera server into this branch (may resolve a simple conflict if it occurs).
Alex
-
@mkriesel , You are welcome!
Right now we have not planned to add this feature to the official build (not high priority). However, if you later need to update to latest voxl-camera-server and still maintain this functionality, you should be able to just merge the latest changes from camera server into this branch (may resolve a simple conflict if it occurs).
Alex
@Alex-Kushleyev Hi again, I'm now testing similar capability for the OnSemi AR0144 https://docs.modalai.com/M0149/ but looking at the debug logs I see
DEBUG: setting thread affinity for cam hires preview
DEBUG: hires checking to see if it is raw8 or raw 10
DEBUG: Frame was actually 8 bit, sending as is
DEBUG: Value of standby_active is: 0so does this mean that I cannot get 10 bit imagery from this sensor? I see messages like <pre><font color="#839496"><b> </b></font>Allocated BO with width=1280 height=800 stride=1600 aligned_w=1600 aligned_h=800 size=1282048 flags=0x20900 format=GBM_FORMAT_RAW10
</pre> as well. I'm just trying to figure what options I have available for this sensor right now.Thanks!
-
@Alex-Kushleyev Hi again, I'm now testing similar capability for the OnSemi AR0144 https://docs.modalai.com/M0149/ but looking at the debug logs I see
DEBUG: setting thread affinity for cam hires preview
DEBUG: hires checking to see if it is raw8 or raw 10
DEBUG: Frame was actually 8 bit, sending as is
DEBUG: Value of standby_active is: 0so does this mean that I cannot get 10 bit imagery from this sensor? I see messages like <pre><font color="#839496"><b> </b></font>Allocated BO with width=1280 height=800 stride=1600 aligned_w=1600 aligned_h=800 size=1282048 flags=0x20900 format=GBM_FORMAT_RAW10
</pre> as well. I'm just trying to figure what options I have available for this sensor right now.Thanks!
@mkriesel , currently our AR0144 driver is set up to send the image in 8 bit mode, so we don't have to convert 10->8 bit on the cpu. The Sensor is capable of sending 10 bit, but we do not have that mode available right now.
Alex
-
@mkriesel , currently our AR0144 driver is set up to send the image in 8 bit mode, so we don't have to convert 10->8 bit on the cpu. The Sensor is capable of sending 10 bit, but we do not have that mode available right now.
Alex
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login