RAW10 output for ov9782

mkriesel

Hello, I have a voxl 2 mini with 1 ov9782 on j7 on sdk 1.1.2 and I was wondering if is it possible to get RAW 10 bit image from the sensor over the modal pipe through the voxl-camera-server?

Alex Kushleyev

@mkriesel ,

Not exactly, but close to that. if you already have the camera set up, add an entry "en_raw_preview": true ,such as:

"cameras":	[
		{
			"type":	"ov9782",
			...
			"en_preview":	true,
			"preview_width":	1280,
			"preview_height":	800,
			"en_raw_preview":	true,
                        ...

When you run voxl-camera-server with this configuration, you will see a image sent out <camera_name>_bayer, which is almost what you want, but it is RAW8 format, with the 2 LSB bits stripped off.

If you REALLY need to the 10 bit RAW image, there are a few changes that would need be made in the camera server (mainly disabling 10->8bit conversion and sending out the image via MPA with correct type). Let me know..

Keep in mind that OV9782 is a color camera, which means the image is in the Bayer format, and it will need to be debayered to get RGB, but camera server has an example implementation for doing that and it already publishes the color image after software debayering (if you enable the raw preview mode).

Here is a sample output from voxl-inspect-cam -a (in my case the camera is called stereo_left). bayer is the raw (converted 10->8bit), grey is debayered into mono, color is debayered into a rgb image.

|          Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
|  stereo_left_bayer | 1024000 | 1280 |  800 |  5.26 |  782 |      120 |     20.5  | 30.0 |  245.6 | RAW8
|  stereo_left_color | 3072000 | 1280 |  800 |  5.26 |  782 |      120 |     23.0  | 30.0 |  736.7 | RGB
|   stereo_left_grey | 1024000 | 1280 |  800 |  5.26 |  782 |      120 |     21.0  | 30.0 |  245.6 | RAW8

Alex

mkriesel

@Alex-Kushleyev ,

I do need the actual 10 bit RAW image .so I would appreciate your help in editing the camera server to get that working. Thanks.

Alex Kushleyev

OK, let me check how we should handle this and get back to you. Currently the MPA interface does not have a 10 bit raw type.

mkriesel

@Alex-Kushleyev Hi Alex, wondering if you have an update for me on this, thanks.

Alex Kushleyev

@mkriesel , i am still discussing with the team how to integrate this long term, however for now i have made a test branch for you to try. You will need to update two packages:

libmodal_pipe : https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/tree/add-raw10-raw12-image-formats
voxl-camera-server: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/test-raw10-bayer-pub

build instructions:

First, you need to check out and build the libmodal_pipe and make deb package (libmodal-pipe_2.10.1_arm64.deb). For your convenience i pre-built it and uploaded here
check out voxl-camera-server from my branch, enter voxl-cross 2.7) and ./install_build_deps.sh qrb5165 dev
place the new libmodal-pipe_2.10.1-raw10-raw12_arm64.deb into voxl-camera-server/deps folder
install the new libmodal-pipe deb inside the voxl-cross docker : dpkg -i deps/libmodal-pipe_2.10.1-raw10-raw12_arm64.deb
build voxl-camera-server inside voxl-cross docker : ./build.sh qrb5165 and make package ./make_package.sh . For your convenience, I put the deb here
install the new libmodal-pipe deb (first) on VOXL2 and the new voxl-camera-server deb second.

start / restart voxl-camera-server test using voxl-inspect-cam -a:

|         Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
| stereo_left_bayer | 1280000 | 1280 |  800 |  5.26 |  782 |    21556 |     21.5  | 30.0 |  307.0 | RAW10
| stereo_left_color | 3072000 | 1280 |  800 |  5.26 |  782 |    21556 |     26.9  | 30.0 |  736.7 | RGB
|  stereo_left_grey | 1024000 | 1280 |  800 |  5.26 |  782 |    21556 |     25.1  | 30.0 |  245.6 | RAW8

dump some raw frames (make sure to run voxl-inspect-cam first for a while, so that auto exposure kicks in because the frames don't get processed if there are no clients and if you just grab one first frame, auto exposure won't have time to apply:

voxl-record-raw-image stereo_left_bayer -d /home/root/ -n 1

The output image is in the MIPI10 bayer format, that is for every 4 pixels, there are 5 bytes in the following format:

[pix0][pix1][pix2][pix3][LSBs_for_pix0-3][pix4][pix5][pix6][pix7][LSBs_for_pix4-7]...

That is to say, every 5th byte has 2 LSB bits for each of the previous 4 pixel bytes.

Each Row is 1280 pixels, 1600 bytes (1280/4 *5=1600)

The pixels are in Bayer pattern (RGGB or BGGR depending on the rotation, you should be able to figure it out based on color, once you de-bayer).

Please note that I cannot provide any information (yet) about when this will be integrated into the main release - please test at your own risk . Hopefully we will be able to add it in the future.

I tested only with OV9782 at 1280x800 resolution. Please let me know if this works for you.

Alex

mkriesel

@Alex-Kushleyev Hi, I got the chance to test this over the past few days. It working and I haven't run into any issues so far. Thank you very much for helping out this this.

Alex Kushleyev

@mkriesel , You are welcome!

Right now we have not planned to add this feature to the official build (not high priority). However, if you later need to update to latest voxl-camera-server and still maintain this functionality, you should be able to just merge the latest changes from camera server into this branch (may resolve a simple conflict if it occurs).

Alex

mkriesel

@Alex-Kushleyev Hi again, I'm now testing similar capability for the OnSemi AR0144 https://docs.modalai.com/M0149/ but looking at the debug logs I see

DEBUG: setting thread affinity for cam hires preview
DEBUG: hires checking to see if it is raw8 or raw 10
DEBUG: Frame was actually 8 bit, sending as is
DEBUG: Value of standby_active is: 0

so does this mean that I cannot get 10 bit imagery from this sensor? I see messages like <pre><font color="#839496"><b> </b></font>Allocated BO with width=1280 height=800 stride=1600 aligned_w=1600 aligned_h=800 size=1282048 flags=0x20900 format=GBM_FORMAT_RAW10
</pre> as well. I'm just trying to figure what options I have available for this sensor right now.

Thanks!

Alex Kushleyev

@mkriesel , currently our AR0144 driver is set up to send the image in 8 bit mode, so we don't have to convert 10->8 bit on the cpu. The Sensor is capable of sending 10 bit, but we do not have that mode available right now.

Alex

mkriesel

@Alex-Kushleyev Alright, thanks for the info