Hadron ov64b snapshots have a vertical image artifact

Alex Kushleyev

@cguzikowski , OK, yes i know what that issue is (unexpected frame size). I will that and test both resolutions.

Alex Kushleyev

Good news. I got it working, but there were a few things needed to update. I packaged everything into a zip : https://storage.googleapis.com/modalai_public/temp/ov64b/20260401/ov64b_20260401.zip . It contains

latest sensormodules for Boson and ov64b
voxl-camera-server conf
updated default tuning file (to fix the gain scale so that gain 1.0 is 100, not 54)
updated com.qti.chi.override.so file which contains some pipeline information, now it will allow the large raw resolutions

(I thought the above changes were already in the latest SDK but somehow they did not make it, I will need to double check).

I also made some changes to the ov64b driver:

updated frame length for the 9248x6944 mode so that it is 10fps (not 9.2) -- this is close to max
added modes 9216x6944@10fps and 4608x3472@30fps (slightly cropped on the right), so that these buffers can be used by MISP / gpu without doing a copy.

I am able to capture the raw bayer at 9216x6944, 9248x6944 ,4624x3472, 4608x3472 resolutions

voxl-inspect-cam hires_bayer
...
|   Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
| hires_bayer |79994880 | 9216 | 6944 | 33.00 | 1594 |      130 |    157.9  | 10.0 | 6415.5 | RAW10

In order to enable the misp support for the 4624x3472 and 4608x3472 resolutions in camera server, need a small update:
https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/389b1f4628a9b9cc0e53c43a5cf0e457717e1270

You can save the raw bayer using the voxl-record-raw-image tool, for example:

voxl-record-raw-image hires_bayer -d .

You should be able to dump some raw bayer images and de-bayer them offline.

Also, here is the contents of the README that is inside the zip:

Supported Hardware

voxl2
Hadron plugged into VOXL2 J8 via M0181 adapter

Instructions

copy Boson sensormodule to /usr/lib/camera/
copy ov64b sensormodule with correct id to /usr/lib/camera/
copy com.qti.tuned.default.bin to /usr/lib/camera to fix the gain scale (1.0x = 100)
back up /usr/lib/hw/com.qti.chi.override.so and replace it with com.qti.chi.override.so.20260401 (rename to com.qti.chi.override.so)
- this updated file allows the pipeline to use the highest resolution of the ov64b camera
copy voxl-camera-server.conf to /etc/modalai

Suported Resolutions

mode 0 : 9248x6944 10 bit 10 fps
mode 1 : 9216x6944 10 bit 10 fps (MISP no copy) -- crop 32 pixels on the right
mode 2 : 4624x3472 10 bit 30 fps
mode 3 : 4608x3472 10 bit 30 fps (MISP no copy) -- crop 16 pixels on the right
mode 4 : 3840x2160 10 bit 60 fps (MISP no copy)
mode 5 : 1920x1080 10 bit 240 fps
mode 6 : 1920x1080 10 bit 30 fps

Notes

even thought the 9248x6944 and 9216x6944 modes are 10 fps, you need to specify 30fps in the camera config file. This will be investigated further.

Alex Kushleyev

Once you confirm this working, we can discuss the options for snapshots without going through the ISP, but you can either saw the raw bayer or any of the misp outputs using voxl-record-raw-image and convert to jpg / png offline if that works for you, but we can also help add a compression option to voxl-record-raw-image.

Please note that the current debayering algorithm in MISP does add some smoothing / interpolation, so the YUV image is not going to be as crisp as possible. You could compare it to the jpg output of the ISP (even thought it has artifacts). If you are looking for highest possible fidelity, it may be best to perform offline processing on the raw bayer, then you have more options.

Alex

cguzikowski

@Alex-Kushleyev Hi Alex, thank you for the new drivers and the instructions. Unfortunately I am still having issues getting the hires_bayer pipe to appear. I followed all of the instructions in the README, and when shooting at 9216x6944, I get the following error:

ERROR:   MISP: Unexpected frame size for camera hires, width 9216, height 6944, stride 11520, alloc 79994880 bytes, calc frame size 79994880
ERROR:   Could not find frame size from the raw buffer

Same error I was getting last week but now the allocated and calculated values are the same. Running it at 9248x6944 gives a similar error:

ERROR:   MISP: Unexpected frame size for camera hires, width 9248, height 6944, stride 11568, alloc 80330752 bytes, calc frame size 80328192
ERROR:   Could not find frame size from the raw buffer

I do see the boson_bayer pipe, and in the portal the camera feed looks great - I believe we were getting a weird preview with the old drivers, but we didn't look too carefully as we are not using the boson yet. Here is the output of voxl-inspect-cam -a:

|          Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
|              boson |  327680 |  640 |  512 |  0.00 |  800 |      295 |     13.2  | 60.0 |  157.3 | RAW8
|        boson_bayer |  327680 |  640 |  512 |  0.00 |    0 |      295 |     12.7  | 60.0 |  157.3 | RAW8
|        boson_color |  983040 |  640 |  512 |  0.00 |  800 |      295 |     15.6  | 60.0 |  471.9 | RGB
| boson_misp_encoded |      32 |  640 |  512 |  0.00 |  800 |      295 |     17.0  | 60.0 |    0.0 | H264 (P)   
|        hires_bayer |
|   hires_misp_color |
|    hires_misp_grey |

Disabling MISP gives the following warning and output of voxl-inspect-cam -a:

------ voxl-camera-server: Camera server is now running
Received RAW10 frame from camera hires, will be converting to RAW8 on cpu
WARNING: preview buffer pool for Cam(hires), Frame(18) has 0 free, skipping request

|          Pipe Name |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  | format
|              boson |  327680 |  640 |  512 |  0.00 |  800 |      507 |     12.4  | 60.0 |  157.3 | RAW8
|        boson_bayer |  327680 |  640 |  512 |  0.00 |    0 |      507 |     10.9  | 60.0 |  157.3 | RAW8
|        boson_color |  983040 |  640 |  512 |  0.00 |  800 |      507 |     13.3  | 60.0 |  471.9 | RGB
| boson_misp_encoded |      32 |  640 |  512 |  0.00 |  800 |      507 |     14.4  | 60.0 |    0.0 | H264 (P)   
|        hires_bayer |
|        hires_color |
|         hires_grey |64218112 | 9248 | 6944 | 12.06 | 1211 |       16 |   3720.9  |  1.8 |  945.2 | RAW8

Alex Kushleyev

@cguzikowski , you need to update your camera server to allow misp to accept the new resolutions. You can install the latest one from voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/

cguzikowski

@Alex-Kushleyev Thank you! This now works great and I can save the raw files as a .bin file. Do you have tips for performing raw-jpg conversion both on and offline? I have also tried to take snapshots with the current setup, by disabling MISP and Raw preview, and enabling snapshot and setting the snapshot dimensions to 9248x6944 (9216x6944, 4624x3472, and 4608x3472 all give the error of unsupported file size). voxl-camera-server seems to start up, then immediately gets killed without an error. Here is the output of the command:

MISP channels enabled in defaults : 0
MISP channels enabled in config file: 0
Setting MISP AWB to Auto
MISP channels enabled in defaults : 0
MISP channels enabled in config file: 0
=================================================================
configuration for 2 cameras:

cam #0
    name:                boson
    type:                boson
    bayer_type:          0
    enabled:             1
    camera_id:           0
    camera_id_second:    -1
    fps:                 30
    en_rotate:           0
    en_rotate2:          0

    en_preview:          1
    en_raw_preview:      1
    preview_width:       640
    preview_height:      512

    en_misp:             1
    misp_width:          512
    misp_height:         640

    en_small_video:      0
    small_video_width:   640
    small_video_height:  480

    en_large_video:      0
    large_video_width:   -1
    large_video_height:  -1

    en_snapshot:         0
    snap_width:          -1
    snap_height:         -1
    exif_focal_length:   0.000000
    exif_focal_len_35mm_format:0
    exif_fnumber:        0.000000

    ae_mode:             off
    msv_exposure_min_us: 20
    msv_exposure_max_us: 33000
    gain_min           : 100
    gain_max           : 100
    standby_enabled:     0
    decimator:           1
    independent_exposure:0

cam #1
    name:                hires
    type:                ov64b
    bayer_type:          1
    enabled:             1
    camera_id:           1
    camera_id_second:    -1
    fps:                 30
    en_rotate:           0
    en_rotate2:          0

    en_preview:          1
    en_raw_preview:      0
    preview_width:       9248
    preview_height:      6944

    en_misp:             0
    misp_width:          1280
    misp_height:         720

    en_small_video:      0
    small_video_width:   1024
    small_video_height:  768

    en_large_video:      0
    large_video_width:   3840
    large_video_height:  2160

    en_snapshot:         1
    snap_width:          9248
    snap_height:         6944
    exif_focal_length:   3.100000
    exif_focal_len_35mm_format:17
    exif_fnumber:        1.240000

    ae_mode:             lme_msv
    msv_exposure_min_us: 20
    msv_exposure_max_us: 33000
    gain_min           : 100
    gain_max           : 1600
    standby_enabled:     0
    decimator:           1
    independent_exposure:0

fsync_en:            0
fsync_gpio:          109
=================================================================
thread is locked to cores: 4 5 6 7
connected to mavlink pipe
Connected to cpu-monitor
Starting Camera: boson (id #0)
ModalExposureMSV: initializing for camera name  type 
gbm_create_device(156): Info: backend name is: msm_drm
MISP Initializing for camera boson
 Detected 1 platform(s)
 Detected 1 GPU device(s)
Estimated imu dt = 0.000977s
ERROR in json_from_yaml, failed to open file
MISP: Loading intrinsics cal file boson_intrinsics.yml for camera boson, ret -1
WARNING: Lens calibration for camera boson is missing (boson_intrinsics.yml). Using defaults:
Starting Camera: hires (id #1)
ModalExposureMSV: initializing for camera name hires type ov64b

------ voxl-camera-server: Started 2 of 2 cameras

------ voxl-camera-server: Camera server is now running
Killed

Don't necessarily need to use the snapshot command, but we want some way to get JPG images off the voxl.

Alex Kushleyev

@cguzikowski , sorry for the delay.

Long term, our plan is to add snapshot functionality directly into misp (just like you were getting snapshots from the ISP). That is not too difficult to achieve, may be a few weeks away.

Also let me ask you this: would you want to save the raw bayer and get that from voxl2 and convert to jpg / png offline? That would give you maximum control over quality / processing algorithm. Alternatively, de-bayering can be done on voxl2 and saved as lossy or lossless image?

We do have tools to convert bayer -> yuv -> jpeg but not yet in a standalone app.

You you can do for now is the following:

set misp output resolution equal to the raw resolution
this will enable misp to publish the de-bayered yuv to the image pipe
then you can use voxl-record-raw-image to save the yuv. Alternatively, you can test a WIP version of this tool which can save yuv as jpeg : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/merge_requests/37
ideally this logging would be done within camera server to reduce overhead for sending these huge images over the pipe. But the approach i described should work for now.

I will be able to test this again mid next week, meanwhile please let me know if you run into the saving issue. I will double check the issue with snapshot resolution being too large. How large was the isp snapshot that you tested before and had artifacts?

Alex

cguzikowski

@Alex-Kushleyev Thank you!

I was able to get both the full YUV and them in JPEG working. We also noticed that the vertical artifact seemed to disappear in these images. Regarding the resolution we were using when we had the artifacts, we were using 9248 × 6944 on the old drivers.

cguzikowski

@Alex-Kushleyev Hey Alex, we have been performing some tests and noticed the images we were getting were quite noisy. What are some of the settings that we can change in the voxl-camera-server to reduce the noise? I messed around with the ae_desired_msv setting and gain_max setting. I also noticed that the white balance seems to have a very visible green shift when the exposure is lower (tried to upload an image but got a "request entity too large" error)

Also as a side note, the metadata seems to not be recording when using voxl-record-raw-image -j

Alex Kushleyev

@cguzikowski , if you want, you can attache a cropped image, so that it's smaller size.

In general, the pixel noise increases as you the pixel gain is increased. Part of Auto Exposure control is controlling exposure time and gain (both of which contribute to the image brightness), but higher gain has higher pixel noise and higher exposure values will result in motion blur.

The Qualcomm ISP has lots of liters, including noise reduction, which are applied when you use the ISP snapshot (even though they have not been tuned for the particular camera, the ISP output may result in better de-noised image). There are several types of filters, for example spatial (such as bilateral filter) and temporal (TNR - temporal noise reduction), and it's usually a combination of both with the filter weights increasing as the gain increases (more noise requires more noise reduction).

Currently, MISP does not have any de-noise filtering, but we are working on adding some. That is why i was discussing with you the ability to process the image offline (from the original bayer source). You could save the full raw image and perform any filtering you need in post processing. This approach is similar to using a RAW image on a fancy camera and then importing that into image processing software on a laptop / desktop, which can perform a lot more filters / effects directly on the raw image (loss-less).

What i suspect is that you are testing in the low-light environment and you are seeing the effects of high gain (high pixel noise). the MISP auto exposure tries to balance exposure and gain and there are a few parameters for that, but in your tests you should see what exposure and gain values the camera is at when you see the noisy image. You can use voxl-portal to control exposure and gain to see what the difference is. I believe the max gain for ov64b is 16x (1600).

I am about to set up your use case again for testing and i will investigate the noise and the original ISP snapshot artifact and we can also compare the image noise from the ISP snapshot and misp snapshot.

Unfortunately the lens for ov64b in the Hadron unit is very small, which reduces the amount of light that gets into the sensor. The sensor is 8+K resolution which means the pixel size is small, so the amount of light that gets to each pixel is small. To compensate for that, we would typically want a larger lens, but it is probably not possible to change the lens in this specific Hadron unit (we have not tried).

If you want to explore offline image processing, you would need:

the original raw bayer (which you can already save)
the gain and exposure used for that frame (i will need to check, i thin we had an option to save the exposure and gain as part of the file name).
then you can have offline processing that is dependent on exposure / gain and apply the filters of your choice.
the only down side is that the raw snapshots are huge, but you can probably zip them up if needed to store a lot of them on voxl2 before offloading.

Alex