IMX412 Poor performance vs IMX214?

jameskuesel

Recently I went about upgrading from the

M0024 IMX214

to the

M0161 IMX412
Lens Mount: m12
Lens Part No: 27629F-16MAS-CM.

However, initial tests left us somewhat puzzled. For the imx412 I used the latest VOXL2 platform release with SDK 1.4.0. I saw a significant degradation in quality of the image with the IMX412. For the IMX412 I used the preview frame at the full resolution (4056x3040). For the IMX214 I also used the preview frame but at (4208x3120) Posted below are some samples. I recorded them via voxl-logger

I believe the imx412 has a wider field of view so perhaps some degradation in quality it to be expected as it might have a slightly lower angular resolution? As perhaps fewer pixels are available per degree of FOV? That being said I wouldn't think this much worse. Similarly, the exposure looked much worse. A good target here is the donut box.

Here are two downsampled images since the forum wouldn't let me post the full images. For the full images straight from voxl-logger I included a link to them in google drive folder (granted google might also downsample them but you'll still be able to see the difference. Any help/input on the quality difference would be much appreciated. Not sure if this is to be expected? Maybe something is wrong with the driver?

IMX214

IMX412

[Google Drive link] (https://drive.google.com/drive/folders/1rZV6LM5qSqLjSTxz58Eq4tsZ0AO3cMxz?usp=sharing)

Alex Kushleyev

@jameskuesel , can you please do a quick test:

move / rename the following file:

/usr/lib/camera/com.qti.tuned.cmk_imx577.bin

then restart camera server and take a look at the image again.

I believe the tuning file is smoothing out the image too much. the imx412 sensormodule file (also in /usr/lib/camera) is linked to com.qti.tuned.cmk_imx577.bin tuning file. By removing the file, the camera pipeline will load the default tuning file, which is what I believe your imx214 is using (com.qti.tuned.default.bin).

another thing to try is to use a different tuning file, which we have been testing, which removes a lot of the smoothing for IMX412:

cd /usr/lib/camera
mv com.qti.tuned.cmk_imx577.bin com.qti.tuned.cmk_imx577.bin.bak
ln -s com.qti.tuned.imx412_fpv.bin com.qti.tuned.cmk_imx577.bin

#then restart voxl-camera-server

Essentially this will replace the tuning file that is currently being used with a custom one we are experimenting with.

Please let me know if these options look better.

In order to see which camera is using which tuning file you can enable a special debug mode:

add to /vendor/etc/camera/camxoverridesettings.txt:

logInfoMask=0x20000

if you see systemLogEnable=0, set it to 1:

systemLogEnable=1

run in one terminal:

logcat | grep -i tuning

Then restart voxl-camera-server and you should see in the first terminal something like the following.. chromatix file is analogous to tuning file.

voxl2-mini:/$ logcat | grep -i tuning
01-29 02:22:27.003  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1762 CreateTuningDataManager() Enter : sensor index: 0, sensor name: imx412_fpv, chromatix name: imx412_fpv
01-29 02:22:27.003  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1798 CreateTuningDataManager() Searching regular tuning file
01-29 02:22:27.014  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1839 CreateTuningDataManager() Loaded tuning file: /usr/lib/camera/com.qti.tuned.imx412_fpv.bin for sensor index: 0
01-29 02:22:27.118  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1762 CreateTuningDataManager() Enter : sensor index: 1, sensor name: ar0144, chromatix name: onsemi_ar0144
01-29 02:22:27.118  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1798 CreateTuningDataManager() Searching regular tuning file
01-29 02:22:27.118  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1811 CreateTuningDataManager() No tuning data file for sensor: ar0144, assigning default
01-29 02:22:27.127  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1839 CreateTuningDataManager() Loaded tuning file: /usr/lib/camera/com.qti.tuned.default.bin for sensor index: 1
01-29 02:22:27.158  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1762 CreateTuningDataManager() Enter : sensor index: 2, sensor name: imx214, chromatix name: imx214
01-29 02:22:27.158  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1798 CreateTuningDataManager() Searching regular tuning file
01-29 02:22:27.158  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1811 CreateTuningDataManager() No tuning data file for sensor: imx214, assigning default
01-29 02:22:27.163  3844  3844 I CamX    : [ INFO][HWL    ] camxhwenvironment.cpp:1839 CreateTuningDataManager() Loaded tuning file: /usr/lib/camera/com.qti.tuned.default.bin for sensor index: 2

I currently have hooked up 3 cameras, my results probably wont match yours since i don't have exactly the same setup and drivers, but you can see what to expect.

After testing, you can disable the log mask and system logging to reduce cpu usage.

Alex

jameskuesel

Hi Alex, Thanks for the reply.

Unfortunately, I didn’t see any improvement with any of the changes you suggested. That being said I was a little confused on some of the setup.  To reiterate, this was done on a

Starling2 with IMX412 M0161 attached via a coax cable.

After a stock flash of Voxl-SDK-1.4.0, running   logcat | grep -i tuning  would return com.qti.tuned.imx412_fpv.bin as being loaded. However, your response seemed to imply that com.qti.tuned.cmk_imx577.bin would be loaded after a stock flash. After removing com.qti.tuned.cmk_imx577.bin there seemed to be no change. Logcat still implied com.qti.tuned.imx412_fpv.bin was being loaded. However, I did also try removing com.qti.tuned.imx412_fpv.bin which then indeed resulted in com.qti.tuned.default.bin being loaded. I saw a difference in how bright the image was but not how sharp it looked.  I did another stock flash to reset everything and proceed with running your linking command. That looked same as stock. But as previously mentioned, if com.qti.tuned.imx412_fpv.bin was already being loaded, then linking com.qti.tuned.imx412_fpv.bin would probably look the same, which it did.  I also attempted to force the system to use com.qti.tuned.cmk_imx577.bin by replacing the contents of com.qti.tuned.imx412_fpv.bin with it. This provided an even brighter image, but still no noticeable improvement in sharpness.

I’ve included the pictures here and what driver I THINK was being used. To me it seems that for the Starling2 kernel that maybe com.qti.tuned.cmk_imx577.bin isn’t being loaded by default.

The downsampled images are hard to compare so I included a drive link again.

com.qti.tuned.imx412_fpv.bin (Stock??) VVV

com.qti.tuned.default.bin VVV
IMX412_M0161_kernel2_J6Uppe_com.qti.tuned.default_downsampled.jpg

com.qti.tuned.cmk_imx577.bin (forced it to use (or at least I think I did))  VVV
IMX412_M0161_kernel2_J6Uppe_com.qti.tuned.cmk_imx577_downsampled.jpg

Moving on, decided to do some more experimentation which was even more puzzling. We happen to have an

IMX412 M0107 that attaches via a ribbon

module that we attached on J6Lower (with a different breakout board). To achieve this had to do a re-flash and use Kernel #1 (not the Starling2 Kernel as it seems like that one is incompatible). This surprisingly produced great results, much brighter, and much much better sharpness. The difference is night and day compared to the coax M0161. This time, it does seem that com.qti.tuned.cmk_imx577.bin was the one being loaded as you implied might happen after a stock flash.

A little confusing on what exactly is going on here. Obviously they must be using different drivers as this test and our Starling2 test use different ports. (J6Lower vs J6Upper). Furthermore, these experiments required using different kernels.

Should there be this big a difference based on using a coax vs the ribbon (M0161 vs M0107) or is it because of the different kernels? Do you have any sample images from the Starling2 Hires (M0161) with coax so we know what we should be expecting?

What we saw with the ribbon (M0107) is what we are aiming for but with the coax (M0161).

Thanks again for the help, any insight would be much appreciated.

Drive Link (better for comparison in sharpness)
Drive Pictures

Alex Kushleyev

@jameskuesel ,

Can you please let me know what steps you follow to set up M0161 with the latest SDK? Do you modify voxl camera server conf file after set up? Also which camera stream are you saving image from and how are you saving?

M0161 and M0107 are more or less identical except for the cable type, so i would expect similar performance.

It looks like the lens may be out of focus. There is usually some glue on the threads that prevents the lens from twisting in the threads. I will double check whether you can just twist the lens to loosen the glue or remove the glue (in order to re-focus).

I will take a few image samples using m0161 and m0107 - it is possible to connect them both to voxl2 at the same time.

We can also do another test - log the raw (unmodified) image, which will be in 10 bit bayer format and i have some tools to convert that to rgb and save to jpg, etc. if raw image looks blurry, then camera must be out of focus. I will provide instructions.

Alex

jameskuesel

@Alex-Kushleyev

Hi Alex,

To get the M0161 (COAX) all I did was run the install script, and then selected all the options relating to the starling2 since that is what I was collecting these on.

I've been recording the preview frame (en_preview: true) modified to be at full res (4056x3040), stream via voxl-logger.

voxl-logger --cam hires_color.

I did a similar process for the M0107 (RIBBON), except selecting the first kernel option. Then I modified the custom_camera_config.txt in /data/modalai and just set

J6_LOWER_SENSOR="imx412"
J6_LOWER_NAME="hires"
J6_LOWER_ROTATE="false"

I think you are correct that our Starling M0161 lens is a little bit out of focus. However, we tried with another M0161 module we had around that looked more "in focus" but still had the sharpness issues. The resulting images look different enough that I feel it is something more than just the focus but I am unsure.

M0161 but DIFFERENT module (more focused?)

Anyways, yes let me know when you get some sample images and would also love to experiment with the logging of the raw images.