# M0149 camera refocusing and tuning parameters

Source: https://forum.modalai.com/topic/3196/m0149-camera-refocusing-and-tuning-parameters
Category: ESCs, Sensors and Accessories (https://forum.modalai.com/category/15/escs-sensors-and-accessories)
Tags: image-sensor
Posted: 2024-03-11 12:13:22 UTC by Aaky
Replies: 56 · Views: 30519

## Aaky · 2024-03-11 12:13:22 UTC

Hello community,
 I am using VOXL2 with latest SDK 1.1.2. I have recently purchased M0149 new tracking camera (AR0144). Since this camera had manual lens installed over it, I am facing bit of trouble while refocusing it. Here are my queries:

1. We need to remove lens while installation of this camera over our custom drone so naturally it needs to be refocused. Well I am not expert at focusing camera, can I get some tips in refocusing this camera with best of its image quality?
2. I am seeing that corners of my image are having some black shadow. My lens is totally clear from any obstructions. How can I remove it or is it expected to be this way?
3. In voxl-camera-server I can see the gain and exposure related parameters are same for OV7251 and this M0149 (AR0144). Any good procedure for tuning M0149 gains in better way. My situation is indoor only with staircase and long corridor scenerio. Please provide some insights into this.
4. Is M0149 been stress tested with QVIO server? I can see it running well over my setup even in flight as well but wanted to gather any insights if possible.

Looking forward to quick response.

![Screenshot from 2024-03-11 17-32-42.png](https://forum.modalai.com/assets/uploads/files/1710159193131-screenshot-from-2024-03-11-17-32-42.png)

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-12 03:33:04 UTC

@Aaky ,

You have good questions. Let me try to answer

1. We are working on some helper tools for our customer use to aid with evaluating the quality of focus. The tool will display an augmented image in voxl-portal, showing the results of the analysis (how sharp the image is). In general, you want to take a high contrast object (checkerboard), place it in the area where you want to focus and adjust the focus until the image is sharpest. The depth of field of a fisheye camera should be very large (everything beyond a small distance (a few inches) should be in focus). This means that placing an object 1-2 feet away from the camera can be easily used to calibrate the focus. I hope to share an initial version of the tool by the end of this week. Meanwhile, you may try to focus the camera by eye while having the camera pointed at a high contrast object

2. This "shadow" artifact happens for two reasons : fisheye lens blocks more light at extreme angles and also camera sensor CRA (chief ray angle) contributes to reduced sensitivity at large angles. The standard way to fix this in camera pipeline is using a lens-specific tuning file, which contains a Lens Shading Correction table and the ISP can do it without any CPU overhead. However, for tracking cameras, we use RAW images (not going through ISP image processing) to avoid any image processing artifacts, so this lens shading correction feature is not available. Nevertheless, the image can be corrected on the CPU and it should not be too expensive to do that. The best place to do that would be during the 10->8 bit conversion that happens on the CPU (camera sends 10 bit data and we convert to 8 bit to use for feature tracking). Using 10 bit data will provide better quality image and will save cpu cycles because the pixels will already be in memory. The 10->8 bit conversion is happening in the voxl-camera-server application. This feature is something we discussed internally, but we have not implemented it yet. the corners of a fisheye camera are usually of a lesser value because the features there are going to be stretched, lower quality (due to optics) and may not appear in the frame for a long time. Nevertheless, with corrected image brightness in the corners, the performance of VIO could be even better.

3. The exposure control algorithm that we implemented attempts to achieve a certain mean pixel (sample) value (MSV). it takes average of all pixels and if it is below the desired one, it increases gain and exposure to achieve the higher MSV. Gain and exposure both contribute to increasing image brightness, but they have their own side effects: long exposure will cause blur during motion and high gain will increase pixel noise. the QVIO algorithm applies a gaussian blur before doing feature detection / tracking, so it would be preferred to use higher gain than higher exposure and this is what we tuned for ov7251. I think going with those settings is a good start. However, the previous issue of dark corners can throw off the exposure control, driving image in the center to be too bright. The image you provided seems a bit too dark to my eye, perhaps you can increase the target MSV and see if that helps? were you able to track good features with QVIO using AR0144?

4. We are slowly switching over to using AR0144 cameras for tracking (VIO). In general M0149 camera module is a better camera module all around compared to ov7251 - better and higher resolution image sensor, better lens, so we expect better QVIO performance. I will double check with the team about any specific tuning for better QVIO performance using AR0144.

## Reply by Aaky · 2024-03-12 09:53:32 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks alot Alex for such detailed and insightful response. Willl keep this star marked for my understanding purpose.

```
1. We are working on some helper tools for our customer use to aid with evaluating the quality of focus. The tool will display an augmented image in voxl-portal, showing the results of the analysis (how sharp the image is). In general, you want to take a high contrast object (checkerboard), place it in the area where you want to focus and adjust the focus until the image is sharpest. The depth of field of a fisheye camera should be very large (everything beyond a small distance (a few inches) should be in focus). This means that placing an object 1-2 feet away from the camera can be easily used to calibrate the focus. I hope to share an initial version of the tool by the end of this week. Meanwhile, you may try to focus the camera by eye while having the camera pointed at a high contrast object
```
Yes Please share the helper tool with me at the earliest to focus the camera in best possible way. Till then will do it manually.

```
2. This "shadow" artifact happens for two reasons : fisheye lens blocks more light at extreme angles and also camera sensor CRA (chief ray angle) contributes to reduced sensitivity at large angles. The standard way to fix this in camera pipeline is using a lens-specific tuning file, which contains a Lens Shading Correction table and the ISP can do it without any CPU overhead. However, for tracking cameras, we use RAW images (not going through ISP image processing) to avoid any image processing artifacts, so this lens shading correction feature is not available. Nevertheless, the image can be corrected on the CPU and it should not be too expensive to do that. The best place to do that would be during the 10->8 bit conversion that happens on the CPU (camera sends 10 bit data and we convert to 8 bit to use for feature tracking). Using 10 bit data will provide better quality image and will save cpu cycles because the pixels will already be in memory. The 10->8 bit conversion is happening in the voxl-camera-server application. This feature is something we discussed internally, but we have not implemented it yet. the corners of a fisheye camera are usually of a lesser value because the features there are going to be stretched, lower quality (due to optics) and may not appear in the frame for a long time. Nevertheless, with corrected image brightness in the corners, the performance of VIO could be even better.
```
Great. Looking forward to image brightness enhancement at corners in future voxl-camera-server.

```
3. The exposure control algorithm that we implemented attempts to achieve a certain mean pixel (sample) value (MSV). it takes average of all pixels and if it is below the desired one, it increases gain and exposure to achieve the higher MSV. Gain and exposure both contribute to increasing image brightness, but they have their own side effects: long exposure will cause blur during motion and high gain will increase pixel noise. the QVIO algorithm applies a gaussian blur before doing feature detection / tracking, so it would be preferred to use higher gain than higher exposure and this is what we tuned for ov7251. I think going with those settings is a good start. However, the previous issue of dark corners can throw off the exposure control, driving image in the center to be too bright. The image you provided seems a bit too dark to my eye, perhaps you can increase the target MSV and see if that helps? were you able to track good features with QVIO using AR0144?
```
Thanks for providing insights in preprocessing steps of QVIO. Yes I am able to track features with QVIO using AR0144 but I have certain observations. I knew about desired MSV and exposure control algorithm. In my indoor scenario I tried with ae_dessired_msv to 80 from default 60. QVIO worked well in few cases but I also saw drift as soon as UAV landed back on ground. In my last observation with OV7251 MSV of 80 helped in indoor scenerios where lighting conditions are low so I was trying with same setup on AR0144. In outdoor with MSV 80, the features which are tracked are always on horizon which ultimately reduces the QVIO quality to 1% probably leading to accumulated error in odometry since I saw UAV from origin showing wrong XYZ coordinates as compared to reality (I was flying in GPS mode and only observing QVIO). So in outdoor after changing MSV to 60 it worked again. So I believe there is need for profiling indoor and outdoor scenerio. Let me know your thoughts.

```
4. We are slowly switching over to using AR0144 cameras for tracking (VIO). In general M0149 camera module is a better camera module all around compared to ov7251 - better and higher resolution image sensor, better lens, so we expect better QVIO performance. I will double check with the team about any specific tuning for better QVIO performance using AR0144.
```
Yes, My thought process was the same AR0144 should perform better. Let me know for any specific tuning would help QVIO further. 

I have also observed as I have mentioned, there are multiple (probably) good features near to UAV but it still finds features extremely far leading to quality getting degraded and eventually might lead to failure. How can we improve feature detection sharply near to UAV?
Also when UAV rotates suddenly, I have seen features been detected after certain time at the center part of image, sometimes leading to no features also. Can this be related to camera focus?

On a seperate note, Is there any tool which can visualize voxl-logger data csv files in better way. Maybe running voxl-portal and loading qvio_extended csv file to view trajectory and tracked features in 3D instead of live?

## Reply by Aaky · 2024-03-15 05:43:20 UTC (in reply to Aaky)

@Alex-Kushleyev Any update over here?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-16 02:45:19 UTC (in reply to Aaky)

@Aaky , thanks for the follow up.

I should be ready to share a test version of a focus helper tool early next week. Then i can try the lens shading correction inside voxl-camera-server.

Regarding tuning target MSV, yes it is not straightforward especially when you are transitioning between indoor and outdoor environments. Those environments are so different and you may need to switch parameters for auto exposure control. For example, the auto exposure control in phone cameras also detects the scenario (indoor, outdoor, daylight, cloudy, even type of light indoors) and switches processing params according to the scenario. This is something we do not currently support. If you want to investigate this yourself, you should collect data logs (using `voxl-logger`) using different MSV in the different types of environments and find the MSV that works best (detects most good features). You can also change the way MSV is computed -- for example if you are seeing a lot of bright sky outdoors, many pixels can be extremely bright, so using them in MSV computation is going to make the ground completely dark. We already have a param for ignoring a certain amount of saturated pixels) https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/config_defaults.cpp?ref_type=heads#L162 -- this is the fraction of all pixels that will be ignored for MSV computation if saturated to max value. If you increase this value (from 0.2 to 0.3) then more saturated pixels will be ignored, if present, putting more weight on non-saturated pixels.

Regarding features that are far away, you should double check your intrinsic calibration of the lens. If the lens is not calibrated well, then feature that are far away will create large errors in state estimation. Have you done the intrinsic camera calibration for AR0144?

@Aaky said in [M0149 camera refocusing and tuning parameters](/post/15245):
> Is there any tool which can visualize voxl-logger data csv files in better way

Can you specify what data you might want to visualize?

## Reply by Aaky · 2024-03-16 06:55:03 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev 

@Alex-Kushleyev said in [M0149 camera refocusing and tuning parameters](/post/15410):
> I should be ready to share a test version of a focus helper tool early next week. Then i can try the lens shading correction inside voxl-camera-server.
> 
> 

Sure. Will be waiting for this tool.

Regarding MSV tuning, Yes I will investigate this further.

@Alex-Kushleyev said in [M0149 camera refocusing and tuning parameters](/post/15410):
> Regarding features that are far away, you should double check your intrinsic calibration of the lens. If the lens is not calibrated well, then feature that are far away will create large errors in state estimation. Have you done the intrinsic camera calibration for AR0144?

Yes I have done the intrinsic calibration with AR0144. What is ideal reprojection error from camera calibration and is there anything to validate that intrinsic are correctly setted up for the camera?

@Alex-Kushleyev said in [M0149 camera refocusing and tuning parameters](/post/15410):
> Can you specify what data you might want to visualize?

I want to visualize the 3D VIO trajectory as shown in voxl-portal in the offline manner like after collecting data with voxl-logger.

Also is changing MSV effective with voxl-replay since it operates over live camera frame directly? What are the parameters which we cant tune with voxl-replay?

## Reply by Aaky · 2024-03-20 05:00:07 UTC (in reply to Aaky)

@Alex-Kushleyev One more input is needed to me in terms of QVIO tuning, I had recorded some dataset for QVIO with "voxl-logger --preset_odometry" which were saved successfully in /data/voxl-logger directory. 
Now if I try to replay them with "voxl-replay -p /data/voxl-logger/log009/" say, then I get following error.

```
voxl2:~$ voxl-replay -p /data/voxl-logger/log0009/ -d
enabling debug mode
opening json info file: /data/voxl-logger/log0009/info.json
using log_format_version=1
log contains 7 channels
log started at 1878796637772ns
log channel 0:
    type: imu
    out path: /run/mpa/imu0/
    log path: /data/voxl-logger/log0009//run/mpa/imu0/
    total samples: 0
log channel 1:
    type: imu
    out path: /run/mpa/imu1/
    log path: /data/voxl-logger/log0009//run/mpa/imu1/
    total samples: 0
log channel 2:
    type: imu
    out path: /run/mpa/imu_apps/
    log path: /data/voxl-logger/log0009//run/mpa/imu_apps/
    total samples: 54491
log channel 3:
    type: cam
    out path: /run/mpa/tracking/
    log path: /data/voxl-logger/log0009//run/mpa/tracking/
    total samples: 1612
log channel 4:
    type: cam
    out path: /run/mpa/qvio_overlay/
    log path: /data/voxl-logger/log0009//run/mpa/qvio_overlay/
    total samples: 268
log channel 5:
    type: vio
    out path: /run/mpa/qvio/
    log path: /data/voxl-logger/log0009//run/mpa/qvio/
    total samples: 1594
log channel 6:
    type: qvio
    out path: /run/mpa/qvio_extended/
    log path: /data/voxl-logger/log0009//run/mpa/qvio_extended/
    total samples: 1595
opening csv file: /data/voxl-logger/log0009//run/mpa/imu0/data.csv
failed to open csv file: /data/voxl-logger/log0009//run/mpa/imu0/data.csv
error: No such file or directory

```

I need to include more pipes also while logging? I am actually unable to replay any of my collected data. Also Will I be able to visualize the results on voxl-portal once voxl-replay works well?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-20 14:16:24 UTC (in reply to Aaky)

@Aaky said in [M0149 camera refocusing and tuning parameters](/post/15525):
> /data/voxl-logger/log0009//run/mpa/imu0/data.csv

can you check if this file exists? `/data/voxl-logger/log0009//run/mpa/imu0/data.csv`

## Reply by Aaky · 2024-03-20 15:34:01 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev This file dosent exist. So in order to record this file, what parameter is essential in voxl-logger?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-20 16:03:59 UTC (in reply to Aaky)

@Aaky oh, i believe imu0 is not accessible directly by the CPU (the DSP is connected to it), so it cannot be logged. Can you please double check `imu_apps` to make sure it has data? I think it is possible that the logging preset was inherited from VOXL1, which could access IMU0.

I believe imu_apps is used for VIO, so other IMUs are not needed. You can modify your logging preset to remove imu0 and imu1. and you should be able to modify the existing log to get rid of the imu0, imu1 entries (modify the info.json in the log)

## Reply by Aaky · 2024-03-22 05:29:46 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex. Will check this out. 

Any update on optimized parameters for AR0144 camera over voxl-camera-server?

Also I have one doubt, let's say I navigate in indoor very slowly say at 0.5 m/sec of max velocity and even my altitude and Yaw movement would be slow, then can I expect my motion blurr problem (Affecting by exposure parameters) won't come up?
If this is true I can focus more on gain, MSV and ae_slope parameters then.
Also does ae_slope parameters increasing makes more sense in outdoor where we can see bright horizon right and not in indoor?

## Reply by Aaky · 2024-03-22 13:18:25 UTC (in reply to Aaky)

@Alex-Kushleyev One more query,
Where to visualize output of VOXL-replay? On voxl-portal? Also I saw few people plotting the estimated x,y and z values of VIO. Is this plotting available somewhere?

Also I am seeing very high Gain value in qvio_overlay image as compared to OV7251. On OV7251 gain used to be around 100-150 but with same environment in AR0144 it's around 400-500. Is this normal? Does this mean granularity of image is good?

Sorry I am throwing multiple queries to you at a time. :)

## Reply by Aaky · 2024-03-23 07:11:03 UTC (in reply to Aaky)

@Alex-Kushleyev One more problem is noticed in voxl-replay,

After removing imu0 and imu1 from my info.json, I keep getting below problem. 

![Screenshot from 2024-03-23 12-13-07.png](https://forum.modalai.com/assets/uploads/files/1711177107337-screenshot-from-2024-03-23-12-13-07.png) 

After digging in code, I found out 'qvio_extended' type of channel isn't handled in the code  [over here](https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-logger/-/blob/dev/tools/voxl-replay.cpp?ref_type=heads) so this unknown log type: 6 is coming up.

Also I removed this channel and tried playing tracking, imu and qvio_overlay channels. Now I am unable to understand even if I change MSV in voxl-camera-server config file in logged data folder I am unable to see this change on camera streams over voxl-portal. How to ensure that the changes in voxl-camera-server.conf are replicated exactly and applied to camera frame in order to fetch qvio results?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-25 20:56:30 UTC (in reply to Aaky)

@Aaky ,

I will double check with the team to see if we need to fix the log playback. Perhaps you should just try to collect a new log that does not have those extra imu channels which do not exist - have you tried that? Also if you dont collect the qvio extended log, the log playback should not complain. But i believe you are correct that there is an issue that the log replay cannot play back the extended qvio data (normally that is not needed since you want to play back the raw data, but not any of the results of qvio. That is if you want to run qvio again using a log, you don't want to play back the results of old qvio data because there would be an attempt for two processes to publish the qvio results during replay (log replay and actual qvio process)

Regarding the MSV tuning.. please understand that by replaying the log with images and IMU, you are bypassing the whole camera pipeline, so the camera exposure and gain cannot be changed. Perhaps i was not clear in my original suggestion, but let me clarify.

By logging data sets with different MSV target values (or different exposure control strategies), you can feed the data into the same VIO algorithm and see which one performs better. Ideally, you would collect exactly the same data set with different MSV parameters, but that is impossible in real life, but you can do something that is pretty close. For indoor environments, you can hand carry the drone along the same path and record the data sets with different MSV target values. Indoor environments are simpler and easier to reproduce the same conditions. Outdoors could be a little bit trickier but still possible.

Additionally, what you can do is after the log is collected, you can process the individual images using some scripts (opencv?) and see how different processing affects the behavior of VIO. Please keep in mind that that will not be exactly the same as changing gain and exposure on the camera (because changing exposure will affect blur and changing gain will affect how noisy the pixels are), but if you are trying to find good starting points for the MSV, you could try post processing the images before the playback. Another approach is to apply some kind of gamma correction but that can also affect the image noise.

## Reply by Aaky · 2024-03-26 03:43:36 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for the explanation. Yes will follow your guidelines for MSV tuning.

I have query in libmodal-exposure codebase.

Check code at [this link](https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-exposure/-/blob/v0.1.0/library/src/exposure-msv.cpp?ref_type=tags#L70) . It says OV7251 needs few frames to be skipped for gain calculation. Is this valid for AR0144 also since I am assuming same MSV calcaulation codebase is been utilized for AR0144 as well at libmodal-exposure v0.1.0 branch.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-26 04:34:32 UTC (in reply to Aaky)

@Aaky , yes there was originally an issue related to ov7251 configuration that resulted in some flashing behavior if gain was updated every frame. However, this has been fixed - we have updated the actual ov72551 sensor driver and also `voxl-camera-server` is setting updated register on the camera to allow gain update every frame : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/cci_direct_helpers.cpp?ref_type=heads#L44

Here is the actual auto exposure configuration : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/config_defaults.cpp?ref_type=heads#L166 (both gain and exposure update period is set to 1)

AR0144 does not have this issue, the default params are here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/config_defaults.cpp?ref_type=heads#L194

## Reply by Aaky · 2024-03-26 07:17:55 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for the input Alex. Much appreciated.

## Reply by Aaky · 2024-03-26 17:18:04 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev By any chance is the camera focus application ready for testing? I am actually struggling to get proper focus on AR0144. Initially I was able to get very good focus but now image just looks very much blurry. Have a look at below images. These images are captured by voxl-logger.

First time focus :

![00000.png](https://forum.modalai.com/assets/uploads/files/1711473406794-00000.png) 

Current focus :

![00001.png](https://forum.modalai.com/assets/uploads/files/1711473366074-00001.png) 

Current image appears to be blurry and also I am having hard time to get camera calibration right (reprojection error below 0.5). Previously it was good and my camera calibration worked well very fast. Let me knoiw.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-26 20:36:17 UTC (in reply to Aaky)

@Aaky , yes there is something you can test.

First, you will need to install updated version of opencv with python3 bindings. i built the package and it can be downloaded [here](https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-opencv_4.5.5-3_arm64.deb) :  . Source is [here](https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-opencv/-/tree/add-python3-bindings) : 

Second, i have some experimental tools that let you subscribe and publish images via mpa. ([source](https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/tree/pympa-experimental)) . You can build the package yourself or grab it [here](https://storage.googleapis.com/modalai_public/temp/voxl2-misc-packages/voxl-mpa-tools_1.1.5_arm64.deb). install it on voxl2.

after installing the two packages and starting camera server, execute this:
```
cd /usr/share/modalai/voxl-mpa-tools
python3 pympa-focus-helper.py -i tracking
```
if all goes well, it will receive images and print dots to show that it's running (note that i used hires camera for testing):
```
voxl2:/usr/share/modalai/voxl-mpa-tools$ python3 pympa-focus-helper.py 
output image dimensions :  [1920, 1080]
connecting to camera pipe hires_color
subscribed to camera pipe /run/mpa/hires_color/, channel 0
created output pipe hires_debug, channel 0, flags 0
waiting for the first image
.width=3840, height=2160, format=1, size=12441600
.............................
```

then you can open voxl-portal and look at the new image `hires_debug`  - the image will be monochrome for now (but i found that mono image is better for focusing)

Focus helper (source here https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/pympa-experimental/tools/python/pympa-focus-helper.py) has options for input image pipe name, so you can change that to your tracking camera (`python3 pympa-focus-helper.py -i <camera_name>`) and other options.

You will see the original image with overlay of zoomed in window (top-left) and output of edge detector (top-right). You can use the zoomed in ROI to tune the lens focus by hand. I suggest pointing the camera at something like a checkerboard at the distance that you want to focus at. Hold the camera steady and turn the lens to see the sharpness of the edges (you can also check the edge detector output). For a wide FOV camera, which has a large depth of field, i think everything beyond several inches away from the camera should be in focus. You can experiment a bit. Currently we do not output any metric, but it is actually pretty easy to tune by eye using the zoomed in ROI.

Please try it out and let me know if you run into any issues.

## Reply by Aaky · 2024-03-27 11:50:47 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thank you Alex for providing this package. I am facing one problem in installing opencv with python3 bindings as shown below.

![Screenshot from 2024-03-27 17-17-46.png](https://forum.modalai.com/assets/uploads/files/1711540132471-screenshot-from-2024-03-27-17-17-46.png) 

This issue is preventing me from running the command  "python3 pympa-focus-helper.py -i tracking"

It isnt able to find cv2. I think the symlink isnt working since ```
```
/usr/lib/python3.6/dist-packages/cv2.so
``` 
``` dosent exist.

Please let me know where to look for ahead.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 11:55:21 UTC (in reply to Aaky)

@Aaky ,

I will check this a bit later but can you verify that “ /usr/lib/python3.6/dist-packages/” path exists on voxl2, if not just create it “mkdir -p /usr/lib/python3.6/dist-packages/“ and then install the package again. Thanks!

## Reply by Aaky · 2024-03-27 14:23:18 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex. This fix worked. Now I can see hires_debug over voxl-portal.

On a seperate note, I need some hardware help over AR0144 camera. Actually while we were assembling this camera over our drone, the lens holder which is black color as shown in picture below came off from one side. 

![M0149_angle_1920x1920 (1).jpg](https://forum.modalai.com/assets/uploads/files/1711549132122-m0149_angle_1920x1920-1.jpg) 

In order to fix this, we poured some fevi quick onto it, this made it sturdy but now voxl isnt able to find this camera. Even voxl-camera-server -l says 1 camera detected which is TOF. 
I am aware this must be some hardware problem, but can you advice me anything else if I can check? 
Logs : 

```
voxl2:~$ voxl-camera-server -d 0
=================================================================
configuration for 2 cameras:

cam #0
    name:                tracking
    sensor type:         ar0144
    isEnabled:           1
    camId:               0
    camId2:              -1
    fps:                 30
    en_rotate:           0
    en_rotate2:          0

    en_preview:          1
    pre_width:           1280
    pre_height:          800
    en_raw_preview:      1

    en_small_video:      0
    small_video_width:   -1
    small_video_height:  -1

    en_large_video:      0
    large_video_width:   -1
    large_video_height:  -1

    en_snapshot:         0
    snap_width:          -1
    snap_height:         -1

    ae_mode:             lme_msv
    standby_enabled:     0
    decimator:           1
    independent_exposure:0

cam #1
    name:                tof
    sensor type:         pmd-tof
    isEnabled:           1
    camId:               1
    camId2:              -1
    fps:                 5
    en_rotate:           0
    en_rotate2:          0

    en_preview:          1
    pre_width:           224
    pre_height:          1557
    en_raw_preview:      1

    en_small_video:      0
    small_video_width:   -1
    small_video_height:  -1

    en_large_video:      0
    large_video_width:   -1
    large_video_height:  -1

    en_snapshot:         0
    snap_width:          -1
    snap_height:         -1

    ae_mode:             off
    standby_enabled:     0
    decimator:           5
    independent_exposure:0

=================================================================
DEBUG:   Attempting to open the hal module
DEBUG:   SUCCESS: Camera module opened on attempt 0
DEBUG:   ----------- Number of cameras: 1

DEBUG:   Cam idx: 0, Cam slot: 3, Slave Address: 0x007A, Sensor Id: 0x003D
GPS server Connected 
DEBUG:   Connected to cpu-monitor
DEBUG:   ------ voxl-camera-server: Starting 2 cameras
Starting Camera: tracking (id #0)
DEBUG:   Checking Gain limits for Camera: tracking
Using gain limits min: 54 max: 8000
DEBUG:   cam ID 0 checking for fmt:   37  w: 1280 h:  800 o:   0
DEBUG:   i:  0 fmt: 34 w: 176 h: 144 o:0
DEBUG:   i:  4 fmt: 34 w: 176 h: 144 o:1
DEBUG:   i:  8 fmt: 35 w: 176 h: 144 o:0
DEBUG:   i: 12 fmt: 35 w: 176 h: 144 o:1
DEBUG:   i: 16 fmt: 33 w: 176 h: 144 o:0
DEBUG:   i: 20 fmt: 37 w: 224 h:1557 o:0
DEBUG:   i: 24 fmt: 38 w: 224 h:1557 o:0
DEBUG:   i: 28 fmt: 32 w: 224 h:1557 o:0
DEBUG:   i: 32 fmt: 36 w: 224 h:1557 o:0
ERROR:   Camera 0 failed to find supported preview config: 1280x800
WARNING: Failed to start cam tracking due to invalid resolution
WARNING: assuming cam is missing and trying to compensate
Starting Camera: tof (originally id #1) with id offset: 1
DEBUG:   Checking Gain limits for Camera: tof
Using gain limits min: 54 max: 8000
DEBUG:   cam ID 0 checking for fmt:   38  w:  224 h: 1557 o:   0
DEBUG:   i:  0 fmt: 34 w: 176 h: 144 o:0
DEBUG:   i:  4 fmt: 34 w: 176 h: 144 o:1
DEBUG:   i:  8 fmt: 35 w: 176 h: 144 o:0
DEBUG:   i: 12 fmt: 35 w: 176 h: 144 o:1
DEBUG:   i: 16 fmt: 33 w: 176 h: 144 o:0
DEBUG:   i: 20 fmt: 37 w: 224 h:1557 o:0
DEBUG:   i: 24 fmt: 38 w: 224 h:1557 o:0
DEBUG:   i: 28 fmt: 32 w: 224 h:1557 o:0
DEBUG:   i: 32 fmt: 36 w: 224 h:1557 o:0
VERBOSE: Successfully found configuration match for camera 0: 224x1557
VERBOSE: Adding preview stream for camera: 0
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
VERBOSE: Opened GBM fd
gbm_create_device(156): Info: backend name is: msm_drm
VERBOSE: Created GBM device
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Converted gralloc flags 0x20900 to GBM flags 0x1400000
VERBOSE: Dumping GBM flags
DEBUG:   	Found flag GBM_BO_USAGE_CAMERA_WRITE_QTI	
DEBUG:   	Found flag GBM_BO_USAGE_HW_COMPOSER_QTI	
DEBUG:   Allocated BO with width=224 height=1557 stride=336 aligned_w=336 aligned_h=1557 size=524288 flags=0x20900 format=GBM_FORMAT_RAW12
DEBUG:   Successfully set up pipeline for stream: PREVIEW
VERBOSE: Entered thread: cam0-request(tid: 2856)
DEBUG:   Started Camera: tof

------ voxl-camera-server: Started 1 of 2 cameras

------ voxl-camera-server: Camera server is now running
VERBOSE: Entered thread: cam0-result(tid: 2857)
VERBOSE: Found Royale module config: imagerType 2, illuminationConfig.dutyCycle: 4 temp_sensor_type: 5
VERBOSE: Found Royale module config: maxImgW 224 maxImgH 172 frameTxMode 1 camName X1.1_850nm_2W
VERBOSE: Found Royale module config: tempLimitSoft 60.000000 tempLimitHard 65.000000 autoExpoSupported yes
VERBOSE: Found Royale usecase: MODE_9_5FPS - phases: 9 fps: 5
VERBOSE: Found Royale usecase: MODE_9_5FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_9_5FPS - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE_9_5FPS - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_limit[0] = (8 2200)
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_limit[1] = (8 2200)
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_limit[2] = (8 2200)
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_time[1] = 2200
VERBOSE: Found Royale usecase: MODE_9_5FPS - exp_time[2] = 2200
VERBOSE: Found Royale usecase: MODE_9_10FPS - phases: 9 fps: 10
VERBOSE: Found Royale usecase: MODE_9_10FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_9_10FPS - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE_9_10FPS - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_limit[0] = (8 1100)
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_limit[1] = (8 1100)
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_limit[2] = (8 1100)
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_time[1] = 1100
VERBOSE: Found Royale usecase: MODE_9_10FPS - exp_time[2] = 1100
VERBOSE: Found Royale usecase: MODE_9_15FPS - phases: 9 fps: 15
VERBOSE: Found Royale usecase: MODE_9_15FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_9_15FPS - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE_9_15FPS - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_limit[0] = (8 750)
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_limit[1] = (8 750)
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_limit[2] = (8 750)
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_time[1] = 750
VERBOSE: Found Royale usecase: MODE_9_15FPS - exp_time[2] = 750
VERBOSE: Found Royale usecase: MODE_9_20FPS - phases: 9 fps: 20
VERBOSE: Found Royale usecase: MODE_9_20FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_9_20FPS - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE_9_20FPS - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_limit[0] = (8 560)
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_limit[1] = (8 560)
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_limit[2] = (8 560)
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_time[1] = 560
VERBOSE: Found Royale usecase: MODE_9_20FPS - exp_time[2] = 560
VERBOSE: Found Royale usecase: MODE_9_30FPS - phases: 9 fps: 30
VERBOSE: Found Royale usecase: MODE_9_30FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_9_30FPS - exposure group[1] = mod1
VERBOSE: Found Royale usecase: MODE_9_30FPS - exposure group[2] = mod2
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_limit[0] = (8 370)
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_limit[1] = (8 370)
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_limit[2] = (8 370)
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_time[1] = 370
VERBOSE: Found Royale usecase: MODE_9_30FPS - exp_time[2] = 370
VERBOSE: Found Royale usecase: MODE_5_15FPS - phases: 5 fps: 15
VERBOSE: Found Royale usecase: MODE_5_15FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_5_15FPS - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE_5_15FPS - exp_limit[0] = (8 1500)
VERBOSE: Found Royale usecase: MODE_5_15FPS - exp_limit[1] = (8 1500)
VERBOSE: Found Royale usecase: MODE_5_15FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_5_15FPS - exp_time[1] = 1500
VERBOSE: Found Royale usecase: MODE_5_30FPS - phases: 5 fps: 30
VERBOSE: Found Royale usecase: MODE_5_30FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_5_30FPS - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE_5_30FPS - exp_limit[0] = (8 750)
VERBOSE: Found Royale usecase: MODE_5_30FPS - exp_limit[1] = (8 750)
VERBOSE: Found Royale usecase: MODE_5_30FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_5_30FPS - exp_time[1] = 750
VERBOSE: Found Royale usecase: MODE_5_45FPS - phases: 5 fps: 45
VERBOSE: Found Royale usecase: MODE_5_45FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_5_45FPS - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE_5_45FPS - exp_limit[0] = (8 500)
VERBOSE: Found Royale usecase: MODE_5_45FPS - exp_limit[1] = (8 500)
VERBOSE: Found Royale usecase: MODE_5_45FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_5_45FPS - exp_time[1] = 500
VERBOSE: Found Royale usecase: MODE_5_60FPS - phases: 5 fps: 60
VERBOSE: Found Royale usecase: MODE_5_60FPS - exposure group[0] = gray
VERBOSE: Found Royale usecase: MODE_5_60FPS - exposure group[1] = mod
VERBOSE: Found Royale usecase: MODE_5_60FPS - exp_limit[0] = (8 370)
VERBOSE: Found Royale usecase: MODE_5_60FPS - exp_limit[1] = (8 370)
VERBOSE: Found Royale usecase: MODE_5_60FPS - exp_time[0] = 200
VERBOSE: Found Royale usecase: MODE_5_60FPS - exp_time[1] = 370
VERBOSE: sid: 0x00AC, addr: 0x0000, data: 0x0050
VERBOSE: sid: 0x00AC, addr: 0x0001, data: 0x004D
VERBOSE: sid: 0x00AC, addr: 0x0002, data: 0x0044
VERBOSE: sid: 0x00AC, addr: 0x0003, data: 0x0054
VERBOSE: sid: 0x00AC, addr: 0x0004, data: 0x0045
VERBOSE: sid: 0x00AC, addr: 0x0005, data: 0x0043
VERBOSE: sid: 0x00AC, addr: 0x0006, data: 0x0007
VERBOSE: sid: 0x00AC, addr: 0x0007, data: 0x0000

```

I dont know if we can recover this camera back it would be really helpful if someone can guide me if we can get some information about this camera also if in future this holder comes out, what should be used to fix this. My observations were due to his lens holder coming out our image was becoming completely blurry even after focusing it, also image quality reduced alot. What are better ways to fix this kind of lenses? Also better way to know if this camera has gone completely bad or can be recovered?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 21:33:19 UTC (in reply to Aaky)

@Aaky , you should check if any components got knocked off , which are sitting on the flex cable right next to the camera lens holder. If something got knocked loose, maybe you could attempt to replace a component (I could tell you what the component is, if there is a specific one that got knocked off).

Besides this, I do not have any other suggestions, unfortunately. It may be worth just plugging in another camera into that camera slot on VOXL2 just to make sure the camera port is still works.

Alex

## Reply by Aaky · 2024-03-28 03:15:45 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for the suggestion Alex. 
My doubt is, Can Adhesive like Fevi quick damage the PCB or components over it? I was under impression fevi quick acts as good insulator and wouldn't cause any problems but it seems to be have gone wrong.

In future if something like this happens, what should be used for sticking the holder back?

Quick Update: Camera seems to be working but unable to focus it now. I might have to clean the surface of CMOS as well. Any suggestions in this? Current image is extremely out of focused and looks very blurry.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-28 03:37:39 UTC (in reply to Aaky)

Hi @Aaky , my guess is that CA-based super glue should not damage pcb components in general (at least in short term). I dont know about this particular glue, but it should be a good insulator. It is probably not officially approved for electronics though :)

My worry about such glue is that its fumes (while drying) could deposit residue or damage the Image sensor surface. Super glue sometimes leaves white residue around application site.

Maybe this is what happened, you can take a look at the sensor surface..

I am curious how the lens holder came off - was a lot of force applied to it? I dont think we have had this happen before.

## Reply by Aaky · 2024-03-28 03:52:24 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev While assembling the sensor, it might have got some force onto it since our drone previously had OV7251 mount and this camera is significantly bigger in size. 

How should I clean the sensor surface? Any particular liquid is recommended? I have lens cleaner with me does that work? [Link](https://aqualens.in/products/spectacle-lens-cleaner).

Let me know if its safe to use this over sensor surface and also over lens surface.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-28 04:26:57 UTC (in reply to Aaky)

@Aaky , I am not sure about that, i can check, but it should be a cleaning solution approved for cleaning camera sensors.

## Reply by Aaky · 2024-03-28 08:30:28 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex. I managed to cleanup the lens and also is in the process of focusing the camera. Please look at below image. Is this ideal focus or can be done better? The checker board was 2.5 - 3 meters away from camera in this scenario. 
![hires_debug_1711613806.png](https://forum.modalai.com/assets/uploads/files/1711614620884-hires_debug_1711613806.png)

## Reply by Aaky · 2024-03-28 13:40:40 UTC (in reply to Aaky)

@Alex-Kushleyev One more problem which I am facing is, with above shown focus I am trying to perform camera calibration but it is always failing. On the same checker board I did calibration of new OV7251 and it worked well providing reprojection error of 0.34 at the best. With AR0144 I am always getting reprojection error of around 0.6 to 0.8 and never going below 0.5. Can this also be somehow related to focus or the repair which I did manually?

Please let me know your thoughts on both these queries.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-28 13:47:09 UTC (in reply to Aaky)

@Aaky ,

I think the focus looks ok. 

Keep in mind AR0144 has 2x resolution and reprojection error is reported in pixels, so 2x larger reprojection error for AR0144 roughly represents the same error as ov7251 in terms of angle that each pixel represents.

I will double check with the team what reprojection error is expected on AR0144

By the way, how did you resolve the issue of camera not being detected?

## Reply by Aaky · 2024-03-28 14:03:49 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Yes I will be waiting for your input on reprojection error on AR0144. 

Camera was not detecting maybe because the glue which I used was wet yesterday night. In the morning, probably after 10 hours when I checked camera was working normally. This might be only due to glue got enough time to dry off.

Also just FYI, I remember first time when I got AR0144 and I performed calibration I did get reprojection error below 0.5.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-28 16:02:43 UTC (in reply to Aaky)

@Aaky , our team provided feedback that typically we get around 0.6 pixel reprojection error when calibrating AR0144, so this is consistent with your results. At least, it seems that your camera is not performing worse than expected. We do see higher variance in reprojection error when calibrated multiple times (compared to ov7251). This could be due to the convergence criteria of the calibration procedure - we did not do a thorough investigation.

## Reply by Aaky · 2024-03-28 16:50:22 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Okay. Thanks for the feedback. In that case should I increase the reprojection criteria in voxl-camera-calibration package and save the calibration below 0.6 reprojection? Any other suggestions are welcome. I will perform handheld qvio tests also with this calibration. Any specific error I should be looking for in qvio? Like BAD_CAM_CAL?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-28 19:06:39 UTC (in reply to Aaky)

@Aaky , yes you can adjust the threshold, since the threshold was tuned for ov7251. Reprojection error of 0.5 or 0.6 is not going to matter too much.

QVIO will complain about BAD_CAM_CAL if the calibration is REALLY bad, so it probably wont be the case here. 

The ultimate metric of QVIO is drift vs distance traveled (and stability), in good feature rich environments it could get as low as few % drift over distance.

## Reply by Aaky · 2024-03-29 14:54:55 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for your feedback Alex. I got re-projection error of 0.51 on AR0144 and now I have started to test QVIO in indoors. On few instances I got BAD_CAM_CAL while testing handheld. My MSV was around 70 and extrinsic is caliberated correctly as per my UAV's IMU and camera positions. Please look at below image. I didnt got any error other then BAD_CAM_CAL (ofcourse discounting IMU and vibration errors)

As a observation I didnt see odometry drift in QVIO while I roam around and came back to same origin where my odometry said same x,y and z axis to be almost 0.

![00467.png](https://forum.modalai.com/assets/uploads/files/1711723981729-00467.png) 

Even if Quality in above image looks fine, there are good enough points for tracking still why does QVIO report BAD_CAM_CAL? I am aware QVIO is blackbox for everyone but just wanted your analysis.

Also on a very seperate note, I read somewhere on forum there is some work happening on dual camera support for QVIO? Can you provide me any early stage update on same? Even how is the software architecture for dual QVIO instance and fusion?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-29 16:09:46 UTC (in reply to Aaky)

@Aaky ,

I am not sure why QVIO is reporting bad cam cal. Please double check that you entered the camera calibration parameters correctly for QVIO.

In your test image, this is actually not a great scenario for VIO because all the points are roughly in the same location and at the same distance. Try to test in a more feature rich environment and see if the bad camera calibration warning is still there.

Sometimes it is possible that something else is off and the algorithm "thinks" that it is caused by a bad camera calibration. It is trying to provide a solution within a large state space (many parameters) so it could incorrectly estimate some parameter and think the camera calibration is off.

We are *not* working on a dual QVIO fusion, but we are working on another flavor of VIO algorithm that supports dual cameras.

Alex

## Reply by Aaky · 2024-03-29 16:16:08 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex. Will test this further. Any update over shading correction for corners of AR0144 camera stream as we discussed previously?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-04-01 19:57:01 UTC (in reply to Aaky)

@Aaky , 

Sorry, there is no update just yet. it is in the queue to experiment with, but we are testing with AR0144 with the same lens shading issues and it seems the performance is good. It could potentially be better with the shading corrected, but it is not a blocking issues at this point, as we understand. Just to be realistic, I would estimate at least a few weeks before I would have something to test for this feature.

Alex

## Reply by Aaky · 2024-04-02 07:02:02 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for the update. Can you provide me with camera parameters of AR0144 with which you got better results? In my tests I am getting good results in indoors but it fails in outdoors (imagine flat grass land) where in both cases camera is looking 35 degree downwards from horizon and ideally should be able to get some features. My camera reporjection error came out to be 0.51. If you can share any test bench data and parameters with which you tested QVIO with AR0144 will act as good reference point for my development. 
Thank you!

## Reply by Alex Kushleyev (ModalAI staff) · 2024-04-03 19:52:07 UTC (in reply to Aaky)

@Aaky , camera calibration parameters are specific to each camera module, so i don't think any other calibration parameters would work for you better than what you have. Does that make sense? Or are you asking about different parameters (please specify).

I will check to see if we can provide reference data sets running QVIO with AR0144

## Reply by Aaky · 2024-04-06 10:22:06 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Alex actually I am asking for voxl-camera-server related parameters for AR0144 and ofcourse not camera calibration parameters. Along with that,  like you mentioned even good test datasets recorded by you would be also beneficial with all camera server parameters and maybe qvio server parameters except intrinsic and extrinsic. Let me know.
Thank you!

## Reply by Gary Holmgren · 2024-06-04 20:59:09 UTC

having very similar issues to you Aaky, qvio does not like this M0149 camera whatsoever.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-05 14:23:19 UTC (in reply to Gary Holmgren)

@Gary-Holmgren , I replied to your new thread, we can discuss your issues there. https://forum.modalai.com/topic/3522/vio-errors-with-new-ar0144-tracking-camera-unstable-flight-crashing

## Reply by Aaky · 2024-06-11 18:20:18 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev I wanted one quick help. I am running my drone with AR0144 in dusty environment, imagine an under construction building with lots of dust around and with UAV's prop wash it increases more and more. VIO resets more often in this situation. I am totally aware computer vision algorithms wont work if its unable to find any features, but can the camera parameter be tuned to make it work in dusty environment? I have very important demo tomorrow with this setup and if you can help me to identify if anything can be done with AR0144 camera would be highly appreciated. My MSV in camera-server configuration is set to 80 since my environment would be bit darker. Let me know apart from this if anything can be done.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-11 19:30:19 UTC (in reply to Aaky)

@Aaky , as you already mentioned, dusty environments can very difficult for computer vision, I would say this is more like an open-ended research topic as opposed to parameter tuning task in current VIO. I imagine, you would need to use a different approach to feature detection and tracking, depending on the particular environment. 

If you wanted to try to push VIO to the limit, my suggestion would be to do this offline with collected data logs and playback (raw images and IMU data) running in the same environment at different exposures. During data collection, you could fly the drone manually or even do hand-held tests without spinning propellers, although spinning propellers would generate even more dust movement.

Without running offline tests and seeing how the algorithm performs with different parameters across many data sets, there is little hope to magically get it right and improve the performance. There is no shortcut here - it is a complicated problem and needs to be approached in a structured way :)

Also, we have never tested VIO in very dusty conditions (i am not sure how much dust you are talking about, but I am assuming significant, since you mentioned complete lack of features due to dust). Therefore, we don't have a quick suggestion how to improve performance in these conditions.

Alex

## Reply by Aaky · 2024-06-20 15:17:55 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex for your inputs.

On a seperate note, I wanted to discuss one thing. Can we expand current QVIO architecture to take input from two cameras? Or run two QVIO instances and fuse their odometries into PX4 (Maybe when one fails other Odometry would be fused and vice versa). Just wanted to check if any such kind of concept would work on current QVIO pipeline?

## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-20 17:14:50 UTC (in reply to Aaky)

@Aaky , please see the following comments:

- the QVIO architecture does not support multiple cameras, there is no way for us to make it work
- You can certainly set up and run multiple instances of QVIO using same IMU and two different cameras. These instances would not know anything about each other, that is not ideal use of the dual-camera set up, but could be used in conjunction with a higher-level fusion software to produce a single estimate. Setting up / running dual (independent) QVIO is the easier part, but then fusing that information is more complicated. However, we do not have any support of this approach (mainly because it has limited value).
  - if you REALLY want to try this approach, i can help you set it up (running two instances of QVIO and publishing the independent QVIO outputs), but would not be able to provide further guidance on actual fusing the outputs of the two instances of QVIO. Please note that we have not done this recently, so there could be some complications that i am not aware of. So there is some risk involved here.
- We are currently working on dual-camera VIO solution (using open vins), but it is not officially released yet. I will check the timeline for that..

Alex

## Reply by Aaky · 2024-06-20 23:58:42 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks Alex. I anticipate challenges in the fusion part for sure.
Regarding two camera based VIO using open vins, Any initial results you can share like it's performance as compared to QVIO?Also this will be part of future SDK releases and I hope we would be having full control over the codebase as a user? Timelines would be highly appreciated.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-25 03:30:14 UTC (in reply to Aaky)

@Aaky , I just checked the status of open vins and we are currently testing it on several drone configurations (using Starling drone) using single and dual cameras. However, the main issue is that documentation for using openvins on voxl2 in not ready, so we cannot really support it until the project is formally released. I don't have the ETA for the release - if I find out something, I will definitely let you know. 

Alex

## Reply by Aaky · 2024-06-25 04:29:39 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev Thanks for providing the status Alex. Can I get a prerelease version of open vins? Even that might be fruitful. I can figure out the bringup and executing it on VOXL2.

## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-26 17:15:51 UTC (in reply to Aaky)

@Aaky , I know this would be an exciting feature to get your hands on, but unfortunately we cannot release it until it is ready with documentation. We need to get it to a point that there is enough documentation so that it can be set up and tested without additional support, otherwise the support process will become very inefficient.

Our goal is to make the open vins application available with the next major SDK release, but I do not have the ETA yet (we are at the SDK 1.3). So it would be SDK 1.4 or whatever the next one will be called..

Alex

## Reply by Aaky · 2024-06-27 07:51:36 UTC (in reply to Alex Kushleyev)

@Alex-Kushleyev No problem Alex. Let me know once it's released. Thank you!

## Reply by Aaky · 2024-07-15 15:37:00 UTC (in reply to Aaky)

@Alex-Kushleyev Sorry to bug you again but is there any update on SDK releases with dual cam openvins?

## Reply by Aaky · 2024-07-17 07:32:50 UTC (in reply to Aaky)

@Alex-Kushleyev On a seperate note, Wanted to discuss one more point on dual cam QVIO pipeline itself.

Is there a real need to actually fuse the dual qvio instances odometry into PX4 together? Can they be mutually exclusive? Something like say if one qvio instance is failing, only then fuse second qvio's odometry? Since EKF in PX4 operates in global coordinate system and it shouldn't matter to EKF from which source the fusion is actually happening. Say this backup fusion source is present as a switch in voxl-vision-hub package which only sends the fusion source whenever needed to PX4 but only one at a time. Also while changing the source, we can set reset_counter flag which has good handling now in PX4 1.14 firmware. Any thoughts on this quick approach? (I aspire to get myself onto openvins anyways for dual cam due to more control over software pipeline but I am anticipating some delay in release so asking for this alternative)

## Reply by Aaky · 2024-08-14 06:00:21 UTC (in reply to Aaky)

@Alex-Kushleyev Sorry to bug you again. Actually I have urgent customer requirements which could be fulfilled with robust VIO which I feel is going to be with dual cam open vins approach.
 1. So I wanted to check with you if there is any update over dual cam open vins release? 
2. Also I have started my study over open source open vins algorithm and also would like to know some insights if there would be any modifications in open source open vins codebase or general camera and IMU driver integration? 
3. Dual camera part would need frame synchronisation if I am not wrong so is that part also handled in current open vins work?

It would really give me some pathway for further work if you can shade some light over here. Thank you!

## Reply by Aaky · 2024-12-09 04:05:33 UTC (in reply to Aaky)

@Alex-Kushleyev Hi Alex, I have one quick query on Open-vins. As mentioned on [this forum link](https://forum.modalai.com/topic/3893/timeline-for-open-vins-documentation/24?_=1733550563034), people are able to run open-vins on nightly builds successfully. Can you please guide me to link of nightly build of openvins? Is it included in latest SDK already or needs to be downloaded and installed?

Also seperately what are the timelines for Open-vins documentation and code to be released?

## Reply by tom (ModalAI staff) · 2024-12-09 17:39:51 UTC (in reply to Aaky)

@Aaky You'll have to use a nightly build of the SDK, here's a link to the latest one: https://storage.googleapis.com/platform-nightlies/voxl2/voxl2_SDK_nightly_20241208.tar.gz  

Keep in mind that these builds are untested so we can't promise any functionality