Fisheye stereo
-
I came across this SKU earlier today: https://www.modalai.com/products/m0072?variant=40414007132211
I have two questions. First, can this sensor substitute the standard tracking camera? It looks like the specs are the same. And the shorter flex cable would be very helpful in one of our integrations. Secondly, can you share any details on how to use it for stereo depth? How does it perform compared with the standard camera? How hard is it to calibrate?
Thanks!
-
Yes, the M0072 image sensor module can be used with the M0010 stereo flex cable.
The fisheye may be more difficult to calibrate, but will otherwise work in the same way as the more narrow lens.
See more here: https://docs.modalai.com/M0072/#voxl-2-integration
-
@Moderator thanks for the information.
Could one of the two M0072 be used on the VIO pipeline at the same time? I am thinking it would be great to have VIO and DFS from just two cameras.
-
Is there any documentation on how to calibrate the fisheye camera when used for stereo? Any example of performance (depth vs. baseline, minimum distance, etc.) would be much appreciated too
-
This post is deleted! -
@david-moro , the calibration procedure itself would be the same as the regular stereo calibration, you would just need to use
-fflag with thevoxl-calibrate-cameracommand. https://docs.modalai.com/calibrate-cameras/#stereo-calibration-processUnfortunately we don't have any performance metrics comparing the fisheye vs non-fisheye ov7251 stereo setups.
I can try calibrating the cameras using our standard calibration procedure, just to make sure that it works, if you are interested.
Regarding using one of the stereo cameras for VIO, yes, that should technically work, but i need to check something - the stereo image is merged (stacked) image that combines two cameras into one image, but QVIO expects a single image with correct dimensions of a single image. Basically, camera server would need to publish a separate image with just left or right in addition to the image with combined stereo. Alternatively, QVIO server could be updated to accept a stereo image. It looks like right now QVIO sever can accept a stereo image (https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/main.cpp?ref_type=heads#L609) but a few lines later it checks for image dimensions to match the dims in calibration file (there would be a mismatch because calibration is mono and image is stereo) .. I will need to double check this..
Alex
-
@Alex-Kushleyev , thanks for the thorough explanation. If you could do a quick and dirty check, that would be much appreciated!
-
@david-moro , I just confirmed that my concern regarding stereo input to vio is not actually an issue. The meta information for stereo image type actually has width and height equal to width and height of mono image, but the buffer contains two images. So, QVIO will just pick the first image if stereo is provided.
I will do a quick test to see how well the fisheye ov7251 calibration works.
Alex
-
With a few fixes, I was able to get the ov7251 fisheye stereo to work. Mainly needed to tweak the calibration procedure a bit, and add a fix to actually run fisheye stereo calibration instead of regular stereo calibration for the extrinsics. I will commit the fixes to the voxl-camera-calibration repository soon. The DFS part ran fine without any changes, after getting a proper camera calibration, but I will double check if I needed to change anything in DFS.
sample outputs:
narrow:
wide (not wearing the same shirt, so the pattern is different
)
I probably won't be able to run detailed analysis, but it looks like the DFS pipeline is working for fisheye use case. As you probably guessed, the fisheye setup will have shorter range but wider FOV, which can be estimated mathematically.
If you decide to go ahead and buy the fisheye cameras and try it, I just wanted to mention that there is still some risk in robustness of performance, since we have not really tested it until today. However, the results are promising.
Also, wanted to mention that the OV7251 are very old cameras. We have much better options available now, specifically AR0144 , which is 1280x800 resolution (monochrome), also global shutter. In order to use synchronized AR0144 cameras, you would need to use our new camera front end (M0173). Please see links below:
- https://docs.modalai.com/M0173/
- https://docs.modalai.com/voxl2-coax-camera-bundles/
- the best deal would probably be getting this C28 configuration (M0173 camera front end + dual AR0144 and dual IMX412 hires cameras + all the coax cables) : https://www.modalai.com/products/m0173?variant=48528274489648
- you can also buy these components separately, but the bundle seems like a better deal (even though you did not ask for two hires cameras )
- the M0173 front end will take up camera connectors J6 and J7 on VOXL2 (J8 will still be available)
If you want to wait for 1-2 days i can send you a sample output of the AR0144 stereo DFS.
Finally, there is another option, which would plug in directly into the stereo flex that you already have for OV7251 : https://docs.modalai.com/M0113/ . It is OV9782 global shutter RGB camera 1280x800 resolution. It is similar to AR0144, although OV9782 is older sensor and we do not recommend for new designs. We do support it though.
Here is a sample image from OV9782 stereo pair, so you can get an idea of the difference. This module does not have a fisheye lens, but it is much wider than the narrow OV7251. Also, the higher resolution makes a big difference.
Alex
-
@Alex-Kushleyev , this is fantastic. Thanks so much!
