Image Stabilization calibration and pipe size clarification

jameskuesel

Working with EIS a bit more and was hoping for some clarification/advice on a couple things

First, want to clarify calibration for EIS. In the doc seems like it’s fine to calibrate at the full 4040x3040 (via disabling eis and setting misp_width and misp_height temporarily) or can alternatively calibrate at half-res at 2020x1520. Could I also hypothetically calibrate at 1010x760? We have our own calibration routine and the smaller the faster it can run.

I’m assuming as long as the aspect ratio is the same it should be fine?   Furthermore, if I were to use the alternate resolution of 1996x1520 I should obviously do a calibration at that resolution? Or, keeping aspect ratio perhaps at 998x760?

Basically, just want to confirm that as long as a calibrate at some scale of whatever I set the preview frame to should be fine? And once I'm done calibrating I'm ok to then set misp_width and misp_height to my desired output resolution? This leads to my next clarification

I also wanted to clarify that EIS runs on whatever I set the preview frame to correct? As in that frame/resolution is what the algorithm runs on? misp_width and misp_height are just what gets outputted (as in its not like the algorithm is also getting run at that resolution)?

In that case what do you recommend eventually setting the output misp_width and misp_height to? Can they be anything? (Obviously something small enough to stream for example) I noticed in one example you chose 1280x720 but notably that is not the same aspect ratio as 4040x3040 or 1996x1520. Will stretching or warping occur? Should I pick something that is the same aspect ratio? Or does that not matter?

Any guidance much appreciated, thanks!

Alex Kushleyev

Hi @jameskuesel ,

When you enable en_raw_preview, which is what we do when MISP is enabled, the preview_width and preview_height basically force the selection of a specific camera mode with those dimensions. This means that the camera will send a bayer image of that size.

After the image is received by the ISP (which does nothing else in RAW-only mode), then MISP consumes the raw bayer image and performs debayering on the GPU. Then misp_width and misp_height will be used to specify the dimensions of the output image, sampled from the original bayer image.

If the output dimensions have exactly the same ratio as the input dimensions, then MISP will perform (arbitrary) down-scaling only (no crop = same horizontal and vertical FOV). However, if the output dims have different aspect ratio, the output image will be a cropped (and downscaled) version of the original image, such that the width and height ratio is maintained (features are not stretched). The crop is selected to maximize the fit of the output image within the input image.

With that in mind, if you select 4040x3040 as preview resolution and select misp resolution such that both width and height are scaled by the same factor (does not have to be integer), you can calibrate the intrinsics using the small image and after the calibration you can upscale the intrinsics:

multiply the principal point offsets by the same factor
multiply the focal length by the same factor
keep the same fisheye distortion coefficients (since they are a function of angle, not pixels)

See another version of this explanation in a different context : https://forum.modalai.com/topic/4900/running-qvio-on-a-hires-camera/12 (look for "intrinsics")

Input to the EIS algorithm is the full bayer image that is defined by the preview_width and preview_height.
the output image size is defined by misp_width and misp_height, just like when EIS is off. The misp resolution can be arbitrary
an important parameter is misp_zoom, which defines the relationship between the original image and the output image, in terms of zoom level. if you set zoom = 1.0 with EIS enabled, then the misp output will produce the largest undistorted image that fits within the original bayer image
there will be no deformation (stretching). the EIS ROI size will fit within the original bayer image and the zoom level is really controlling how much the FOV is reduced and how much stabilization margin you have.
you can enable the ROI display (PIP or side-by-side) to see how the EIS ROI fits within the original full image
the actual resolution of the misp output has nothing to do with how the EIS ROI fits within the original bayer image : the zoom value and fisheye undistortion dictate that. The output misp resolution just defines how many pixels the output image will contain, not the field of view (FOV).
for output resolution, you can pick whatever you need for your application.

By the way, you can have multiple EIS outputs on the same camera concurrently (MISP supports up to 4 channels). So you can have one stream that is small resolution (for streaming over wifi) and another one higher resolution (and can be different aspect ratio) for high quality recording. https://docs.modalai.com/camera-video/electronic-image-stabilization/#using-eis-with-misp-channels

Please note that currently, when you use multiple misp channels from the same camera, the image is processed separately, meaning if you have a large resolution and small resolution misp stream, the small resolution is NOT generated by downsampling the output of the large stream (which could be a nice optimization, in some cases).

Please let me know if you have any other questions.

Alex