Change the default frame rate of IMX412 and the image fov changes

AidanGallagher

Hi,

I have noticed that if I increase or decrease the frame rate of the IMX412 camera in voxl-camera-server.conf the fov of the images changes quite significantly.
It is as if a digital soon has been applied to the image.

Is there any way to prevent this - I assume is it driver related.

Any advice would be appreciated

Alex Kushleyev

@AidanGallagher ,

Can you please provide information about the resolution and frame rates you are using and better yet paste the whole section of the voxl-camera-server.conf that relates to that camera.

Also, please let me know which imx412 sensormodule you are using:

ls /usr/lib/camera/*sensormodule.imx412*
...

Alex

AidanGallagher

Hi @Alex-Kushleyev I have changed to the EIS drivers in order to test the MISP (you kindly gave me that information on another post) and I am not at all certain the same issue exists on these drivers.

I know that I was testing with the default small and large streams (1024x768 and 4056x3040) and if I change the FPS to any value other than 30fps the digital zoom effect occurred. I would need to go back to the default driver and settings to double check.

However, I am likely to stick with the new drivers as they are very impressive so we could park this question for now.

Regards
Aidan

Alex Kushleyev

@AidanGallagher , thanks for clarifying.

I will give you some background information that should help you understand what has happened before.

Since the camera itself can only output a single resolution and FPS at a time, the camera pipeline has to select one of several available modes / resolutions for the camera to operate. Since there are possibly multiple output streams when using Qualcomm ISP (yuv preview, small encoded, large encoded), there is some logic that looks at the resolutions of enabled streams and figures out what resolution should be picked for the requested FPS.

For example, if you request to streams : small encoded and large encoded, with resolutions 1280x720 and 1920x1080 respectively, and lets say fps is 30, the camera pipeline will try to pick resolution at least 1920x1080 in order to satisfy the largest requested output resolution. If the camera has both 1920x1080 and 3840x2160 available at 30fps, it will actually pick the larger one because the output quality will be a bit better and the ISP will do a good job downscaling from 4K to 1080p.

Now, if you then proceed to increase the FPS to 60, it is possible that the mode that was selected for 30fps is not actually available at 60FPS. Then the pipeline will try to find the best mode that the camera can provide, which can achieve 60FPS. It is sometimes the case that lower resolution / higher FPS modes are also cropped (in camera), so that is possibly why you were losing some FOV. If you needed to know exactly what is happening, then i need to know the specific resolutions and fps, etc and show you how to figur out which mode is selected, etc.

If you are using MISP and disable the Qualcomm ISP outputs, such as non-raw preview, small and large encoded, then the only stream that the camera pipeline is concerned about is the RAW stream, so the MISP input should be exactly the resolution and fps that you request using preview_width and preview_height. Then depending on the output of MISP, the MISP processing will downscale and crop (but crop only if needed). If the input and output are the same aspect ration, only down-scaling will be done.

I will also share with you that we recently added a feature to MISP that supports 3 additional output channels, individually enabled. In your IMX412 camera configuration, you can add the following entry (within the specific camera config) to enable up to 3 more channels with different resolutions, encoding type and bitrate. There are some limitations as the output channel fps cannot exceed the maximum fps specified by the top of the config for that camera.

So, together with the original misp output, this allows up to 4 output streams for the single camera (we kept the original misp output for backwards compatibility). The use case is that you can have different resolution streams and switch between them in real time (if there are no clients for the streams, they won't use cpu / gpu resources). The camera FPS is also adjusted based on the highest FPS of an active stream.

For each enabled output stream, you will have a new output "pipe" just like what you are already using.

Also when picking the output resolutions and fps, pay attention to the preview width and height you are selecting together with the FPS to make sure that is supported by the IMX412 driver. Also, MISP can automatically upscale to a larger resolution, but i don't know why you would want to do that.

This is an experimental feature, but feel free to check it out. the osd feature is not enabled, just a placeholder for now. You should use the latest camera server, which you can find in the dev package repository : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64//

"misp_channels":  [
                {
                    "enable":       1,
                    "name":         "low_latency",
                    "width":        960,
                    "height":       540,
                    "fps":          120,
                    "mbps":         1.5,
                    "codec":        "h265",
                    "nPframes":     119,
                    "osd":          0
                },
                {
                    "enable":       1,
                    "name":         "long_range",
                    "width":        1280,
                    "height":       720,
                    "fps":          30,
                    "mbps":         2.5,
                    "codec":        "h265",
                    "nPframes":     29,
                    "osd":          0
                },
                {
                    "enable":       1,
                    "name":         "high_quality",
                    "width":        1920,
                    "height":       1080,
                    "fps":          60,
                    "mbps":         5.0,
                    "codec":        "h265",
                    "nPframes":     59,
                    "osd":          0
                }

AidanGallagher

Thanks a million @Alex-Kushleyev that certainly explains the effect - I really appreciate you taking the time