(max) framerate of M0149 and M0061-2 ?
-
Hello,
we are working with image sensors M0149 and M0061-2.
Platform: Voxl2-MiniWhat are the supported (max) framerates for each sensor? The datasheet does not mention that.
https://docs.modalai.com/M0149/
https://docs.modalai.com/M0061/Thank you
-
We support the following modes for IMX412 cameras:
- Full frame 4056x3040 : 30FPS, possibly 60FPS (need to double check)
- 4K 60FPS
- 1080P 180FPS (maybe 240 FPS, need to double check)
- 720P (cropped) 240FPS
AR0144 FPS is currently limited to 50FPS (camera maximum is 60fps, but we have seen some stability issues). Resolution for AR0144 is fixed at 1280x800.
If you provide your specific requirements, I can confirm whether that is supported. Are you looking for the h264/5 encoded video for IMX412 camera? What about AR0144 tracking cameras - just RAW?
Alex
-
@Alex-Kushleyev Thank you for the answer.
I am looking for FPS higher than 60. So 180 would be great, 240 even better.
Resolution is secondary.
Raw or H264/5 - we can work with both.
Platform is Voxl2-Mini.Which cameras could achieve that?
Thank you.
-
If you need a global shutter monochrome camera, then OV7251 can go up to 120 FPS at full resolution 640x480 (although we currently only support 90FPS). Supposedly it supports 160x120 @ 360fps (with binning and skipping) but we have never tested that. AR0144 (which is our latest global shutter camera) is currently configured for 1280x800 at max 50fps (maximum is 60fps) but it also supports binning and skipping, so perhaps lower resolution would be faster (we do not support that yet).
IMX412 is the best choice in terms of resolution + FPS. The camera is color and rolling shutter, currently we support
- 1920x1080 (1080p) at 120fps
- 1280x720 (720p) at 180fps
However I am pretty sure that both modes can go a little faster. The fastest we could achieve on IMX412 would be a 720p mode with some cropping on the camera (reduced FOV), which is 240FPS (that i have tested in the past).
If you going to try IMX412, I would suggest going with M0107 or M0161 (https://www.modalai.com/collections/image-sensors), depending on which type of cable you need that connects the camera to VOXL2 (flex cable or micro coax).
We do have another camera in the pipeline (IMX664), which can output 2704x1540 at 120fps. it has better low light sensitivity than IMX412. Unfortunately, IMX664 does not allow binning and increasing the FPS and it is not yet available for purchase.
Let me test to make sure we can do 240fps at 1280x720 cropped. I am sure that 180fps non cropped works. In terms of cropping, here is what happens on the camera depending on the output resolution:
- the full frame resolution of IMX412 is 4056x3040 (4:3 aspect ratio). Max FPS is about 60fps, may be a bit lower
- 3840x2160 (4K) is obtained by cropping (on camera) a little on the sides and a lot on top/bottom. Max FPS is at least 60 (tested 60)
- 1920x1080 (1080p) is obtained by binning (in camera ISP) the above 3840x2160 (no FOV loss) and we also increase FPS to at least 120 (less data to send). I believe 180fps should be possible if not 240 (technically 1920x1080 is 4x less data to send compared to 3820x2160, but i would need to verify this)
- 1280x720 can be obtained from the above 1080p by resizing (in camera ISP), but you lose a little bit quality / sharpness due to scaling - this mode currently supports up to 180fps. In this mode the camera still processes 1920x1080 size frame but uses camera's ISP to resize to 1280x720.
- 1280x720 can also be obtained by cropping the 1920x1080 to 1280x720 (in camera ISP), which allows the camera to process even less lines and bump the FPS to 240. but in this mode you are only using 2560 out of 4056 horizontal pixels and 1440 out of 3040 vertical pixels (binned in both dimensions to get 1280x720). That is large FOV loss.
Please note that maximum FPS mentioned above a bit vague (when i say at least 60 fps, etc or 180fps should be possible) because we have not formalized the high fps parameters for the extreme cases (to squeeze out the maximum fps).
I will double check max fps in the following
- 1920x1080 (crop and bin from 4056x3040) : currently 120fps
- 1280x720 resized on camera from above 1920x1080 : currently 180fps
- 1280x720 (binned + large crop) : currently disabled (but i can re-enable this mode and check 240fps).
If you need even faster FPS, this could be achieved by reducing the vertical FOV even further. I have never tested above 240fps, but it should technically be possible on VOXL2. If you are interested in a mode with even less FOV, i can try to push it to 300 or even higher by reducing the frame height.
Also how many high fps cameras do you need connected to the same board?
Alex
-
thank you very much for explaining in detail. This helps a lot.
That means I would go with "M0107"
https://www.modalai.com/collections/image-sensors/products/msu-m0107?variant=45747768557872- the datasheet does not confirm Voxl2-Mini as "Compatible Products" - but I assume you confirmed that in your reply
If you could confirm the stated approximate fps, that would be great:
- 1920x1080 (crop and bin from 4056x3040) : currently 120fps
- 1280x720 resized on camera from above 1920x1080 : currently 180fps
- 1280x720 (binned + large crop) : currently disabled (but i can re-enable this mode and check 240fps).
Currently I need only one high fps camera connected to the same board.
Thank you
-
Hi @extmoddrv ,
Yes M0107 is supported by VOXL2 Mini.
In order to connect it you will need
- M0107 (camera itself). Choose with IR filter or no IR filter
- M0036 (14.5cm) or M0074 (6cm) extension flex that goes between the camera and interposer (see below). In general the shorter the cables the better (to reduce chance of interference), but both work (I personally use the long one for testing M0107)
- M0135 (dual) or M0076 (single) interposer that plugs into VOXL2 (mini)
The two resolutions / FPS below are definitely supported now:
- 1920x1080 (crop and bin on camera from 4056x3040) : currently 120fps
- 1280x720 resized on camera from above 1920x1080 : currently 180fps
I will be able to check how fast we can push the FPS, but it will probably be in a week or so due to holidays.
Alex
-
@Alex-Kushleyev Thank you, understood ...
-
@extmoddrv , I just did some quick testing and tried to push the limits of IMX412. At the moment i was testing M0161, which is the micro coax version of the IMX412 camera.
I was able to get the following modes working besides the standard ones that we already support:
- 1920x1080 @ 240FPS
- 1920x480 @ 480 FPS !!
The 1920x1080@240 mode is just 2x2 binned and cropped from the full frame image.
The 1920x480 mode is also 2x2 cropped but also every other image row is skipped (and slightly smaller vertical FOV).
Please note that these are just preliminary results and i have not tested stability over time (although i have not had issues). It is possible that we may need to back off some of these FPS numbers in order to avoid overloading the system. Also, depending on what else you will be doing on VOXL2 and what you will be doing with the high fps images, VOXL2 may not be able to keep up. However, just recording the encoded video to a file (which includes running auto exposure algorithm at 480fps!! , which i will need to throtlle..) the system load was about 15% of the total cpu usage of all cores.
I am attaching link to a video of a 1000Hz LED counter captured with the 480fps mode. The video has been also resized from 1920x480 to 960x480 in real time on GPU before encoding on voxl2) . There are two digits showing first two digits of the counter and a dot on top represents 3rd digit of the counter (it moves across 10 spots instead of showing the full 3rd digit). The counter increments every 1ms.
The exposure time is set to 2ms and this is a relatively low light indoor environment, however performance is actually pretty good, i think.
https://storage.googleapis.com/modalai_public/temp/imx412_480fps_h265.mp4 . The video is set to playback at 30fps but was recorded at 480fps.
I am going to repeat the test on the M0107 version of the camera and do some more testing to check stability. Please note that it may be a while before these high FPS modes are officially supported but i can guide you how to set it up with custom camera drivers and camera server configuration.
Alex
-
@Alex-Kushleyev great news, thank you! These fps are sufficient for our usecase. I will order the sensors and message you again in case we need support with camera drivers and camera server configuration.
Thank you very much for your efforts. I appreciate that
-
@extmoddrv , yes you WILL need additional instructions for testing higher FPS
I just confirmed that M0107 version of IMX412 works at 480fps as well (tested M0107 on voxl2 mini with M0036 extension flex).
I did notice that if you try to view the color images using voxl-portal at this high rate, it seems the cpu is loaded too much (due to using software jpeg encoder to stream the images to the browser) and camera server aborts. However, when using
voxl-inspect-cam, it reports stable 480fps and also i was able to get the encoded 480fps out and recorded just fine. You probably don't want to stream 480fps to your browser using mjpeg (which is what voxl-portal is doing).Depending on the additional processing on voxl2, you may need to cut down the FPS a bit or if you do need real-time processing, it should be optimized.
We are starting to make use of GPU for real-time image processing in the camera server, so depending on what you need to do with the images, perhaps the processing can be moved to the GPU.
When you order the camera(s), I suggest getting 1-2 extra flex extension cables. In the past we have seen that occasionally some flex cables do not work consistently when you try to push the limits like this. I would not want you to get stuck because of a cable that may be slightly out of spec. Even though i have not seen this issue in a while, it will not hurt to have an extra flex cable, since we are exploring somewhat uncharted territory in terms of FPS.
Once you get the camera, please let me know, i will provide instructions how to test the high FPS use case.
Alex
-
@Alex-Kushleyev thank you, understood. Will do so. Looking forward to it
Kind regards
