ModalAI Forum

@Jetson-Nano The M0173 kits include the IR filter on all sensors.

@LR If you just want to change the camera that is streaming then modify /etc/modalai/voxl-streamer.conf to specify the desired camera. If you want to stream two or more cameras then you need to start a voxl-streamer instance for each one and specify a different port for each one. Then use voxl-mavcam-manager to inform QGC of the availability of the multiple streams. See https://docs.modalai.com/voxl-mavcam-manager/

Hi @TomP Absolutely No. Do Not use our Micro Coax cables or adapter boards with ANY sensor other than provided by ModalAI. We spent months designing this new ecosystem of connectors, pinouts, and adapter boards with a ton of thought and care. Part of that is complete incompatibility with other vendors' products. https://docs.modalai.com/micro-coax-user-guide/ Thanks!

@Jetson-Nano , just to add to what Vinny said and answer your specific question about vibration affecting the micro coax connectors. The difficulty associated with removal is separate from the connector’s robustness under vibration. Once the connectors are inserted (mated), they are difficult to disconnect without applying significant force that might actually break the connector (that is why a tool is required, which correctly separates the connector). This type of stress is very unlikely in typical applications, unless the drone crashes and camera or board is somehow ejected from the vehicle.

@wilkinsaf , IMX327 no, but IMX664 maybe I think IMX664 is close to or better than IMX327 for low light, what do you think? Alex

@Magnus-Eide-Fredriksen Please take a look at the following spectral sensitivity plots for the cameras we support: IMX412: https://www.framos.com/wp-content/uploads/FSM-IMX412_V1B_Datasheet_v1.1c_Brief.pdf IMX678: https://www.framos.com/wp-content/uploads/FSM-IMX678_V1A_Datasheet_v1.0d_Brief.pdf It seems IMX678 is more sensitive at the 940nm (relative sensitivity at 940nm to normal color range). Also, IMX678 by itself has higher sensitivity / lower noise compared to IMX412. We currently sell the following camera, which integrates the Framos module. It is not recommended for new designs, as quantities may be limited. https://www.modalai.com/collections/image-sensors/products/mdk-m0061-2 We are also working on a new version of IMX678 , similar to this IMX412 module : https://www.modalai.com/collections/image-sensors/products/msu-m0161 , but i don't have the ETA. If you wanted to try something that we have now, the best option may be M0061 camera module, but please review the information I linked above. Additionally, you can review the following page for comparison ; https://www.sony-semicon.com/en/products/is/security/security.html -- check the SNR1s (lx) *2 column for the two cameras, which is the "index used to quantitatively evaluate picture quality at low illumination" - basically the lower the number, the less pixel noise in low light conditions and IMX678 has about 4x lower quanity (which may or may not translate to 4x less noise, but something along those lines). Please note the IMX678 has less pixels than IMX412 (8.3 vs 12.3 Mpix) and aspect ratios are different (16:9 vs 4:3). Both IMX412 and IMX678 are supported by our SDK and we can help with testing if needed. Alex

@Alex-Kushleyev Thanks for getting back to me. For what I am working on setting a fixed whitebalance seems like it would work, but I would at least find it useful if the white balance were tune-able. For my application I'm using the Modal drone for supervising Gaussian Splat training with an image stream, and color consistency across frames is important for good reconstruction otherwise I'll have to do color correction in the map which is a pain. Not sure if a lot of other people are using the HiRes for scene reconstructions so maybe its not universally useful though!

Hi @Leon-Vaidman , We offer several high resolution cameras: IMX214 (4208x3120) : 13Mpix IMX412 (4056x3040) : 12Mpix IMX678 (3840x2160) : 8Mpix We are looking into supporting a 64Mpix camera, but it is not yet available. There is no problem getting RAW10 frames at 30fps from 2 cameras. I believe the limitation will come from saving the frames to VOXL2 storage (which may not be an issue if your target FPS is very low). Each 4056x3040 frame is 4056 * 3040 * 1.2= 14.8MB, so 30FPS would be around 440MB/s. I have set up a test where i connected two IMX214 cameras and two IMX412 cameras and configured for maximum resolution and the system is able to grab 30fps from each camera using voxl-inspect-cam, however streaming all this data to the inspect tool puts too much stress on the system and is only possible for a short period of time. | Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | hires0_bayer |18534400 | 4056 | 3040 | 14.34 | 1330 | 73 | 70.6 | 30.1 | 4462.2 | RAW10 | hires1_bayer |19722240 | 4208 | 3120 | 33.00 | 800 | 74 | 104.8 | 30.3 | 4781.1 | RAW10 | hires2_bayer |18534400 | 4056 | 3040 | 17.20 | 1442 | 74 | 76.5 | 30.1 | 4466.4 | RAW10 | hires3_bayer |19722240 | 4208 | 3120 | 33.00 | 800 | 73 | 130.5 | 30.3 | 4780.8 | RAW10 Interacting with RAW frames of this resolution at high rate is not practical because even sending them from one process to another takes substantial cpu time. It seems that in your case the FPS will be very low, so that should not be an issue. if you wanted to save a few FPS RAW10 data from 2 cameras. I have described, more or less, the current limit of VOXL2 capabilities in terms of getting RAW images. Please keep in mind if you wanted processed images (YUV or video), you can currently only get that for two cameras. With this in mind, let me know please let me know what else you need to know or if you are ready to start testing, i can help you set this up. For low enough FPS, you may be able to use voxl-logger or voxl-record-raw-image tools or may need to modify them depending on your needs (I can help you with that). Alex

@Alex-Kushleyev sorry for the delay, this sounds about right. I'm not sure about the 15 fps but we do use raw preview only for our application and no other stream this is my camera conf file: cat /etc/modalai/voxl-camera-server.conf /** * voxl-camera-server Configuration File * * Each camera has configurations for up to 4 HAL3 streams: * - `preview` stream for raw unprocessed images from CV cameras * - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming * - `large_video` 4k (ish) h264/h265 for onboard video recording to disk * - `snapshot` ISP-processed JPG snapshots that get saved to disk * * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled * * This file is generated from default values by voxl-configure-cameras. * Do not expect arbitrary resolutions to work, the ISP and video compression * pipelines only support very specific resolutions. * * The default video compression mode is cqp or Constant Quantization Parameter * * * */ { "version": 0.1, "cameras": [{ "type": "ov7251", "name": "stereo", "enabled": true, "camera_id": 0, "camera_id_second": 1, "independent_exposure": false, "fps": 30, "en_rotate": false, "en_rotate_second": false, "en_preview": true, "preview_width": 640, "preview_height": 480, "en_raw_preview": true, "ae_mode": "lme_msv", "ae_desired_msv": 60, "exposure_min_us": 20, "exposure_max_us": 33000, "gain_min": 54, "gain_max": 8000, "exposure_soft_min_us": 5000, "ae_filter_alpha": 0.600000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 1 }, { "type": "imx412", "name": "hires2", "enabled": true, "camera_id": 2, "fps": 3, "en_rotate": true, "en_preview": true, "preview_width": 4056, "preview_height": 3040, "en_raw_preview": false, "en_small_video": false, "small_video_width": 1024, "small_video_height": 768, "small_venc_mode": "h265", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "en_large_video": false, "large_video_width": 2048, "large_video_height": 1536, "large_venc_mode": "h265", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 38, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 30, "en_snapshot": false, "en_snapshot_width": 4056, "en_snapshot_height": 3040, "ae_mode": "lme_msv", "ae_desired_msv": 65, "exposure_min_us": 20, "exposure_max_us": 33000, "gain_min": 20, "gain_max": 250, "exposure_soft_min_us": 3000, "ae_filter_alpha": 0.700000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 0.5 }, { "type": "imx412", "name": "hires", "enabled": true, "camera_id": 3, "fps": 3, "en_preview": true, "preview_width": 4056, "preview_height": 3040, "en_raw_preview": false, "en_small_video": false, "small_video_width": 1024, "small_video_height": 768, "small_venc_mode": "h265", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "en_large_video": false, "large_video_width": 4056, "large_video_height": 3040, "large_venc_mode": "h265", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 38, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 30, "en_snapshot": false, "en_snapshot_width": 4056, "en_snapshot_height": 3040, "ae_mode": "lme_msv", "ae_desired_msv": 65, "exposure_min_us": 20, "exposure_max_us": 33000, "gain_min": 20, "gain_max": 250, "exposure_soft_min_us": 3000, "ae_filter_alpha": 0.700000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 0.5 }, { "type": "ov7251", "name": "tracking", "enabled": true, "camera_id": 4, "fps": 30, "en_rotate": false, "en_preview": true, "preview_width": 640, "preview_height": 480, "en_raw_preview": true, "ae_mode": "lme_msv", "ae_desired_msv": 60, "exposure_min_us": 20, "exposure_max_us": 33000, "gain_min": 54, "gain_max": 8000, "exposure_soft_min_us": 5000, "ae_filter_alpha": 0.600000023841858, "ae_ignore_fraction": 0.20000000298023224, "ae_slope": 0.05000000074505806, "ae_exposure_period": 1, "ae_gain_period": 1 }] } hope this helps

@TomP , if you look at C19 configuration (https://docs.modalai.com/voxl2-camera-configs/#c-17-tof-tracking-stereo0v9782-and-hiresimx412) , the IMX412 camera is connected to the other camera port on the dual camera flex. You have it connected to the connector that goes straight out (7JU), C19 shows IMX412 connected to the port going off to the side (7JL). If you want to keep IMX412 connected as shown in your picture, please copy the following file, which will tell the camera pipeline to try to detect the IMX412 camera on 7JU: cp /usr/share/modalai/chi-cdk/imx412/com.qti.sensormodule.imx412_3.bin /usr/lib/camera/ Please double check this

That pink hue usually ends up being related to white balance or ISP tuning rather than the sensor itself. I’ve also seen it happen under certain lighting conditions where the auto settings overcompensate a bit. Might be worth checking the raw output vs processed output to narrow it down. If the RAW image looks fine, then it’s probably somewhere in the color pipeline. Also, for comparing the tint between frames, I’ve used https://colorpickerimage.org/ before to inspect the actual RGB values in different areas of the image. Makes it easier to confirm whether the pink cast is consistent or only affecting highlights/shadows.

@Alex-Kushleyev , this is fantastic. Thanks so much!

@Jetson-Nano , the ToF V1 calibration file is downloaded from the sensor itself via i2c . No internet connection is required.

@tom said in Can someone tell me which camera is used in this video?: Flir Boson connected over USB Thank you!