@Moderator said in Thermal Camera to VOXL2:

@H-AL first you need to determine if the sensor's electrical interface is compatible. What interface exists on the sensor? Can you find an appropriate electrical connection on VOXL 2?

Then, you need to find if there is software compatibility, does the driver current exist in ModalAI's reference code? Or, is there an implementation you can leverage for your work?

What thermal camera do you recommend using?

@Jetson-Nano ,

Can you please double check your /etc/modalai/voxl-camera-server.conf to make sure that the tracking camera has "en_raw_preview" set to true?

It is hard to tell whether there is haze in the image (the tracking camera has 640x480 resolution and if you zoom in, it may appear that the details are blurry). If you have another tracking camera to compare to, please try. Also please try looking at the image from voxl-portal (original tracking image and overlay image).

Alex

@tom said in Can someone tell me which camera is used in this video?:

Flir Boson connected over USB

Thank you!

@Moderator thanks for the information.

Could one of the two M0072 be used on the VIO pipeline at the same time? I am thinking it would be great to have VIO and DFS from just two cameras.

@linus777 , the instructions you referenced were made for VOXL1, which had all cameras connected to the same i2c bus and slave address conflicts were not allowed. On VOXL2, each camera port has its own i2c bus and there is no possibility of a conflict when using several cameras with the same i2c slave addresses.

Is your question related to usage on VOXL1 or VOXL2?

Alex

@Jetson-Nano , the ToF V1 calibration file is downloaded from the sensor itself via i2c . No internet connection is required.

@TomP , if you look at C19 configuration (https://docs.modalai.com/voxl2-camera-configs/#c-17-tof-tracking-stereo0v9782-and-hiresimx412) , the IMX412 camera is connected to the other camera port on the dual camera flex. You have it connected to the connector that goes straight out (7JU), C19 shows IMX412 connected to the port going off to the side (7JL).

If you want to keep IMX412 connected as shown in your picture, please copy the following file, which will tell the camera pipeline to try to detect the IMX412 camera on 7JU:

cp /usr/share/modalai/chi-cdk/imx412/com.qti.sensormodule.imx412_3.bin /usr/lib/camera/

Please double check this 🙂

@zcurtis070 , I am sorry for the delay.

I would like to help you with this but i need some more information. Does the white balance appear to ever work properly outdoors? (like sometimes it works fine and sometimes does not "kick in"?)

Also, are you able to re-build voxl-camera-server by yourself, i may have a few suggestions for you to try (different white balance modes), it would be easier if you can make the change in the code and re-build the camera server yourself.

Additionally, i am going to look into adjusting the RGB gains on the camera itself so that the image comes back more correct in the first place so that AWB has easier time doing a better job.

While we are trying to get this fixed, you may also consider pre-processing the image by running a white balance algorithm (there are some options if you search online) to normalize the colors. I know that ideally you would get a well-balanced image, but if that is not working well, you can try to do it yourself.

A nice read on this topic : https://pyimagesearch.com/2021/02/15/automatic-color-correction-with-opencv-and-python/

Another example : https://jmanansala.medium.com/image-processing-with-python-color-correction-using-white-balancing-6c6c749886de

Alex

Hi @Leon-Vaidman ,

We offer several high resolution cameras:

IMX214 (4208x3120) : 13Mpix IMX412 (4056x3040) : 12Mpix IMX678 (3840x2160) : 8Mpix

We are looking into supporting a 64Mpix camera, but it is not yet available.

There is no problem getting RAW10 frames at 30fps from 2 cameras. I believe the limitation will come from saving the frames to VOXL2 storage (which may not be an issue if your target FPS is very low). Each 4056x3040 frame is 4056 * 3040 * 1.2= 14.8MB, so 30FPS would be around 440MB/s.

I have set up a test where i connected two IMX214 cameras and two IMX412 cameras and configured for maximum resolution and the system is able to grab 30fps from each camera using voxl-inspect-cam, however streaming all this data to the inspect tool puts too much stress on the system and is only possible for a short period of time.

| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | hires0_bayer |18534400 | 4056 | 3040 | 14.34 | 1330 | 73 | 70.6 | 30.1 | 4462.2 | RAW10 | hires1_bayer |19722240 | 4208 | 3120 | 33.00 | 800 | 74 | 104.8 | 30.3 | 4781.1 | RAW10 | hires2_bayer |18534400 | 4056 | 3040 | 17.20 | 1442 | 74 | 76.5 | 30.1 | 4466.4 | RAW10 | hires3_bayer |19722240 | 4208 | 3120 | 33.00 | 800 | 73 | 130.5 | 30.3 | 4780.8 | RAW10

Interacting with RAW frames of this resolution at high rate is not practical because even sending them from one process to another takes substantial cpu time. It seems that in your case the FPS will be very low, so that should not be an issue. if you wanted to save a few FPS RAW10 data from 2 cameras.

I have described, more or less, the current limit of VOXL2 capabilities in terms of getting RAW images. Please keep in mind if you wanted processed images (YUV or video), you can currently only get that for two cameras.

With this in mind, let me know please let me know what else you need to know or if you are ready to start testing, i can help you set this up. For low enough FPS, you may be able to use voxl-logger or voxl-record-raw-image tools or may need to modify them depending on your needs (I can help you with that).

Alex

@Alex-Kushleyev Thanks for getting back to me. For what I am working on setting a fixed whitebalance seems like it would work, but I would at least find it useful if the white balance were tune-able. For my application I'm using the Modal drone for supervising Gaussian Splat training with an image stream, and color consistency across frames is important for good reconstruction otherwise I'll have to do color correction in the map which is a pain. Not sure if a lot of other people are using the HiRes for scene reconstructions so maybe its not universally useful though!

@Magnus-Eide-Fredriksen

Please take a look at the following spectral sensitivity plots for the cameras we support:

IMX412: https://www.framos.com/wp-content/uploads/FSM-IMX412_V1B_Datasheet_v1.1c_Brief.pdf
IMX678: https://www.framos.com/wp-content/uploads/FSM-IMX678_V1A_Datasheet_v1.0d_Brief.pdf

It seems IMX678 is more sensitive at the 940nm (relative sensitivity at 940nm to normal color range). Also, IMX678 by itself has higher sensitivity / lower noise compared to IMX412.

We currently sell the following camera, which integrates the Framos module. It is not recommended for new designs, as quantities may be limited. https://www.modalai.com/collections/image-sensors/products/mdk-m0061-2

We are also working on a new version of IMX678 , similar to this IMX412 module : https://www.modalai.com/collections/image-sensors/products/msu-m0161 , but i don't have the ETA.

If you wanted to try something that we have now, the best option may be M0061 camera module, but please review the information I linked above.

Additionally, you can review the following page for comparison ; https://www.sony-semicon.com/en/products/is/security/security.html -- check the SNR1s (lx) *2 column for the two cameras, which is the "index used to quantitatively evaluate picture quality at low illumination" - basically the lower the number, the less pixel noise in low light conditions and IMX678 has about 4x lower quanity (which may or may not translate to 4x less noise, but something along those lines).

Please note the IMX678 has less pixels than IMX412 (8.3 vs 12.3 Mpix) and aspect ratios are different (16:9 vs 4:3).

Both IMX412 and IMX678 are supported by our SDK and we can help with testing if needed.

Alex

@wilkinsaf , IMX327 no, but IMX664 maybe 🙂

I think IMX664 is close to or better than IMX327 for low light, what do you think?

Alex

Hi @TomP
Absolutely No.
Do Not use our Micro Coax cables or adapter boards with ANY sensor other than provided by ModalAI. We spent months designing this new ecosystem of connectors, pinouts, and adapter boards with a ton of thought and care. Part of that is complete incompatibility with other vendors' products.
https://docs.modalai.com/micro-coax-user-guide/

Thanks!

@Jetson-Nano , just to add to what Vinny said and answer your specific question about vibration affecting the micro coax connectors. The difficulty associated with removal is separate from the connector’s robustness under vibration. Once the connectors are inserted (mated), they are difficult to disconnect without applying significant force that might actually break the connector (that is why a tool is required, which correctly separates the connector). This type of stress is very unlikely in typical applications, unless the drone crashes and camera or board is somehow ejected from the vehicle.

@LR If you just want to change the camera that is streaming then modify /etc/modalai/voxl-streamer.conf to specify the desired camera. If you want to stream two or more cameras then you need to start a voxl-streamer instance for each one and specify a different port for each one. Then use voxl-mavcam-manager to inform QGC of the availability of the multiple streams. See https://docs.modalai.com/voxl-mavcam-manager/

@Jetson-Nano The M0173 kits include the IR filter on all sensors.