# VOXL2 Camera Configuration 2XHires + Tracking Source: https://forum.modalai.com/topic/3178/voxl2-camera-configuration-2xhires-tracking Category: VOXL 2 and VOXL 2 Mini (https://forum.modalai.com/category/26/voxl-2-and-voxl-2-mini) Tags: voxl-2 Posted: 2024-03-07 14:28:16 UTC by jacob yaacubov Replies: 11 · Views: 2603 ## jacob yaacubov · 2024-03-07 14:28:16 UTC Hi all, have a small issue here haven't found this kind of setup on the voxl2 docs, is this technically possible ? its not a built in configuration .. what will this require to make it work ? can i create new config from scratch or based on existing ones? thanks ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-07 14:44:10 UTC @jacob-yaacubov , which hires cameras are you trying to connect and do you have a preference where to plug them in? I can test and provide a configuration for you. ## Reply by jacob yaacubov · 2024-03-09 12:28:24 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev thanks , i use the imx214, don't have a specific preference ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-11 03:26:38 UTC (in reply to jacob yaacubov) @jacob-yaacubov Here is a quick tutorial. - Connect two M0024 IMX214 cameras to VOXL2 J7 via MSU-M0084 dual camera flex adapter. Disconnect any other cameras for now to avoid confusion. - power on voxl - copy imx214 sensormodules for camera slots 2 and 3 because that is the assignment for J7 ``` cp /usr/share/modalai/chi-cdk/imx214/com.qti.sensormodule.imx214_2.bin /usr/lib/camera/ cp /usr/share/modalai/chi-cdk/imx214/com.qti.sensormodule.imx214_3.bin /usr/lib/camera/ ``` - copy the tuning file for imx214 (this tuning file is specific for M0024 / M0124), and it has some lens correction params specific to that lens. If you are using another variant IMX214, let me know, i can make a suggestion how to get good image quality. ``` cp /usr/share/modalai/chi-cdk/tuned/com.qti.tuned.imx214_m0024.bin /usr/lib/camera/com.qti.tuned.imx214.bin ``` quick test using camera server ``` voxl-camera-server -l DEBUG: Attempting to open the hal module DEBUG: SUCCESS: Camera module opened on attempt 0 DEBUG: ----------- Number of cameras: 2 DEBUG: Cam idx: 0, Cam slot: 2, Slave Address: 0x0020, Sensor Id: 0x0214 DEBUG: Cam idx: 1, Cam slot: 3, Slave Address: 0x0020, Sensor Id: 0x0214 ... ``` - update `/etc/modalai/voxl-camera-server.conf` ``` { "version": 0.1, "fsync_en": false, "fsync_gpio": 111, "cameras": [{ "type": "imx214", "name": "hires2", "enabled": true, "camera_id": 0, "fps": 30, "en_preview": true, "preview_width": 1920, "preview_height": 1080, "ae_mode": "isp", "en_raw_preview": false, "en_small_video": false, "small_video_width": 1280, "small_video_height": 720, "small_venc_mode": "h265", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "en_large_video": false, "large_video_width": 1920, "large_video_height": 1080, "large_venc_mode": "h265", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 38, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 30, "en_snapshot": false, "en_snapshot_width": 4208, "en_snapshot_height": 3120 }, { "type": "imx214", "name": "hires3", "enabled": true, "camera_id": 1, "fps": 30, "en_preview": true, "preview_width": 1920, "preview_height": 1080, "ae_mode": "isp", "en_raw_preview": false, "en_small_video": false, "small_video_width": 1280, "small_video_height": 720, "small_venc_mode": "h265", "small_venc_br_ctrl": "cqp", "small_venc_Qfixed": 30, "small_venc_Qmin": 15, "small_venc_Qmax": 40, "small_venc_nPframes": 9, "small_venc_mbps": 2, "en_large_video": false, "large_video_width": 1920, "large_video_height": 1080, "large_venc_mode": "h265", "large_venc_br_ctrl": "cqp", "large_venc_Qfixed": 38, "large_venc_Qmin": 15, "large_venc_Qmax": 50, "large_venc_nPframes": 29, "large_venc_mbps": 30, "en_snapshot": false, "en_snapshot_width": 4208, "en_snapshot_height": 3120 }] } ``` then start `voxl-camera-server` and you should be able to see the output of the two cameras in `voxl-portal`. I called them `hires2` and `hires3` corresponding to their hardware slot IDs. You can modify the `voxl-camera-server.conf` further to enable encoded streams. A few considerations - if you find any interference (extra noise or any glitches) while two hires cameras are connected to the same VOXL2 camera connector (e.g. J7), it may be best to split them up: connect one camera to J6 and another to J7. I have seen that sometimes i get a noticeable horizontal line a few pixels wide in one or both images, i suspect this maybe a short power spike when one camera starts acquiring an image while another camera is already in process of acquiring. I don't see this every time, but it's not something i have seen before when running a single hires camera, so i just wanted to mention that. - if you are changing sensor resolutions, you can always check which native camera resolution the camera pipeline has selected by running (before starting `voxl-camera-server`) : ``` logcat | grep -i selected ``` in our case, you will see `Selected sensor Mode W=1920, H=1080` since i requested 1920x1080 preview frame. - just FYI, if you use resolutions 1920x1080 or smaller, the camera will be in 2x2 binning mode with 4x sensitivity (good for low light conditions). While if you use large resolutions like 3840x2160, the camera will be in regular mode, not as sensitive in low light conditions. - if you request multiple resolutions, the camera native mode will be chosen to satisfy the largest requested resolution, as the camera can only stream raw images of only one size, so largest will be selected and smaller images will be obtained by downscaling in the isp. Once you get two hires cameras working, you can add the tracking camera as third camera, follow the same logic for figuring out slot ID and sensormodule that you need to copy. More details about hardware sensor IDs (i called them slot ids) here : https://docs.modalai.com/voxl2-camera-configs/#overview Lots of details here, try this out and let me know if you have any questions.. Alex ## Reply by jacob yaacubov · 2024-03-25 13:38:15 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thank you for the detailed reply i am checking this right now, regarding the slots assignment i might need to check other configurations so do have an idea where can i find which connector mapped to which slot ? haven't found this in the Docs in theory what is the maximum mun of image sensors (Not TOF) the Voxl2 will support? (i include the stereo as two separate sensors for the purpose of this question ) ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-25 15:20:40 UTC (in reply to jacob yaacubov) @jacob-yaacubov , You can find the slot information in this diagram here : https://docs.modalai.com/voxl2-camera-configs/#overview (also pasting it below). We use `hw sensor id` and `slot id` terms interchangeably. Upper / Lower is marked on the flex if you are using the dual-camera flex (JU / JL) ![1db04d34-6246-46cf-a7a8-a20a2814a8bb-image.png](https://forum.modalai.com/assets/uploads/files/1711379709656-1db04d34-6246-46cf-a7a8-a20a2814a8bb-image.png) In terms of maximum number of cameras.. this configuration https://docs.modalai.com/voxl2-camera-configs/#c-11-front-stereo-rear-stereo-hires-and-tracking-d0006-sentinel shows 6 cameras (1 hires, 5 ov7251). There is is a limit in terms of how many concurrent cameras can be using the video processing pipeline (outputting YUV or encoded streams). Let me check what that number is (at least 2). The number of RAW streams does not have this limit and will be limited by the physical hardware ports. If you have more specific camera configuration in mind, i can check that for you. ## Reply by jacob yaacubov · 2024-03-25 16:15:02 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Great i understand thank you very much, i understand the most extreme case i need to check is the following : two hi-res IMX214 sensor working at 5hz 2 sets of stereo ov7251 VGA - ( front & rear) - 30hz 1 - tracking - 30hz this is just one more HI-RES that the max you suggested i plan to use the Dual Image Sensor Expander to group 0 - 1 x Stereo , 1 x Tracking group 1 - 1 x HI-RES, 1 x Stereo group 2 - 1 x HI-RES if this is possible it will be great Again Thanks for the quick response ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-26 03:58:02 UTC (in reply to jacob yaacubov) @jacob-yaacubov , it does not look like a stereo pair can share the dual camera expander. i tried it and only one of the two sensors in the pair was detected. I will check why. However, i was able use the configuration 11 (https://docs.modalai.com/voxl2-camera-configs/#c11---front-stereo-rear-stereo-hires-and-tracking) and replace the tracking camera with a 2nd imx214 and it worked (2x imx214 and 2x stereo). A small change in camera config was required along with a copying of `com.qti.sensormodule.imx214_2.bin` from `/usr/share/modalai/chi-cdk/imx214/` to `/usr/lib/camera` (after running voxl-configure-cameras with C11) output of `voxl-inspect-cam -a` (please note that is a very stressful test to use `voxl-inspect-cam -a` because the inspect tool will pull all this data from `voxl-camera-server`, you can see it is over 10Gbit/second due to un-encoded frames. Normally you would not be sending 30fps 4K frames via MPA, but i think you mentioned 5Hz each, but did not specify format - yuv or encoded video). i named hires2 the camera connected to slot 2 and hires3 connected to slot 3. ``` Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | hires2_large_color |12441600 | 3840 | 2160 | 33.23 | 800 | 146 | 224.3 | 30.0 | 2986.9 | NV12 | hires2_large_encoded | 158743 | 3840 | 2160 | 33.23 | 800 | 147 | 212.6 | 30.0 | 38.2 | H265 (P) | hires2_large_grey | 8294400 | 3840 | 2160 | 33.23 | 800 | 148 | 177.9 | 30.0 | 1991.2 | RAW8 | hires3_large_color |12441600 | 3840 | 2160 | 11.11 | 575 | 140 | 307.2 | 30.0 | 2986.8 | NV12 | hires3_large_encoded | 96018 | 3840 | 2160 | 11.11 | 575 | 140 | 287.0 | 30.0 | 24.1 | H265 (P) | hires3_large_grey | 8294400 | 3840 | 2160 | 11.11 | 575 | 141 | 259.6 | 30.0 | 1991.3 | RAW8 | stereo_front | 614400 | 640 | 480 | 6.50 | 538 | 166 | 15.3 | 30.0 | 147.5 | STEREO_RAW8 | stereo_rear | 614400 | 640 | 480 | 12.49 | 1238 | 153 | 16.3 | 30.0 | 147.5 | STEREO_RAW8 ``` If i change the fps of both hires (in `voxl-camera-server.conf) to 5, the load is much lower, you can also see the latency for hires stream is reduced: ``` | Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format | hires2_large_color |12441600 | 3840 | 2160 | 66.67 | 800 | 108 | 79.8 | 5.0 | 497.9 | NV12 | hires2_large_encoded | 179368 | 3840 | 2160 | 66.67 | 800 | 108 | 89.8 | 5.0 | 7.2 | H265 (P) | hires2_large_grey | 8294400 | 3840 | 2160 | 66.67 | 800 | 108 | 73.7 | 5.0 | 331.9 | RAW8 | hires3_large_color |12441600 | 3840 | 2160 | 11.11 | 512 | 107 | 83.2 | 5.0 | 497.8 | NV12 | hires3_large_encoded | 62102 | 3840 | 2160 | 11.11 | 512 | 107 | 85.3 | 5.0 | 2.7 | H265 (P) | hires3_large_grey | 8294400 | 3840 | 2160 | 11.11 | 512 | 107 | 72.8 | 5.0 | 331.9 | RAW8 | stereo_front | 614400 | 640 | 480 | 6.47 | 531 | 669 | 12.1 | 30.0 | 147.4 | STEREO_RAW8 | stereo_rear | 614400 | 640 | 480 | 11.78 | 1194 | 657 | 11.5 | 30.0 | 147.5 | STEREO_RAW8 ``` output of `voxl-camera-server -l` (you can see two IMX214 cameras in slots 2 and 3 and the rest ov7251 in slots 0,1,3,4). Sensor ID for IMX214 is 0x0214 and for ov7251 is 0x7750: ``` DEBUG: ----------- Number of cameras: 6 DEBUG: Cam idx: 0, Cam slot: 0, Slave Address: 0x00E2, Sensor Id: 0x7750 DEBUG: Cam idx: 1, Cam slot: 1, Slave Address: 0x00E4, Sensor Id: 0x7750 DEBUG: Cam idx: 2, Cam slot: 2, Slave Address: 0x0020, Sensor Id: 0x0214 DEBUG: Cam idx: 3, Cam slot: 3, Slave Address: 0x0020, Sensor Id: 0x0214 DEBUG: Cam idx: 4, Cam slot: 4, Slave Address: 0x00E2, Sensor Id: 0x7750 DEBUG: Cam idx: 5, Cam slot: 5, Slave Address: 0x00E4, Sensor Id: 0x7750 ``` So there are some preliminary results.. I am not sure if 7-camera setup will work, but I will check. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-26 16:48:49 UTC (in reply to Alex Kushleyev) Quick update.. Currently on VOXL2 we support only 1 regular camera or 1 stereo pair on J6 and J8. J7 supports up to two regular cameras, so you can either have 4 single cameras (in slots 0, 2, 3, 4) or you can use stereo pair in slots 0 and 4 using M0076 adapters and single cameras in slots 2,3 . Support for more cameras is possible in hardware, but not yet implemented in software. ## Reply by jacob yaacubov · 2024-03-27 14:26:29 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thank You, i understand makes things clearer now since you mention software still does not support this... how complicated and even if its possible on my side to open J6 or J8 for One More single camera , or one more stereo?? since i see there is two different ID_SLOTS on them which make me wonder question 2: regarding the previous setup you inspected 2 X HIRES (5hz) J7 and 2 X Stereo (30hz) J6, J8 on the hires, we process the raw8 (yuv) format at 5hz (plan maybe at 3hz) **based on what you checked whats your verdict will the mpa support this or is this a stretch?** ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 22:14:05 UTC (in reply to jacob yaacubov) @jacob-yaacubov , regarding question2: yes, yuv is fine, it is already supported. To provide more details.. Actually the raw frames that come in from hires camera are 10bit Bayer format. If you want to debayer yourself, this is already supported in camera server (you can use as example), but you can also get YUV which goes through a camera processing inside the ISP (debayering, denoise, gamma correction, color correction, etc etc). There is an option whether to do a raw preview (which would give you the 10bit bayer format) or otherwise you will get YUV preview frame. "preview" is just a name of the processing pipeline and we typically use it for the uncompressed images, while the small / large video streams will still get YUVs and push them directly into hardware encoder to get h264/5). Currently there is a limit of 2 concurrent cameras which use the ISP for processing, such as hires camera. Stereo and tracking cameras are typically running in RAW only mode, so they don't really use the ISP. It seems that it should not be an issue for you, since you need only two hires cameras which would potentially use the ISP to generate YUV. I believe there is a way to get more processed streams but it is not yet possible using the camera server. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-03-27 22:26:52 UTC (in reply to Alex Kushleyev) Regarding enabling additional cameras, it is is not trivial and it would require a special kernel (because it would break the existing stereo camera support). Additionally, it would require a special interposer / cable in order to use a stereo pair with another camera on a single connector (this hardware does not exist now). At this time, we do not have plans to enable this functionality. As an option, you could potentially hook up a USB camera (will require an usb expansion adapter) to supplement existing cameras (using UVC interface), but we don't provide as much support for that option since that is 3rd party hardware and software.