AR0144 RGB output on VOXL2

Jordyn Heil

@Alex-Kushleyev, with the smaller number of cameras I can connect to the VOXL2, I’d like to increase the field of view of each AR0144 to maximize coverage. From my research, I understand the AR0144 sensor modules use a 1/4" optical format and are compatible with M12 lenses. Can you confirm if that’s correct?

Also, how well do the AR0144-based camera modules (like the M0166) perform with large field-of-view M12 fisheye lenses (e.g., >190°)? Are there any known issues with image quality, distortion correction, or sensor limitations when using those types of lenses?

Jordyn Heil

@Alex-Kushleyev, any update on whether I can buy 4 of these RGB AR0144s?

Alex Kushleyev

@Jordyn-Heil , sorry for the delay.

We do have the color AR0144 in stock. I just need to double check there will be no conflicts in communication.

I will get back to you today or tomorrow after I test the 4 ar0144 and one imx412.

Alex

Alex Kushleyev

Regarding the lens FOV, the lens specs are provided in the camera docs (162 deg diagonal). It calibrates just fine as a fisheye lens with very low reprojection error (<0.5 pixels). The AR0144 lens does have significant lens shading, which is evident around the edges. We have computed the lens shading compensation that normalized those areas, but the result is higher noise in those areas due to quantization. The intensity drop right at the corners is around 8x compared to the center (significant!). This is because the lens is small (the imx412 lens has much better response but it is also a much larger lens).

For the color use case, the lens shading becomes even mote interesting as the different colors are distorted differently, so each color needs to have its own LSC (lens shading compensation). I have started working on RGB LSC but it is not finished yet. As a result you get not only brightness drop at the edges but also discoloration.

Will send you some image samples.

The lens has an M7 thread (i believe), so using M12 is not possible.

Let me know if you have any more questions about this or if i missed something.

Alex

Jordyn Heil

@Alex-Kushleyev, great! If there are no conflicts in communication, I'd still like to move forward with buying them as soon as possible, so just let me know what I need to do to purchase them.

If I'm trying to do full 360 panoramic stitching with four of these sensors, will the fisheye lens distortion drastically impact the efficacy of doing a seamless stitch (particularly with the discoloration from lack of an RGB LSC)?

Alex Kushleyev

@Jordyn-Heil , for ordering, since it's a custom order, please send us a message via http://modalai.com/contact and provide the following part number. We will send you an invoice.

MSU-M0166-2-01

(which includes MCCA-M0166-2 + M10000858 lens (AR0144 Camera Color Coax))

Do you want to wait until I confirm 4x concurrency?

Regarding panoramic stitching, yes obviously lens shading will be an issue, however if you have 4 cameras, you only need 90 degrees of horizontal FOV per camera and the current lens provides 139 degrees of horizontal FOV (https://docs.modalai.com/M0166/). So i think 90 degrees + small overlap won't be too bad (but LSC correction will be required to achieve seamless stitching)
. Also, the VOXL2 GPU can do the stitching, I can help you get started (should be a fun project).

Here is an old example i found of original vs LSC corrected monochrome images:

original:

LSC corrected:

You see the corners are heavily quantized because they were so dark and pixel resolution is not enough. The original RAW image is 8 bit. The AR0144 camera does support 10 and 12 bit output, and the 12 bit data would help a lot with Quantization effects after correction. the 10 and 12 bit stream is experimental and there are some issues (will not go into detail here), that is why we using 8 bit, which is stable.

Alex

Alex Kushleyev

One more thing.. Do you need global shutter cameras? or is a small size a requirement? You could use the IMX412 cameras, which are rolling shutter, but much higher resolution and color quality. Have you considered that option?

Alex

Jordyn Heil

@Alex-Kushleyev, thanks for the photos! Yes, global shutter and small size are requirements, so I think the color AR0144s are my best option, but I'll wait to order until you confirm 4x concurrency.

Also, if you can help me get started with using the VOXL2 GPU for panoramic stitching, that would be awesome! Thanks for all your help!

Alex Kushleyev

Hi @Jordyn-Heil ,

OK, so did some digging. Essentially, the 4x AR0144 + 1x IMX412 would be the same as C27 config (https://docs.modalai.com/voxl2-coax-camera-bundles/#mdk-m0173-1-02) + another AR0144 connected to J8 via M0155 (so that would be J8L, camera slot 4).

There are also some other options like moving IMX412 to J8, but in either case, unfortunately there is a change required in one of the AR0144 to switch the i2c slave address from default. This means moving a resistor on the camera from one pad to another.

I am going to double check this by actually switching the slave ID, but just wanted to give you an update..

Alex

Alex Kushleyev

OK, I was able to confirm this configuration:

VOXL2
M0173 + 3x AR0144 + 1x IMX412 (C27 config)
M0155 + 1x AR0144 plugged into J8
- AR0144 i2c slave id change via resistor
- small camera driver modification for new slave id
also verified the synchronization between the 4 AR0144

The resistor swap was relatively easy, but i used a special "tweezer" soldering iron tip. I can be difficult without the right tools.. Let me check if we can do a swap for the custom order..

See the image with marked slave id select resistor that needs to move to a pad right above the current placement. It can be a bit tricky to do it

Alex

Alex Kushleyev

Here are two image samples taking using IMX412 and AR0144. IMX412 is slightly cropped on the sides, but it's also a slightly narrower lens. AR0144 image is without LSC correction. Also images slightly out of focus because the test pattern is very close to the camera. The brightness in the IMX412 image is more evenly distributed (larger, better lens), so the center of the image is not over-exposed, unlike the AR0144. This should be corrected with the proper LSC..

IMX412

AR0144-color

Jordyn Heil

@Alex-Kushleyev, I just got the color AR0144s and was wondering what changes I need to make to the config file to get the RGB output from the AR0144s. I chose configuration 27, but when I open the stream in the VOXL web portal, I only get monochrome streams from the AR0144.

Alex Kushleyev

@Jordyn-Heil , you will need to use a recent version of voxl-camera-server (you can build it from dev or get a package from the nightly builds).

Then edit the camera server config file (/etc/modalai/voxl-camera-server.conf) and replace the camera type ar0144 with ar0144-color. If you get an error (unknown camera type), it means you are using older version of camera server.

Currently, the debayering is happening on the CPU.

Here is the place where processing is happening for the color version of AR0144. https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/hal3_camera_mgr.cpp?ref_type=heads#L1477

debayer from GRBG to RGB
run AWB (auto white balance)
run AE (auto exposure)
send out RGB, grey, and original raw bayer frames

Please try it out.

Alex

Jordyn Heil

@Alex-Kushleyev, building from the dev branch worked great, thanks!

On a separate note, for my application, I will need the coax cables connecting the cameras to be slightly longer. Do you all make longer cables, or are you aware of any other suppliers that make longer versions of these cables?
Screenshot 2025-07-08 at 10.28.36 AM-min.png

Alex Kushleyev

@Jordyn-Heil , yes we make cables from 35 to 180mm : https://www.modalai.com/products/mcbl-00084

If you are purchasing new cameras and need a non-standard cable length (I believe standard is 80mm), you may need to contact us by email for a custom order.

Alex

Jordyn Heil

@Alex-Kushleyev, I recently got the M0173, so I can finally test all cameras running concurrently. I made the resistor swap for the AR0144 plugged into J8, but could you walk me through the camera driver modification I need to make for the new slave id?

I have voxl-camera-server v2.2.4 of the dev build, but I am only able to view my streams in color (through the VOXL Web Portal) when I set misp_venc_enable to false in the configuration file. Otherwise, I get errors for each AR0144 camera that say the following:

"MISP Initializing for camera tracking_front
Got unsupported format in getUVStartFromFmt, returning nullptr"

Also, I am unable to view the IMX412 stream when it is connected to the M0173 in the C27 config. What additional modifications need to be made to solve these issues?

Alex Kushleyev

@Jordyn-Heil ,

Please use the sensormodule driver for the specific camera slot (which has been updated to use alternate slave address 0x20 instead of default 0x30 via the resistor change): https://storage.googleapis.com/modalai_public/temp/ar0144/ar0144_drivers_alt_slave_addr_0x20_20250709.zip

In my test with 4 AR0144, i connected the standard C27 configuration, and added the 4th AR0144 like so:

M0173 + 3x AR0144 + 1x IMX412 (C27 config)
M0155 + 1x AR0144 plugged into J8

So the 4th AR0144 camera will have the camera Slot 4,

If you tell me where exactly you are plugging in all the cameras, i can help with figuring out which slot IDs each camera has.

Regarding the following error, "Got unsupported format in getUVStartFromFmt, returning nullptr", i think it should be ok. Does the camera server exit in this case? it should not exit. I just tried it (to enable misp venc) and it still provides the color stream (although the h264/h265 stream is does not get published, since this specific processing pipeline is not set up for that).

For using M0173, did you set up your kernel for 1.0.1 variant? I assume so, otherwise, voxl-configure-cameras would not let you select C27? Please confirm. Do you have the cameras connected in the same way as shown in the C27 diagram? https://docs.modalai.com/voxl2-coax-camera-bundles/#mdk-m0173-1-02 . It is possible that because one of the cameras is missing (not detected) because you swapped the resistor for alternate i2c id, then the camera ID for hires is wrong in the voxl_camera_server.conf.

Here are all the sensormodules that i have for this setup in /usr/lib/camera/ (assuming the modified AR0144 is connected to J8, so slot 4.

com.qti.sensormodule.ar0144_combo_0.bin
com.qti.sensormodule.ar0144_combo_6.bin
com.qti.sensormodule.ar0144_fsin_0x20_4.bin
com.qti.sensormodule.ar0144_fsin_2.bin
com.qti.sensormodule.imx412_fpv_1.bin
com.qti.sensormodule.irs2975c_3.bin

voxl-camera-server -l shows (four AR0144 visible with sensor ID 0x0356) and the corresponding slot IDs and camera IDs:

DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 1, Cam slot: 1, Slave Address: 0x0034, Sensor Id: 0x0577
DEBUG:   Cam idx: 2, Cam slot: 2, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 3, Cam slot: 3, Slave Address: 0x007A, Sensor Id: 0x2975
DEBUG:   Cam idx: 4, Cam slot: 4, Slave Address: 0x0020, Sensor Id: 0x0356
DEBUG:   Cam idx: 5, Cam slot: 6, Slave Address: 0x0030, Sensor Id: 0x0356

The 4th AR0144 will actually be in slot 4, so it will bump the other AR0144 in slot 6 from being cam id 4 to cam id 5. Slot 6 is the AR0144 that is standard in the C27 config.

Please double check everything step by step and let me know if something is still not working.

Alex

Jordyn Heil

@Alex-Kushleyev, I am still unable to see anything from the IMX412 or the extra AR0144 that has the new slave address of 0x20.

I have the 0x30 address AR0144s and the IMX412 connected exactly as shown in this photo (minus the TOF sensor).

The additional AR0144 with the resistor change is connected to J8 via the M0155.

I downloaded the given sensormodule drivers, and this is my setup in /usr/lib/camera/:

com.qti.sensormodule.ar0144_combo_0.bin
com.qti.sensormodule.ar0144_combo_6.bin
com.qti.sensormodule.ar0144_fsin_0x20_4.bin
com.qti.sensormodule.ar0144_fsin_2.bin
com.qti.sensormodule.imx412_fpv_1.bin
com.qti.sensormodule.irs2975c_3.bin
com.qti.tuned.cmk_imx577.bin
com.qti.tuned.cmk_ov9282.bin
com.qti.tuned.default.bin
com.qti.tuned.imx412_fpv.bin
com.qti.tuned.ov9782.bin
com.qti.tuned.sony_imx335.bin

I am on kernel variant 1.01, and when I select C27, the only things I change are "type" (switch from ar0144 to ar0144-color), misp_venc_enable (set false), delete the entry for the TOF sensor, and add an identical entry for the additional ar0144-color but with "camera_id" = 5.

When I run voxl-camera-server -l, I am not able to see the DEBUG: lines. This is my output:

====================================
Number of cameras detected: 5
====================================

Jordyn Heil

@Alex-Kushleyev, I've done some more digging, and I think I found the issue.

I have confirmed that all 5 cameras are seen properly and stream to the Voxl Web Portal when I disable venc on all cameras and just enable preview.

DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 1, Cam slot: 1, Slave Address: 0x0034, Sensor Id: 0x0577
DEBUG:   Cam idx: 2, Cam slot: 2, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 3, Cam slot: 4, Slave Address: 0x0020, Sensor Id: 0x0356
DEBUG:   Cam idx: 4, Cam slot: 6, Slave Address: 0x0030, Sensor Id: 0x0356

However, when I enable venc, I get this error:

DEBUG:   Buffer Count Expected: 16
DEBUG:   Buffer Count Actual: 16
DEBUG:   Port Def 0:
	Count Min: 8
	Count Actual: 16
	Size: 0x10
	Buffers Contiguous: Yes
	Buffer Alignment: 0
DEBUG:   Buffer Count Expected: 16
DEBUG:   Buffer Count Actual: 16
DEBUG:   Port Def 1:
	Count Min: 4
	Count Actual: 16
	Size: 0x900000
	Buffers Contiguous: No
	Buffer Alignment: 0
DEBUG:   OMX_EventCmdComplete
DEBUG:   OMX_EventCmdComplete
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize encoder for camera: hires
ERROR:   Failed to start camera: hires

My Voxl is running the most recent voxl-camera-server dev branch, and in hal3_camera_mgr.cpp, the following are all defined:

#define NUM_PREVIEW_BUFFERS  16 // used to be 32, really shouldnt need to be more than 7
#define NUM_SNAPSHOT_BUFFERS 16 // used to be 8, just making it consistent with the rest that are now 16
#define NUM_MISP_BUFFERS     16
#define NUM_STREAM_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames
#define NUM_RECORD_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames

All of these are set to 16, so why is my Voxl saying that it doesn't have 16 buffers available? It's also not competing for resources since I tested this out with just one camera having venc enabled.

Jordyn Heil

After adding in some extra debug statements in the voxl-camera-server codebase, these are my findings:

The mpa_ion_buf_pool_alloc_bufs function is allocating 16 buffers but only properly initializing 1 buffer out of 16. The remaining 15 buffers contain corrupted or invalid data, as evidenced by the debug output showing mostly NULL pointers, invalid file descriptors, and bad memory addresses.

[DEBUG] Buffer[0]: vaddress=0x5588128d50, fd=-2010712080, size=85, width=85, height=-2013126160, stride=85, slice=-2011714000
[DEBUG] Buffer[1]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[2]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[3]: vaddress=(nil), fd=-1, size=0, width=127, height=-2044747776, stride=127, slice=0
[DEBUG] Buffer[4]: vaddress=(nil), fd=800, size=127, width=127, height=0, stride=0, slice=102
[DEBUG] Buffer[5]: vaddress=0x24000000000000, fd=1024, size=127, width=0, height=104, stride=0, slice=0
[DEBUG] Buffer[6]: vaddress=0x50000000000, fd=-2014735536, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[7]: vaddress=0x60000000320, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[8]: vaddress=0x400, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[9]: vaddress=0x5587d77ba0, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[10]: vaddress=(nil), fd=-2079350784, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[11]: vaddress=(nil), fd=-2082496512, size=0, width=0, height=0, stride=2359296, slice=0
[DEBUG] Buffer[12]: vaddress=0x900000000, fd=-1, size=0, width=2359296, height=0, stride=1280, slice=800
[DEBUG] Buffer[13]: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800, stride=1536, slice=1024
[DEBUG] Buffer[14]: vaddress=0x7f8307a000, fd=0, size=1280, width=1536, height=1024, stride=0, slice=-2012890624
[DEBUG] Buffer[15]: vaddress=(nil), fd=-1, size=1536, width=0, height=-2014372240, stride=85, slice=0
[DEBUG] Found 1 valid buffers out of 16 allocated

The debug output shows that only Buffer[13] has valid data with proper values: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800. The library reports success but returns mostly unusable buffers, which causes the OMX encoder to fail during initialization since it expects a minimum number of valid buffers, of which, it only found 1 before (hence, the message, WARNING: Encoder expecting(16) more buffers than module allocated(1)).

I implemented a temporary workaround that identifies the single valid buffer from the corrupted pool and duplicates its metadata to create 8 usable buffers (I was able to decrease the expected to 8 from 16, but I couldn't get any lower). This involves moving valid buffers to the front of the pool and zeroing out remaining buffers to prevent crashes. The workaround allows the OMX encoder to initialize with the minimum required buffers instead of failing with only 1 valid buffer.

This is clearly a temporary solution and not a permanent fix. The root cause lies in the mpa_ion_buf_pool_alloc_bufs function, which appears to have a memory management issue. I was unable to find the implementation of that function, so I was unable to debug any further.

I'm not sure if something else regarding the configuration on my Voxl was causing that issue. I was unable to find anything, but let me know if I'm missing something very obvious.