@Alex-Kushleyev,

Yes, what you have described for the client application is what I am trying to design. Ideally, I would have time synchronization so that frames always line up, but if latency isn't too big of an issue, this should work fine. I was planning on working with the RGB image, but I'm open to using YUV if it offers a noticeable reduction in latency. Also, to clarify my current setup, I have 4 AR0144 cameras placed at 90-degree angles to one another, so there is plenty of overlap between adjacent cameras for stitching.

A simple example would be extremely useful! Thanks for your help!

@Alex-Kushleyev, were you able to update the dev build of voxl-camera-server to encode the ar0144 color streams?

@Alex-Kushleyev, no worries! I appreciate your help on this!

In the meantime, would you be able to share the CAD file for the M0173 breakout board? I was not able to find it anywhere.

@Alex-Kushleyev, great, thanks for doing that!

OpenCL should work just fine!

@Alex-Kushleyev, thank you! That solved my issue!

I can successfully stream the grey MISP streams over RTSP from my color AR0144s, but I am unable to stream colored frames over RTSP. Is the processing pipeline to do this implemented, and if so, what modifications need to be made to send and access these streams over RTSP?

Also, I believe you mentioned earlier that the VOXL GPU can handle panoramic stitching between camera streams? I am currently using the CPU to do panoramic stitching, but there is too much lag for my application, so I would like to try and use a compute shader to offload the work to the GPU. What is the best way to do this on the VOXL2?

After adding in some extra debug statements in the voxl-camera-server codebase, these are my findings:

The mpa_ion_buf_pool_alloc_bufs function is allocating 16 buffers but only properly initializing 1 buffer out of 16. The remaining 15 buffers contain corrupted or invalid data, as evidenced by the debug output showing mostly NULL pointers, invalid file descriptors, and bad memory addresses.

[DEBUG] Buffer[0]: vaddress=0x5588128d50, fd=-2010712080, size=85, width=85, height=-2013126160, stride=85, slice=-2011714000
[DEBUG] Buffer[1]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[2]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[3]: vaddress=(nil), fd=-1, size=0, width=127, height=-2044747776, stride=127, slice=0
[DEBUG] Buffer[4]: vaddress=(nil), fd=800, size=127, width=127, height=0, stride=0, slice=102
[DEBUG] Buffer[5]: vaddress=0x24000000000000, fd=1024, size=127, width=0, height=104, stride=0, slice=0
[DEBUG] Buffer[6]: vaddress=0x50000000000, fd=-2014735536, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[7]: vaddress=0x60000000320, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[8]: vaddress=0x400, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[9]: vaddress=0x5587d77ba0, fd=0, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[10]: vaddress=(nil), fd=-2079350784, size=0, width=0, height=0, stride=0, slice=0
[DEBUG] Buffer[11]: vaddress=(nil), fd=-2082496512, size=0, width=0, height=0, stride=2359296, slice=0
[DEBUG] Buffer[12]: vaddress=0x900000000, fd=-1, size=0, width=2359296, height=0, stride=1280, slice=800
[DEBUG] Buffer[13]: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800, stride=1536, slice=1024
[DEBUG] Buffer[14]: vaddress=0x7f8307a000, fd=0, size=1280, width=1536, height=1024, stride=0, slice=-2012890624
[DEBUG] Buffer[15]: vaddress=(nil), fd=-1, size=1536, width=0, height=-2014372240, stride=85, slice=0
[DEBUG] Found 1 valid buffers out of 16 allocated

The debug output shows that only Buffer[13] has valid data with proper values: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800. The library reports success but returns mostly unusable buffers, which causes the OMX encoder to fail during initialization since it expects a minimum number of valid buffers, of which, it only found 1 before (hence, the message, WARNING: Encoder expecting(16) more buffers than module allocated(1)).

I implemented a temporary workaround that identifies the single valid buffer from the corrupted pool and duplicates its metadata to create 8 usable buffers (I was able to decrease the expected to 8 from 16, but I couldn't get any lower). This involves moving valid buffers to the front of the pool and zeroing out remaining buffers to prevent crashes. The workaround allows the OMX encoder to initialize with the minimum required buffers instead of failing with only 1 valid buffer.

This is clearly a temporary solution and not a permanent fix. The root cause lies in the mpa_ion_buf_pool_alloc_bufs function, which appears to have a memory management issue. I was unable to find the implementation of that function, so I was unable to debug any further.

I'm not sure if something else regarding the configuration on my Voxl was causing that issue. I was unable to find anything, but let me know if I'm missing something very obvious.

@Alex-Kushleyev, I've done some more digging, and I think I found the issue.

I have confirmed that all 5 cameras are seen properly and stream to the Voxl Web Portal when I disable venc on all cameras and just enable preview.

DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 1, Cam slot: 1, Slave Address: 0x0034, Sensor Id: 0x0577
DEBUG:   Cam idx: 2, Cam slot: 2, Slave Address: 0x0030, Sensor Id: 0x0356
DEBUG:   Cam idx: 3, Cam slot: 4, Slave Address: 0x0020, Sensor Id: 0x0356
DEBUG:   Cam idx: 4, Cam slot: 6, Slave Address: 0x0030, Sensor Id: 0x0356

However, when I enable venc, I get this error:

DEBUG:   Buffer Count Expected: 16
DEBUG:   Buffer Count Actual: 16
DEBUG:   Port Def 0:
	Count Min: 8
	Count Actual: 16
	Size: 0x10
	Buffers Contiguous: Yes
	Buffer Alignment: 0
DEBUG:   Buffer Count Expected: 16
DEBUG:   Buffer Count Actual: 16
DEBUG:   Port Def 1:
	Count Min: 4
	Count Actual: 16
	Size: 0x900000
	Buffers Contiguous: No
	Buffer Alignment: 0
DEBUG:   OMX_EventCmdComplete
DEBUG:   OMX_EventCmdComplete
WARNING: Encoder expecting(16) more buffers than module allocated(1)
ERROR:   OMX Set config failed!
ERROR:   Failed to initialize encoder for camera: hires
ERROR:   Failed to start camera: hires

My Voxl is running the most recent voxl-camera-server dev branch, and in hal3_camera_mgr.cpp, the following are all defined:

#define NUM_PREVIEW_BUFFERS  16 // used to be 32, really shouldnt need to be more than 7
#define NUM_SNAPSHOT_BUFFERS 16 // used to be 8, just making it consistent with the rest that are now 16
#define NUM_MISP_BUFFERS     16
#define NUM_STREAM_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames
#define NUM_RECORD_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames

All of these are set to 16, so why is my Voxl saying that it doesn't have 16 buffers available? It's also not competing for resources since I tested this out with just one camera having venc enabled.

@Alex-Kushleyev, I am still unable to see anything from the IMX412 or the extra AR0144 that has the new slave address of 0x20.

I have the 0x30 address AR0144s and the IMX412 connected exactly as shown in this photo (minus the TOF sensor).

The additional AR0144 with the resistor change is connected to J8 via the M0155.

I downloaded the given sensormodule drivers, and this is my setup in /usr/lib/camera/:

com.qti.sensormodule.ar0144_combo_0.bin
com.qti.sensormodule.ar0144_combo_6.bin
com.qti.sensormodule.ar0144_fsin_0x20_4.bin
com.qti.sensormodule.ar0144_fsin_2.bin
com.qti.sensormodule.imx412_fpv_1.bin
com.qti.sensormodule.irs2975c_3.bin
com.qti.tuned.cmk_imx577.bin
com.qti.tuned.cmk_ov9282.bin
com.qti.tuned.default.bin
com.qti.tuned.imx412_fpv.bin
com.qti.tuned.ov9782.bin
com.qti.tuned.sony_imx335.bin

I am on kernel variant 1.01, and when I select C27, the only things I change are "type" (switch from ar0144 to ar0144-color), misp_venc_enable (set false), delete the entry for the TOF sensor, and add an identical entry for the additional ar0144-color but with "camera_id" = 5.

When I run voxl-camera-server -l, I am not able to see the DEBUG: lines. This is my output:

====================================
Number of cameras detected: 5
====================================

@Alex-Kushleyev, I recently got the M0173, so I can finally test all cameras running concurrently. I made the resistor swap for the AR0144 plugged into J8, but could you walk me through the camera driver modification I need to make for the new slave id?

I have voxl-camera-server v2.2.4 of the dev build, but I am only able to view my streams in color (through the VOXL Web Portal) when I set misp_venc_enable to false in the configuration file. Otherwise, I get errors for each AR0144 camera that say the following:

"MISP Initializing for camera tracking_front
Got unsupported format in getUVStartFromFmt, returning nullptr"

Also, I am unable to view the IMX412 stream when it is connected to the M0173 in the C27 config. What additional modifications need to be made to solve these issues?

@Alex-Kushleyev, building from the dev branch worked great, thanks!

On a separate note, for my application, I will need the coax cables connecting the cameras to be slightly longer. Do you all make longer cables, or are you aware of any other suppliers that make longer versions of these cables?