ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    AR0144 RGB output on VOXL2

    Image Sensors
    3
    38
    1022
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      Jordyn Heil @Alex Kushleyev
      last edited by

      @Alex-Kushleyev, I've done some more digging, and I think I found the issue.

      I have confirmed that all 5 cameras are seen properly and stream to the Voxl Web Portal when I disable venc on all cameras and just enable preview.

      DEBUG:   Cam idx: 0, Cam slot: 0, Slave Address: 0x0030, Sensor Id: 0x0356
      DEBUG:   Cam idx: 1, Cam slot: 1, Slave Address: 0x0034, Sensor Id: 0x0577
      DEBUG:   Cam idx: 2, Cam slot: 2, Slave Address: 0x0030, Sensor Id: 0x0356
      DEBUG:   Cam idx: 3, Cam slot: 4, Slave Address: 0x0020, Sensor Id: 0x0356
      DEBUG:   Cam idx: 4, Cam slot: 6, Slave Address: 0x0030, Sensor Id: 0x0356
      

      However, when I enable venc, I get this error:

      DEBUG:   Buffer Count Expected: 16
      DEBUG:   Buffer Count Actual: 16
      DEBUG:   Port Def 0:
      	Count Min: 8
      	Count Actual: 16
      	Size: 0x10
      	Buffers Contiguous: Yes
      	Buffer Alignment: 0
      DEBUG:   Buffer Count Expected: 16
      DEBUG:   Buffer Count Actual: 16
      DEBUG:   Port Def 1:
      	Count Min: 4
      	Count Actual: 16
      	Size: 0x900000
      	Buffers Contiguous: No
      	Buffer Alignment: 0
      DEBUG:   OMX_EventCmdComplete
      DEBUG:   OMX_EventCmdComplete
      WARNING: Encoder expecting(16) more buffers than module allocated(1)
      ERROR:   OMX Set config failed!
      ERROR:   Failed to initialize encoder for camera: hires
      ERROR:   Failed to start camera: hires
      

      My Voxl is running the most recent voxl-camera-server dev branch, and in hal3_camera_mgr.cpp, the following are all defined:

      #define NUM_PREVIEW_BUFFERS  16 // used to be 32, really shouldnt need to be more than 7
      #define NUM_SNAPSHOT_BUFFERS 16 // used to be 8, just making it consistent with the rest that are now 16
      #define NUM_MISP_BUFFERS     16
      #define NUM_STREAM_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames
      #define NUM_RECORD_BUFFERS   16 // shouldn't need more than 10, if the buffer pool is empty then OMX should be dropping more frames
      

      All of these are set to 16, so why is my Voxl saying that it doesn't have 16 buffers available? It's also not competing for resources since I tested this out with just one camera having venc enabled.

      J 1 Reply Last reply Reply Quote 0
      • J
        Jordyn Heil @Jordyn Heil
        last edited by Jordyn Heil

        After adding in some extra debug statements in the voxl-camera-server codebase, these are my findings:

        The mpa_ion_buf_pool_alloc_bufs function is allocating 16 buffers but only properly initializing 1 buffer out of 16. The remaining 15 buffers contain corrupted or invalid data, as evidenced by the debug output showing mostly NULL pointers, invalid file descriptors, and bad memory addresses.

        [DEBUG] Buffer[0]: vaddress=0x5588128d50, fd=-2010712080, size=85, width=85, height=-2013126160, stride=85, slice=-2011714000
        [DEBUG] Buffer[1]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[2]: vaddress=(nil), fd=-1, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[3]: vaddress=(nil), fd=-1, size=0, width=127, height=-2044747776, stride=127, slice=0
        [DEBUG] Buffer[4]: vaddress=(nil), fd=800, size=127, width=127, height=0, stride=0, slice=102
        [DEBUG] Buffer[5]: vaddress=0x24000000000000, fd=1024, size=127, width=0, height=104, stride=0, slice=0
        [DEBUG] Buffer[6]: vaddress=0x50000000000, fd=-2014735536, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[7]: vaddress=0x60000000320, fd=0, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[8]: vaddress=0x400, fd=0, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[9]: vaddress=0x5587d77ba0, fd=0, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[10]: vaddress=(nil), fd=-2079350784, size=0, width=0, height=0, stride=0, slice=0
        [DEBUG] Buffer[11]: vaddress=(nil), fd=-2082496512, size=0, width=0, height=0, stride=2359296, slice=0
        [DEBUG] Buffer[12]: vaddress=0x900000000, fd=-1, size=0, width=2359296, height=0, stride=1280, slice=800
        [DEBUG] Buffer[13]: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800, stride=1536, slice=1024
        [DEBUG] Buffer[14]: vaddress=0x7f8307a000, fd=0, size=1280, width=1536, height=1024, stride=0, slice=-2012890624
        [DEBUG] Buffer[15]: vaddress=(nil), fd=-1, size=1536, width=0, height=-2014372240, stride=85, slice=0
        [DEBUG] Found 1 valid buffers out of 16 allocated
        

        The debug output shows that only Buffer[13] has valid data with proper values: vaddress=0x7f8337a000, fd=120, size=2359296, width=1280, height=800. The library reports success but returns mostly unusable buffers, which causes the OMX encoder to fail during initialization since it expects a minimum number of valid buffers, of which, it only found 1 before (hence, the message, WARNING: Encoder expecting(16) more buffers than module allocated(1)).

        I implemented a temporary workaround that identifies the single valid buffer from the corrupted pool and duplicates its metadata to create 8 usable buffers (I was able to decrease the expected to 8 from 16, but I couldn't get any lower). This involves moving valid buffers to the front of the pool and zeroing out remaining buffers to prevent crashes. The workaround allows the OMX encoder to initialize with the minimum required buffers instead of failing with only 1 valid buffer.

        This is clearly a temporary solution and not a permanent fix. The root cause lies in the mpa_ion_buf_pool_alloc_bufs function, which appears to have a memory management issue. I was unable to find the implementation of that function, so I was unable to debug any further.

        I'm not sure if something else regarding the configuration on my Voxl was causing that issue. I was unable to find anything, but let me know if I'm missing something very obvious.

        Alex KushleyevA 1 Reply Last reply Reply Quote 0
        • Alex KushleyevA
          Alex Kushleyev ModalAI Team @Jordyn Heil
          last edited by

          @Jordyn-Heil , i have seen this issue before. This is because you are compiling the camera server app against one version of libmodal-pipe library, but a different version is running on voxl2. There was an api change in the header file causing this issue. The solution is to use latest libmodal pipe (from dev) while cross compiling and running on voxl. You dont need to update the whole sdk, just update that lib.

          Alex

          J 1 Reply Last reply Reply Quote 0
          • J
            Jordyn Heil @Alex Kushleyev
            last edited by

            @Alex-Kushleyev, thank you! That solved my issue!

            I can successfully stream the grey MISP streams over RTSP from my color AR0144s, but I am unable to stream colored frames over RTSP. Is the processing pipeline to do this implemented, and if so, what modifications need to be made to send and access these streams over RTSP?

            Also, I believe you mentioned earlier that the VOXL GPU can handle panoramic stitching between camera streams? I am currently using the CPU to do panoramic stitching, but there is too much lag for my application, so I would like to try and use a compute shader to offload the work to the GPU. What is the best way to do this on the VOXL2?

            Alex KushleyevA 1 Reply Last reply Reply Quote 0
            • Alex KushleyevA
              Alex Kushleyev ModalAI Team @Jordyn Heil
              last edited by

              @Jordyn-Heil , yes i need to add the part that feeds the frames into the video encoder for the ar0144 color use case. it's not difficult, i will do it this week.

              Regarding GPU usage, I have only done OpenCL programming on VOXL2, but OpenGL is also supported, I can probably find some resources to help with that. Which one of the two do you prefer? OpenCL would probably be easier (for me to help you) since we have done much more with that already.

              Alex

              J 2 Replies Last reply Reply Quote 0
              • J
                Jordyn Heil @Alex Kushleyev
                last edited by

                @Alex-Kushleyev, great, thanks for doing that!

                OpenCL should work just fine!

                Alex KushleyevA 1 Reply Last reply Reply Quote 0
                • Alex KushleyevA
                  Alex Kushleyev ModalAI Team @Jordyn Heil
                  last edited by Alex Kushleyev

                  @Jordyn-Heil , sorry for the delay.

                  I will provide an initial example of getting 4 AR0144 streams into a standalone app and doing something with all of them on the GPU using OpenCL. This should be a nice example / starting point for multi-camera image fusion.

                  Initially I will just use our standard mpa approach to get the image buffers from camera server to the standalone app. After that i will also try to do the same using the ION buffer support (which is already in camera server) which will allow us to map the buffers to GPU without doing any copy operations or sending image data over pipes. using ION buffer sharing will allow to further reduce the cpu and memory bandwidth usage, but it's still a bit of experimental feature and will need some more testing.

                  Please give me a few more days.

                  Alex

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    Jordyn Heil @Alex Kushleyev
                    last edited by

                    @Alex-Kushleyev, no worries! I appreciate your help on this!

                    In the meantime, would you be able to share the CAD file for the M0173 breakout board? I was not able to find it anywhere.

                    VinnyV 1 Reply Last reply Reply Quote 0
                    • VinnyV
                      Vinny ModalAI Team @Jordyn Heil
                      last edited by

                      Hi @Jordyn-Heil
                      We try to keep this regularly updated, but sometimes miss some... in this case, M0173 is there 🙂 https://docs.modalai.com/pcb-catalog/

                      1 Reply Last reply Reply Quote 1
                      • J
                        Jordyn Heil @Alex Kushleyev
                        last edited by

                        @Alex-Kushleyev, were you able to update the dev build of voxl-camera-server to encode the ar0144 color streams?

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Powered by NodeBB | Contributors