• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Register
  • Login
ModalAI Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
    • Register
    • Login

    EIS merge

    VOXL 2
    6
    28
    1.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Alex Kushleyev ModalAI Team @h3robotics
      last edited by 2 May 2025, 13:41

      @h3robotics ,

      I am working on merging the current EIS features to our main branch, this should be done sometime next week.

      Meanwhile, it would help if you briefly describe your use case for EIS, so I can make sure it is supported in the release. Are you using IMX412 camera?

      The Rolling Shutter correction won't yet be supported, but I wanted to release what we have so far, which works pretty under low vibration conditions.

      If you don't have the build environment set up for building voxl-camera-server, i can provide the deb package for you to test what is on EIS branch right now.

      Alex

      H 1 Reply Last reply 2 May 2025, 14:27 Reply Quote 1
      • H
        h3robotics @Alex Kushleyev
        last edited by 2 May 2025, 14:27

        @Alex-Kushleyev thank you so much. I can wait till the changes are merged and available to update directly on the voxl2.

        Yes, I am using the IMX412 camera.

        I can start off with whatever you have got for now and hope for a good video feed on my drone without any gimbal. As new changes come along, I will update.

        I have never setup a build environment for voxl repos. I am used to building PX4 etc. with some of my custom changes. I would love to learn how to setup the build environment for the voxl repos. So if you can point me in the right direction, that would be great! I see in the docs, under "voxl SDK > custom voxl applications", there are some instructions.

        Thank you!

        H 1 Reply Last reply 30 May 2025, 17:46 Reply Quote 0
        • H
          h3robotics @h3robotics
          last edited by 30 May 2025, 17:46

          Hi @Alex-Kushleyev ,

          Any update on the EIS feature merge?

          Thanks.

          A 1 Reply Last reply 2 Jun 2025, 17:44 Reply Quote 0
          • A
            Alex Kushleyev ModalAI Team @h3robotics
            last edited by 2 Jun 2025, 17:44

            Hi @h3robotics, Thanks for your patience.

            Actually as of right now, the EIS branch has been updated to have all the latest features from dev.

            One big improvement from prior EIS version is that now for each camera we support up to 4 MISP outputs of different resolution, fps, encoder params and each stream can have EIS on or off.

            Please hang tight, i just need to tie up some loose ends and add documentation. I will post an update most likely tomorrow. I will also share a voxl-camera-server.conf with the usage examples.

            Meanwhile please make sure you can build the eis branch from here, https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/eis , using voxl-cross:4.0 docker image, which is required.

            Alex

            A 1 Reply Last reply 10 Jun 2025, 18:11 Reply Quote 0
            • A
              Alex Kushleyev ModalAI Team @Alex Kushleyev
              last edited by 10 Jun 2025, 18:11

              Hi All,

              Sorry for the delay again. We were chasing down a vibration issue which was apparently causing significant rolling shutter artifacts on the front camera in my tests. Apparently some of the lenses for IMX412 cameras have loose internal components, causing the vibration which is impossible to correct. On my test platform (Starling 2 Max), I had this issue and no frame modifications improved anything. The last thing to try was to switch out the lens and the issue was completely gone.

              We will investigate the reason for the loose lenses and the solution. In my case, shaking the lens would result in audible rattle inside the lens. If anyone has "shaky" lens, we will send a replacement.

              I will post some sample videos of what a bad lens might look like.

              The good news is that there is actually no major vibration issue on the Starling 2 Max and I can move forward with the EIS sw release.. stay tuned for another day or so! 🙂

              Alex

              J 1 Reply Last reply 11 Jun 2025, 06:16 Reply Quote 0
              • J
                JP Drone @Alex Kushleyev
                last edited by 11 Jun 2025, 06:16

                Hi @Alex-Kushleyev
                Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
                I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
                Would it be possible to include a replacement lens with my current order?
                Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

                A 1 Reply Last reply 12 Jun 2025, 20:27 Reply Quote 0
                • A
                  Alex Kushleyev ModalAI Team @JP Drone
                  last edited by 12 Jun 2025, 20:27

                  @JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

                  Here is an example of EIS running with bad and good lenses (just a short test):

                  bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
                  good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

                  Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

                  Alex

                  A 1 Reply Last reply 12 Jun 2025, 22:15 Reply Quote 0
                  • A
                    Alex Kushleyev ModalAI Team @Alex Kushleyev
                    last edited by Alex Kushleyev 12 Jun 2025, 22:16 12 Jun 2025, 22:15

                    The EIS branch is ready for initial tested. Please see the doc file:

                    https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

                    You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

                    The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

                    Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

                    This release also supports eis on 3 additional MISP channels (can be different resolutions).

                    We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

                    Please try it out, i would be excited to get initial feedback.

                    Alex

                    J 1 Reply Last reply 20 Jun 2025, 01:57 Reply Quote 1
                    • J
                      JP Drone @Alex Kushleyev
                      last edited by 20 Jun 2025, 01:57

                      Hi @Alex-Kushleyev
                      Happy to test it.
                      My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
                      Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

                      Thanks a lot for all the support!

                      A 1 Reply Last reply 20 Jun 2025, 02:28 Reply Quote 0
                      • A
                        Alex Kushleyev ModalAI Team @JP Drone
                        last edited by Alex Kushleyev 20 Jun 2025, 02:29 20 Jun 2025, 02:28

                        @JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

                        https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/voxl2-eis.md?ref_type=heads#extrinsic-calibration

                        Instead of right side up

                        "RPY_parent_to_child":    [0, 90, 90]
                        

                        your extrinsic file should be

                        "RPY_parent_to_child":    [0, 90, -90]
                        

                        In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

                        {
                            "parent": "imu_apps",
                            "child":  "<camera_name>",
                            "T_child_wrt_parent": [0.0, 0.0, 0.0],
                            "RPY_parent_to_child":    [0, 90, -90]
                        }
                        

                        (I am assuming VOXL2 is mounted in normal orientation, right side up)

                        Please try.

                        Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

                        Alex

                        A 1 Reply Last reply 20 Jun 2025, 04:08 Reply Quote 0
                        • A
                          Alex Kushleyev ModalAI Team @Alex Kushleyev
                          last edited by 20 Jun 2025, 04:08

                          Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/

                          S 1 Reply Last reply 6 days ago Reply Quote 0
                          • S
                            SKA @Alex Kushleyev
                            last edited by 6 days ago

                            Hi @Alex-Kushleyev! Where can I find the IMX214 drivers for EIS? Thanks!

                            A 1 Reply Last reply 5 days ago Reply Quote 0
                            • A
                              Alex Kushleyev ModalAI Team @SKA
                              last edited by 5 days ago

                              Hello @SKA,

                              I just posted the IMX214 sensormodules here : https://storage.googleapis.com/modalai_public/temp/imx214_test_bins/imx214_eis_drivers_20250815.zip

                              These support the new resolutions 4196x3120 and 2096x1560 needed for MISP / EIS.

                              IMX214 (M0025-2)
                              
                              4196x3120 - calibration file resolution 4196x3120 or 2098x1560
                              2096x1560 - calibration file resolution 2096x1560 or 2098x1560
                              

                              (calibration file is optional, EIS will run without one)

                              I also just updated the voxl camera server (dev branch) to enable MISP support for these resolutions and also a new feature to allow specifying the bayer pattern (in case you use the flipped sensormodule).

                              Here is an example snipped of voxl-camera-server.conf. The parameter bayer_type is optional and hidden by default. you can use rggb or bggr for IMX214. You can change this parameter if you see that your colors are flipped (red and blue).

                                                      "type": "imx214",
                                                      "name": "hires0",
                                                      "bayer_type": "rggb"
                              

                              The performance of EIS with IMX214 has not been tuned - specifically the behavior of the ROI controller. We can come back to that if there is interest.. Mostly focusing on IMX412 with the fisheye lens for now.

                              Please try and make sure you can at least make it work and we can discuss performance tuning.

                              Alex

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Powered by NodeBB | Contributors