ModalAI Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    EIS merge

    VOXL 2
    6
    30
    1821
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JP DroneJ
      JP Drone @Alex Kushleyev
      last edited by

      Hi @Alex-Kushleyev
      Some of the videos and photos taken with the IMX412 I have on hand were blurry due to vibration.
      I tried various things but couldn’t resolve the issue—however, it was resolved after I replaced the lens with a different one.
      Would it be possible to include a replacement lens with my current order?
      Also, I have a few units in stock, and I’m concerned there might be similar issues with them.

      Alex KushleyevA 1 Reply Last reply Reply Quote 0
      • Alex KushleyevA
        Alex Kushleyev ModalAI Team @JP Drone
        last edited by

        @JP-Drone , I will double check, but we should be able to send replacements. However, we first need to figure out what's going on with these lenses and make sure the replacements are good.

        Here is an example of EIS running with bad and good lenses (just a short test):

        bad : https://drive.google.com/file/d/1g3dCVs64bBuDhSWJ-ZiKMbyHUBmR7AgB/view?usp=drive_link
        good: https://drive.google.com/file/d/1R6GCH3l2d6CbwRPCrMRbvD21e2yg3vfq/view?usp=drive_link

        Please note that when tested indoor (low light conditions), the rolling shutter "jello" may not be as obvious, since it will get smoothed out by the long exposure times.

        Alex

        Alex KushleyevA 1 Reply Last reply Reply Quote 0
        • Alex KushleyevA
          Alex Kushleyev ModalAI Team @Alex Kushleyev
          last edited by Alex Kushleyev

          The EIS branch is ready for initial tested. Please see the doc file:

          https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/eis/voxl2-eis.md

          You will need to build the camera server from this branch (and it's best to be able to do that anyway, if experimenting with eis..). Please note that you will need the latest voxl-cross:V4.0 docker image to build the camera server.

          The eis branch is sync'ed to latest dev as of today. Added basic params for specifying EIS behavior, as documented (eis_mode, eis_view, eis_follow_rate). Parameter usage is documented as well as basic voxl-camera-server.conf example is provided. Parameter names may change in the future, but the changes will be documented.

          Please use the latest IMX412 driver as instructed. For best results, use input resolution of 4040x3040 and output resolution can be anything. including 3840x2160. 4K60 output is supported (h265 strongly recommended), but you will not be able to view that via the voxl-portal.

          This release also supports eis on 3 additional MISP channels (can be different resolutions).

          We will be focusing next on improving stabilized ROI behavior for smoother output and experimenting more with rolling shutter compensation.

          Please try it out, i would be excited to get initial feedback.

          Alex

          JP DroneJ 1 Reply Last reply Reply Quote 1
          • JP DroneJ
            JP Drone @Alex Kushleyev
            last edited by

            Hi @Alex-Kushleyev
            Happy to test it.
            My IMX412 is mounted upside-down on J7-Lower (cam ID 2).
            Could you provide a flip version of the EIS driver, e.g. com.qti.sensormodule.imx412_fpv_flip_eis_2.bin?

            Thanks a lot for all the support!

            Alex KushleyevA 1 Reply Last reply Reply Quote 0
            • Alex KushleyevA
              Alex Kushleyev ModalAI Team @JP Drone
              last edited by Alex Kushleyev

              @JP-Drone , If your camera is flipped (rotated 180), you just need to update your extrinsics to tell EIS the orientation of camera w.r.t IMU:

              https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/voxl2-eis.md?ref_type=heads#extrinsic-calibration

              Instead of right side up

              "RPY_parent_to_child":    [0, 90, 90]
              

              your extrinsic file should be

              "RPY_parent_to_child":    [0, 90, -90]
              

              In other words, your /etc/modalai/extrinsics.conf should contain the following entry:

              {
                  "parent": "imu_apps",
                  "child":  "<camera_name>",
                  "T_child_wrt_parent": [0.0, 0.0, 0.0],
                  "RPY_parent_to_child":    [0, 90, -90]
              }
              

              (I am assuming VOXL2 is mounted in normal orientation, right side up)

              Please try.

              Also, we recently merged the EIS feature to dev branch of camera server, still working on better documentation.

              Alex

              Alex KushleyevA 1 Reply Last reply Reply Quote 0
              • Alex KushleyevA
                Alex Kushleyev ModalAI Team @Alex Kushleyev
                last edited by

                Initial docs are up : https://docs.modalai.com/camera-video/electronic-image-stabilization/

                S 1 Reply Last reply Reply Quote 0
                • S
                  SKA @Alex Kushleyev
                  last edited by

                  Hi @Alex-Kushleyev! Where can I find the IMX214 drivers for EIS? Thanks!

                  Alex KushleyevA 1 Reply Last reply Reply Quote 0
                  • Alex KushleyevA
                    Alex Kushleyev ModalAI Team @SKA
                    last edited by

                    Hello @SKA,

                    I just posted the IMX214 sensormodules here : https://storage.googleapis.com/modalai_public/temp/imx214_test_bins/imx214_eis_drivers_20250815.zip

                    These support the new resolutions 4196x3120 and 2096x1560 needed for MISP / EIS.

                    IMX214 (M0025-2)
                    
                    4196x3120 - calibration file resolution 4196x3120 or 2098x1560
                    2096x1560 - calibration file resolution 2096x1560 or 2098x1560
                    

                    (calibration file is optional, EIS will run without one)

                    I also just updated the voxl camera server (dev branch) to enable MISP support for these resolutions and also a new feature to allow specifying the bayer pattern (in case you use the flipped sensormodule).

                    Here is an example snipped of voxl-camera-server.conf. The parameter bayer_type is optional and hidden by default. you can use rggb or bggr for IMX214. You can change this parameter if you see that your colors are flipped (red and blue).

                                            "type": "imx214",
                                            "name": "hires0",
                                            "bayer_type": "rggb"
                    

                    The performance of EIS with IMX214 has not been tuned - specifically the behavior of the ROI controller. We can come back to that if there is interest.. Mostly focusing on IMX412 with the fisheye lens for now.

                    Please try and make sure you can at least make it work and we can discuss performance tuning.

                    Alex

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      SKA @Alex Kushleyev
                      last edited by

                      @Alex-Kushleyev The drivers worked, and the EIS as expected isn't tuned well. The MISP brightness looks alright outdoors, but indoors its very low. Increasing exposure_max_us helps but adds blur. Not sure if that's to be expected with this sensor, since it isn't as good in low-light conditions as the IMX412. Let me know if there's anything I can tune in the conf file to help with the brightness, and EIS output. Thanks!

                      Alex KushleyevA 1 Reply Last reply Reply Quote 0
                      • Alex KushleyevA
                        Alex Kushleyev ModalAI Team @SKA
                        last edited by

                        Hi @SKA,

                        The down side of using IMX214 vs IMX412 is that IMX214 is a much older sensor and it has several attributes that make it less desirable for using with EIS:

                        • lower pixel sensitivity and lower maximum gain range, requiring increasing exposure to achieve desired brightness, but this creates motion blur. The image brightness can be increased via digital gain, but will add some pixel noise
                        • smaller package / lens -- smaller lens does not allow as much light to enter the sensor
                        • smaller FOV (larger FOV with fisheye lens allows for more compensation margin)
                        • lower readout speed (about 2x slower). This increases the readout time, which is related to rolling shutter effect.

                        If the image brightness is the most concern to you right now, I can add a digital gain feature to boost the image brightness, but we are going to see some more pixel noise. I wanted to do that anyway, so may be a good chance to try it. After boosting the digital gain, ideally you would want to run some de-noising algorithm, but we don't have that yet in the MISP pipeline.

                        Are you working with an existing system that has IMX214 already integrated and want to see if you can enable EIS on that set up?

                        Alex

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Powered by NodeBB | Contributors