Posts made by Alex Kushleyev

Here is an outline of the data flow that you may want to start with:

Choose the camera resolution, since it cannot be changed after logging

• IMX412: 4040x3040 or 1996x1520 (2x2 binned), use full frame to maximize FOV (4:3 aspect ratio, not 16:9, which will crop the image)

voxl-logger + copy intrinsics / extrinsics

• log raw10 bayer and grey or normalized (or all 3)
• log imu data
• save camera intrinsics
• save camera and imu extrinsics (extrinsics.conf)

voxl-playback

• option 1: simple:

  • playback grey / normalized, feed directly into voxl-qvio server

• option2 : more complex

  • playback raw10 bayer
  • run offline misp implementation to debayer + AWB + normalize
  • publish grey / normalized

• both options:

  • run voxl-qvio-server, which will load voxl-qvio-server.conf
  • playback imu into voxl-qvio-server
  • qvio server loads camera calibration and extrinsics
  • qvio server outputs vio pose
  • use voxl-logger to log the output of qvio
  • analyze the results of vio logs

Misc Notes

• QVIO performs better if the imu data is temperature compensated
  • during drone take-off, the imu temperature and therefore imu biases can change quickly and QVIO may have trouble tracking the biases
  • the bias correction can also be done offline and applied to the logged data imu data to produce better bias-compensated imu data (gyro, accel)
• Consider also logging the global shutter camera (AR0144?) at the same time as IMX412, so that it is possible to compare output of QVIO using a global shutter camera vs rolling shutter.

Hi @Rowan-Dempster ,

The only things that you cannot change after the raw10 images are collected are:

• raw frame resolution and any other camera settings (which we typically don't change anyway)
• exposure and gain, as they are controlled by the auto exposure algorithm in real time
• the auto exposure algorithm only runs if at least one processed stream is requested (yuv, encoded, normalized, etc), you can see the logic here. If the camera server only has a client for the bayer image, the Auto Exposure will not be updated, which is not good. The solution should be:
  • either log or use voxl-inspect-camera or any other client to get the yuv or grey stream going (or even encoded)
  • log raw10 bayer only, but you could also log the output video or YUVs
  • *** i will look into potentially adding a param to force the camera to always do AE processing even if there are no clients for the processed images (essentially disable idling).

I would suggest not collecting too many data sets until you actually get the pipeline working, as you may realize that there is an issue in the data sets or something is missing. Probably best to focus on the processing pipeline..

• You cannot simultaneously receive 2x2 binned and not binned images from the camera, so it either has to be in unbinned mode (4040x3040 resolution) or binned (1996x1520). You can always do binning / resizing in misp, but the readout time will change for the larger resolution, as we discussed before. So one thing to test would be potentially putting two IMX412 cameras side by side and simultaneously logging at 4040x3040 (cam1) and 1996x1520 (cam2), and then run offline processing pipeline to see if QVIO can indeed compensate properly for the larger rolling shutter skew. If you are able to run QVIO using full 4040x3040 resolution, then you can have EVERYTHING : 4k video, EIS, vio... you can still run EIS and save videos from the binned resolution, but they will be lower quality.

Another thing to keep an eye on is what you are logging (bandwidth). I did some tests a while ago and voxl2 write speed to disk is quite high, about 1.5GB/s, but you can run out of disk space pretty quickly. However, it should definitely be able to log 4040x3040@30 + 1996x1520@30 (or @60). will probably need to use the ion buffers to log the raw10 images (which is supported, i just need to test), to skip the overhead for sending huge images over the pipes. Camera server already publishes the raw bayer via regular pipes and also ion buffers.

I am going to look over the components needed for this and make sure they are in good state:

• merge the new logging modes to dev (voxl-logger, voxl-replay)
• make a cleaner example of simple standalone misp to include AWB, although you don't need AWB for VIO

But either way, you should be able to start with logging tests and just see if you can playback the logs and get QVIO to do something reasonable from the log playback on target. To bypass MISP, you could just log the output of MISP for now (grey or normalized), so you have less components in the initial test pipeline. then build it up to include debayering, etc.

Perhaps it is easy to log the raw10 bayer + grey or normalized image, so that you have the AE issue solved as well (making sure the AE is running), then for playback you can choose either normalized (feed directly into voxl-qvio-server) or raw10, which will need some debayering and other processing. It is good to have choices. But i would suggest starting with lower resolution first until you also double check ability to log 4K raw images.

Alex

@Rowan-Dempster ,

Please see the following commit where we recently enabled publishing the normalized frame from IMX412 and IMX664 camera via regular and ION buffers: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/c42e2febbc6370f9bbc95aff0659718656af6906

The parameters for 1996x1520 look good, basically you will be getting 2x2 binned (full frame) and then further down-scale to 998x760. since you are doing exact 2x2 downscale in misp, you can also remove interpolation, which will make the image a bit sharper, you can see this for reference: link -- basically change the sampler filter mode from linear (interpolate) to nearest. If you use non-integer down-sample, keep the linear interpolation.

Regarding the resolution to use with VIO.. i think the 998x760 with nearest sampling should behave the same or better than AR0144 with 1280x800 resolution, mainly because the IMX412 has a much bigger lens (while still being pretty wide), so image quality is going to be better (of course, you need to calibrate intrinsics). Also the 4:3 aspect ratio may help capture more features in the vertical direction. That of course does not account for rolling shutter effects..

There can definitely be benefit in going up in resolution and using 1996x1520, but you kind of have to use the extra resolution correctly.. typically you would detect features on lower resolution image and then refine using the full resolution (also for tracking features). However, in practice, often some small blur is applied to the image to get rid of pixel noise, etc, so very fine features won't get picked up. Unfortunately, we do not know exactly what QVIO does internally. it may do some kind of pyramidal image decomposition to do these things in a smart way. You should try it and check the cpu usage.

Using MISP you can downsample and crop (while maintaining aspect ratio) to any resolution, so it's easy to experiment.

If i had to test QVIO at different resolutions, i would log raw bayer images and imu data using voxl-logger and then use voxl-replay + offline misp + offline qvio to run tests on the same data sets with different processing parameters. This may sound complicated, but it's really not:

• voxl-logger can log raw10 frames (on this branch) : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-logger/-/tree/extend-cam-logging
• qvio relies on timestamps from incoming messages, so it can work with live data or playback data
• the only missing piece is offline MISP implementation, which is partially available in this tool : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/add-new-image-tools/tools/voxl-convert-image.cpp -- and we are working on being able to run exactly the same implementation as in camera-server. The only missing piece in the code listed here is AWB -- however, white balance should not affect VIO too much, since only Y channel is used, so you can set white balance gains to a fixed value.
• running misp offline allows you to experiment with different resolutions / processing directly from the source raw10 bayer image (lossless)

So, if you are really serious about using hires camera for QVIO, since there are a lot of unknowns, you should consider setting up an offline processing pipeline, so that you can run repeatable tests and parameter sweeps. It requires some upfront work, but the pay-off will be significant. You can also use the offline pipeline for regression testing of performance and comparing to other VIO algorithms (which just need the MPA interface). We can discuss this topic more, if you are interested.

imx412_fpv_eis_20250919_drivers.zip are the latest for IMX412. We should really make them default ones shipped in the VOXL2 SDK, but we have not done it.

Since you are maintaining your own version of voxl-camera-server, you should add them to your voxl-camera-server repo and .deb and install them somewhere like /usr/share/modalai/voxl-camera-server/drivers. then modify the voxl-configure-camera script to first look for imx412 drivers in that folder and then fallback to searching /usr/share/modalai/chi-cdk/. In fact this is something I am considering, as maintaining camera drivers in the core system image is less flexible.

EDIT: i guess the older version of the imx412 drivers are already in the repo, so you can just replace them with new ones in your camera server repo: link

Let me know if you have any more questions. Sounds like a fun project

Alex

@Myles-Levine , if you are masking out 90% of down-facing camera, then it is probably useless any movement of the features will get it out of the unmasked region very easily and feature will be dropped.. It may hurt VIO as there may be features going in and out, only trackable for a few frames (just adds to the complexity).

You should try to move the down-facing camera to free up it's FOV or even have it facing slightly angled to the back of the drone..

Alex

Hello @Catalystmachine ,

Sorry for the confusion. We have updated the page to be more clear that Boson is not included, see below: "(No Boson)" was added to the kit description.

The title of the product already states "VOXL 2 MIPI Boson+ Adapters for Thermal IR FPV" as well as the Kit description used to say (Before we added (No Boson)) : "PCB Adapters Only..."

The price of the Boson cameras varies significantly, depending on the specifications. You can contact a distributor to get more details. For example : https://www.oemcameras.com/product-category/thermal-imaging-cameras/thermal-imaging-cores/flir-boson-series-htm/boson-plus/ (you can find more at https://oem.flir.com/contact/find-a-dealer/)

Please note that Boson+ series of the Boson sensor is required for compatibility with VOXL2.

Regarding the pricing of the adapter kit, we do offer discounts in higher volumes, if you are interested, please send us a request: https://www.modalai.com/pages/contact-us

@Kashish-Garg-1 , please see the following discussion regarding M0149 without an IR filter as well as a lens part to reference if you want to buy replacement lenses only.

If you would like to use M0149 (as opposed to M0166), the best approach would be to buy the M0149 cameras in original configuration (with IR filters) as well as additional lenses without IR filter and then swap the lenses (use contact form for a custom order with lenses). Sorry, we won't modify the standard M0149 units. I hope this helps!

Please note that lenses that are shipped with M0166 are about 1mm shorter in Z direction (along the optical axis) compared to the lenses shipped with M0149, however the lens specs are very similar. Please perform testing to ensure compatibility for your application.

https://forum.modalai.com/topic/4826/msu-m0149-1-ir-filter

Alex

@groupo , sorry, you are right, an update is overdue. Please give us a few more days to provide an official response.

Alex

Hi @austin-c ,

I ran some tests with Tmotor MN4006-23 380kV and 15-inch MS1503 propeller, using M0138 ESC. Tests were done with a battery and a power supply. Please see results below.

• the colors may vary from plot to plot, but the one with slower deceleration is "regen off"
• did not run large rpm steps on power supply because the power supply was already current limited
• regen peaks were 31V (pwm step) and 33V (rpm step) for the power supply test
• motor acceleration is almost identical for regen on/off tests
• very large difference in motor deceleration, especially for the large high-to-low rpm transitions
• also note that the power supply tests result in slower deceleration even with regen on (because the power supply cannot sink the current)
• TVS diode did not blow out in this test, because the spikes were short and only a few at a time

Please note that, in theory, it is possible to decelerate the motor faster without regenerative braking (by applying more power, but out of phase), however we do not have this implemented.

If the motor deceleration is slowed down due to lack of regenerative braking, the attitude control will be affected because the motor response is highly asymmetric for acceleration vs deceleration. Overall, this will result in effectively reducing the average responsiveness of the motor, so you would need to decrease control gains and probably will not be able to handle large disturbances very well and cause oscillations if not tuned properly.

Please review the plots (which are not surprising), and we can discuss further..

Alex

6S battery, Power step test:

6S battery, RPM small step test

6S battery, RPM large step test

11A Power Supply, Power step test:

11A Power Supply, RPM small step test

@Myles-Levine ,

You should not have any unmasked static objects appear in the camera view because they may have features that will be tracked and will confuse the algorithm because these features will be stationary with respect to the drone even if the drone is moving.

open vins should support masking parts of the camera(s) to ignore any features in this region. I am not sure how to enable the mask, will double check with colleagues (you could also do a search..).

worst care scenario, the part of the image to be ignored should be blacked out before feeding into open vins - that would be the solution if the mask is not supported. Hopefully that is not needed and mask can be used.

Alex

@SKA , I am not sure. please disable all streams except preview and try again. specifically disable:

• small_video
• large_video
• snapshot

Double check to make sure:
en_raw_preview: true
en_preview: true
en_misp : true

It seems you have at least small_video enabled.

What camera resolution are you requesting? (preview_width, preview_height)

Alex

@ey , please send us a request via http://www.modalai.com/pages/contact-us

Alex

@austin-c , i don't have any concerns for disabling active freewheeling other than the fact that the responsiveness of the ESC will be reduced, as the RPM reduction will be purely due to air drag acting on the propeller (and small friction in the motor).

I can set up the MN4006 motor with a 13 or 15 inch propeller, do a RPM control tune and compare the RPM response results with regen on and off. This is actually pretty quick to do, i will try to do it in the next few days.

Please ping me if you don't hear back by early next week

Alex

@Jetson-Nano , since we currently have no plans to do the MIPI integration of this sensor, you always have the option to use USB connection and voxl-uvc-server to publish images via MPA. You would need to check if the sensor supports the standard UVC interface and try it out...

Alex

Also, you should note - the motor's open loop RPM speed increases at the same power setting -- this is a consequence of not having active freewheeling. This does not mean, though, that if active freewheeling (regen braking) is disabled, that you would get higher max rpm or more efficiency. It just changes the motor pwm % to rpm mapping.

Alex

@austin-c , i just checked and actually we have an option to disable regen braking, it is just not available yet via a param. Here is the test result with same setup (no prop) with regen braking disabled. You can see that the motor takes a long time to coast down, but there is no regen voltage spike or regen current at all. If you application does not require rapid rpm change (high to low), then it will work fine.

We have not tested this in a while, but at some point we did testing for an application with a smart battery which did not like being re-charged with the regen current.

I will see if we can enable this via param or worst case scenario I can share the latest firmware with regen off.

Regarding your other questions:

• i don't think you can put enough capacitance to really absorb all the charge that needs to be stored.. You could do some basic calculations using the current spikes to see how much capacitance you would need, but it think it would be a LOT.
• regarding multiple TVS diodes, yes just connect an array of them in parallel, simply to distribute the energy (heat) across multiple diodes to prevent them from overheating (also can choose larger package, but they all would have to have same specs).

Also, since your motor is low kv, i just want to make sure you saw this document (you probably did, just double checking) : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-esc/-/blob/master/voxl-esc-tools/doc/low_kv_motor_tuning.md

Actually the app note for low kv tuning was using the Tmotor MN4006 motor with even larger 15 inch propeller (see Tuning Example section). It should work fine with appropriate parameters, we can help tuning if you tell me the exact propeller.

motor MN4006-23 380kV, 18N24P winding configuration
12 pole pairs (24/2)
15-inch propeller MS1503
6S battery voltage

Alex

I went ahead and ran a few tests with a 11A power supply set to 25V.

Using M0138 ESC, Xing2 2207 1855kV motor and 5x4x3 propeller

Both tests using the same voxl-esc command, executed on a linux pc that is connected to the ESC directly via usb to serial adapter.

Test command:

./voxl-esc-spin-step.py --id 2 --power 20 --step-amplitude 30 --step-frequency 2.0 --timeout 2.0 --cmd-rate 2000 --enable-plot 1

Results / notes:

• no load test showed a higher peak of 33.8V vs about 31.8V loaded with prop, which actually makes sense because the no load motor RPM is higher (larger back-emf).
• only testing one channel at a time to reduce voltage spike
• start with small step to get a sense of regenerative voltage rize
• short test to avoid any damage
• ran the test on ESC id 2, so that total current is measured and displayed
• this motor has pretty high kV and small size propeller, so it decelerates quickly. However, other motor and larger propeller combinations (and larger rpm changes) may generate a lot larger spike (if the motor spends more time at higher rpms).
• also did a test to see at which point the ESC starts drawing more current - TVS diode starts conducting (short voltage increase on the power supply, i.e. for 1 second). As the TVS diode warms up, the current drops a bit because of the diode characteristics (breakdown voltage increases with temperature). However, the performance of TVS diode varies significantly across the voltage range, so it cannot be used for any exact measurements.
  • 25-27.5V : 0.02A
  • 28V: 0.035A
  • 29V: 0.09A
  • 30V: 0.15A
  • 31V: 0.20A
    ..

Test 1: no propeller

Test 2: with propeller

Hi @austin-c ,

Please see answers below:

• brake_to_stop parameter does exit and it works. setting it to 0 will result in the motor coasting to a stop when it is commanded to stop or there is a command timeout. it is used here : link
• the hardware supports disabling regenerative braking (we have tested this before), but the software does not support disabling regenerative braking during motor control
• the TVS breakdown voltage is around 29-30V. You can actually test this by slowly increasing voltage on the power supply (in 0.1V increments) and observe the current draw increase from nominally low value (be careful)

If you would like to test with a power supply, that's fine, but you have to be careful about the types of tests you are doing. Please understand the following:

• regenerative braking will happen any time there is a transition from higher rpm to a lower rpm
• this will happen regardless whether the ESC is rpm control mode or power control because that is a principle of "synchronous rectification" which is a technique for controlling mosfets (widely used on most ESCs). Any time the back-emf of the motor is higher than the average applied voltage, the regenerative current will flow back into the battery or power supply (roughly speaking).

In general, here are our guidelines for testing with a power supply

• keep in mind that the dynamic response (even with rpm increase) will be different from a battery, as the battery typically has lower output impedance from a power supply
• testing steady state response or continuous spinning with slow transitions is a perfect applications
• aggressive transitions from low to high rpm (or from low to high power) are ok
• avoid aggressive transitions from high to low rpm, since that's when the back-emf will be generated
• we actually have a test script voxl-esc-spin-step.py which shows you how to you can use the control software to slew rate limit the commands to avoid fast up or down transitions. you can use a similar approach for your bench testing.
• exploring the TVS diode specs (clamping voltage) can be done using an unloaded motor, where the energy stored in the motor is small, so the TVS diode can absorb the regen energy. You can do a small step test with unloaded motor and see what what voltage the voltage spike gets clipped. There is a risk of burning out the TVS diode if the test is done with a loaded motor with a large high->low transition
• avoid loaded motor step-down tests while using a power supply
• set the brake_to_stop esc parameter to 0 and stop the script or send zero rpm/ power in order to stop the motor (or just stop sending commands) instead of sending low rpm before 0 rpm.
• adding external capacitance and an additional array of similar TVS diodes (placed as close as possible to the ESC) will help reduce (using capacitors) and absorb (additional tvs diodes to distribute the heat) the regen spikes .
• sink-capable power supplies are usually expensive, may not be worth it, depending on what you are trying to do. If you are trying to test performance of the motor using a power supply that models the battery (a battery simulator), that would work, but would be even more expensive

What kind of tests are you planning to run? I can provide more guidance.

Alex

@mark , it looks like your version of voxl sdk has an early version of the imx664 driver (not supporting 2704x1540 resolution.

Are you able to upgrade sdk to 1.5.x or 1.6.x?

Alternatively, i could share the latest camera drivers to use with 1.4.5.

I would also recommend updating voxl-camera-server to the latest, but that would also require updating other packages.. would you consider upgrading the sdk or are you locked to 1.4.5?

Alex

@Leo-Allesch , if you are talking about VSLAM, which stands for Visual Simultaneous Localization and Mapping, then this is a sub-class of SLAM which uses only cameras (and often IMUs, then it is VISLAM).

VSLAM and VISLAM usually generate and maintain a sparse map of 3D features (points), which is used for self localization.

If you want to be able to detect a target, and you have your 3D pose from VIO (VIO = VISLAM), then you can either triangulate the target location or get it's location from a single frame if you know the size of the target (such as April tag, etc). Then you can transform the coordinate of the tag from the camera frame into your "world" (VIO) coordinates.

The depth sensor would allow you to build a more detailed 3D map / point cloud using the 3D pose output from the VIO algorithm (or just using a depth sensor, which may be more complicated). You can also build a dense 3D map using camera only (and IMU), but that typically requires a lot more computation than using a depth sensor.

You don't necessarily need a depth sensor for identifying the position of a specific target that is easily detected by a camera.

There is a lot of research material you can find on these topics, please look into it deeper and let us know if you have specific questions about using VOXL2 hardware and software.

Alex

@mark ,

Please try using input resolution of 2704x1540, not the binned one which you have currently selected. Then you can set MISP width and height whatever you need and the output should be properly downscaled and cropped. Choosing higher input resolution will result in a better downscaled image due to debayering artifacts at lower resolution.

The reason why the image decoding does not work has to do with the image line stride of the raw10 buffer not matching the requirement of the opencl debayering functions. In newer versions of camera server, we automatically re-align the image to make it work, but that requires copying the whole bayer image into a different buffer (more cpu overhead).

Using input resolution of 2704x1540 will result in best results in terms of image quality and cpu usage. This resolution works up to 120FPS. Binned mode does not actually provide any advantages (specifically for the IMX664 camera), not even faster read-out or higher max fps or anything like that.

Please try that and let me know if that works for you.

Alex