ToF v2 outdoor noise

[MattK]

Hey there,

I've been testing out the new ToF v2 sensor and have been encountering a lot of noise when trying to use it outdoors in the sun.

Attaching some depth image screenshots of the noise that I'm referring to. In both images, you can see me standing on ground, with a bunch of staticky noise all over the place.

I'm not surprised that noise gets introduced when sunlight is added to the equation, but I also noticed that the new Starling Max v2, which is advertised as an outdoor drone, claims to rely on this same ToF sensor for obstacle avoidance:

The Starling Max v2 also has dual color image sensors, so I'm wondering if this was a misprint and it actually uses depth-from-stereo for outdoor obstacle avoidance. Either that, or maybe my ToF sensor is damaged?

I've started to work on implementing my own filtering to voxl-camera-server (looks promising so far, but still more work to do), but I wanted to sanity check myself in case something else is wrong.

[MattK]

So I actually was able to filter out the noise pretty well by adding a temporal filter to confidence and depth values in PerCameraMgr::RoyaleDataDone in hal3_camera_mgr.cpp.

^ I used an exponential moving average with alpha = 0.25. After updating the EMA buffer, I cleaned things up further by reducing all confidence values below an ~80% threshold (200/255) to zero,

That seems to be working pretty well at reducing/ignoring noise artifacts from sunlight while still being able to see large objects.

I've only tested it out on the ground so far though, and expect to encounter reliability issues in the air when everything is in motion. I've already noticed that when I walk around in frame, I ghost and become lost in the noise due to the temporal nature of the filter. I'll probably tweak things more and try to experiment with median/spatial filters as well, but for now this at least seems like it should work for things like large trees, buildings, and the ground, when the drone is flying slowly.

Still interested to know more about how outdoor obstacle avoidance is addressed with the Starling Max v2.

[Alex Kushleyev]

@MattK , there are some filtering settings that can be tweaked in TOF processing libraries. https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/tof_interface.cpp?ref_type=heads#L1017

@James-Strawson, do you have any recommendations for better tuning parameters for outdoor use case?

Alex

[Moderator]

@Alex-Kushleyev Generally these sensors are not expected to work well outdoors as they are IR-based. The v2 modules operate at 940nm, which should be a little better than the previous gen which operated at 850nm. That being said, the sun will create a lot of noise in the image. You should try the shortest range mode possible

[James Strawson]

That's a typo on the website we've just corrected. The Starling 2 Max does not include a TOF sensor, and this sensor is not intended for outdoor use.

I spent some time tuning the TOF point cloud processing filter params on Friday to reduce noise. These changes are in the dev branch now: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/919c24bef21cbea2eb7be33853884f5f606a0763

These these changes are likely to help in the outdoor use case too so I suggest giving it a go.

[MattK]

@James-Strawson Hmmm, unfortunate typo. Does the Starling Max 2 use its dual IMX412 color image sensors to get depth-from-stereo, in order to do outdoor VOA? I wasn't too impressed with the quality of outdoor DFS that I got from the M0015, but perhaps the IMX412s are better?

Appreciate the extra help/guidance on TOF processing, I'll try take a look. Even though the outdoor use case is not ideal, I'm still leaning towards trying to make the TOF sensors work for us, as the results I'm getting still seem more promising than what I was getting from stereo.

[Alex Kushleyev]

@MattK ,

James is out of office, so i will respond. The dual IMX412 will not be used for DFS, in fact the cameras will be pointed in different directions (one forward, one down).

Also, IMX412 are rolling shutter cameras, which would make it difficult to get accurate DFS while in motion.

[MattK]

Thanks, @Alex-Kushleyev .

If that's the case, then what kind of VOA solution is present on the Starling 2 Max?

[Moderator]

@MattK There is currently no VOA capability on Starling 2 Max

[MattK]

@Moderator Oh... interesting. You might want to remove this bullet point from its sales page then. https://www.modalai.com/products/starling-2-max?variant=48172375310640

Would you be able to recommend a drone with a working VOA solution? We're interesting in getting one to get a baseline for the quality we can expect from VOXL2's built in VOA SDK. I'm particularly interested in assessing depth-from-stereo, I've gotten good enough results for now from the TOF v2 sensor. But TOF recommendations are welcome as well.

[Moderator]

@MattK said in ToF v2 outdoor noise:

to remove this bullet point from its sales pag

Thank you, fixed

[dlee]

@MattK Have you ever tested the EXTRA_LONG_RANGE mode of tof v2? I am also testing VOA outdoors using tof sensor. But it was not enough for me. I tested all of the modes in v2, Extra long-range mode has less noise than other modes. Even though extra long-range mode is better than others, VOA using tof v2 makes the drone stop 2m away from obstacles.

[MattK]

@dlee Ah, no I have not. I'll give that a try when I get the chance. Thanks!

[Jetson Nano]

@MattK @Moderator
If the Starling V2 Max doesn't support a DFS server due to IMX412 being rolling shutter cameras and no TOF, then how is the 3D map generated?
Since in my understanding, to generate a map you need depth values/ point cloud data.
If the depth estimates are being obtained by some other means then why can't the same point cloud be made available for VOA in vision-hub server?