Add mask file to QVIO algorithm

david.moro

The mvVISLAM library supports a mask file to remove regions like legs, landing gear, etc. Adding that to the voxl-qvio-server would be very helpful. The file could live in /data/modalai/mask.pgm or something like that. If no file is present, then no mask is used.

A possible sticky point is that the Qualcomm API does not make it clear what resolution the mask needs to be if we are using anything other than a VGA sensor, for example, the OV9782 sensor.

Alex Kushleyev

Hello @david-moro ,

mvVISLAM library supports a mask file in the following format : 8 bits per pixel PGM with white (255) being where NO features should be found and black (0) where it is OK to have features. For a frame of aspect ratio 4:3, typically a mask of size 160x120 pixels should be used, although you can have a higher resolution mask if you want to. The mask will be resized to the actual image size or the smaller resolution mask will be properly queried (i do not know which one, but it should not matter). I would just suggest keeping the aspect ratio of the mask the same as the actual image.

You can test this easily by creating a mask that splits the image in half and seeing if features get detected in masked out region. Please note that the mask only affects detection of new features -- if the feature was detected in unmasked region and then the feature ends up in the masked region, it will still be tracked.

It does not look like a mask file is an option in voxl-qvio-server at the moment. You could try to add it yourself, here is an example how it was used in another project : https://github.com/ATLFlight/snap_vio/blob/master/src/snap_vio.cpp#L345C19-L345C19 . If mvVISLAM is not able to open the mask file you provided (wrong path, etc), it should output an error message like this : AR ERROR: arFileOpen(): Failed to open file: <path to mask file name>

I hope this helps!

Alex

david.moro

@Alex-Kushleyev . Just went through the process of compiling the code and pushing it to the VOXL2. I can confirm the mask works (i.e. no new points in the masked region, however, points can move into it if captured before). One thing to note is that the qvio server did not complain when the file was missing. Maybe I am not looking at the right output though.

Thanks a lot for the help!

afdrus

@david-moro I am trying to do the same thing, where did you locate the .pgm file in the voxl-qvio-server package? In the github example there is no pgm file.
Thank you in advance for your response

Alex Kushleyev

@afdrus , there is no mask example in the voxl qvio server package, but here is an example which has half of image black (top) and half white (bottom). You can use it for testing and modify for your application.

edit: the forum is not recognizing the mask pgm file, i will post it another way.

update: the mask example is now located here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/misc/mask_example_160x120.pgm

afdrus

@Alex-Kushleyev Thank you for the help! Can I locate the pgm file anywhere I want on the Voxl2 and specify the absolute path to the pgm in staticMaskFileName or do I have to put the file in a specific path in order for the mvSLAMinitialize function to find it?

Moreover, are there any parameters that can be tweaked to force the qvio to maximize the amount of features across the whole image? When yawing I noticed that the algorithm tends to stick to the features detected in the previous frames rather than picking new ones. As a result, the at the end of the yaw the drone has 20ish features on one side of the image and none on the other side (despite the same lighting conditions) and as a result the quality of the vio can drop below 10% quite easily. Basically I would like the algorithm to switch earlier on new feature candidates rather than keeping old ones in the buffer.

Alex Kushleyev

@afdrus , just to be safe, you should put the absolute path of the .pgm file staticMaskFileName, otherwise the qvio executable will probably try to load from the path that is being executed from. Absolute path would look like /etc/modalai/vio_mask.pgm or similar.

Regarding feature allocation, I don't think that there is an option to tweak this directly. FYI here is a documentation for the all the parameters that qvio algorithm accepts:

https://developer.qualcomm.com/sites/default/files/docs/machine-vision-sdk/api/v1.2.13/group__mvvislam.html#ga6f8af5b410006a44dbcf59f7bc6a6b38

Keep in mind that there are two types of features, what we typically call in-state and not in-state. in-state are the features that are used in the pose solution directly. not in-state features are the features that are tracked but not used in the pose solution. If your vehicle is stationary and performs a yaw, you can detect new features, but it is impossible to figure out how far they are - you need to actually move around. So it is possible, that you are looking at the in-state features only, so after a yaw maneuver, there will be no new in-state features due to lack of motion. So the worst case scenario is if you do a 180 degree yaw without any motion, so you will pick up a lot of new features but would not know how far they are. Luckily, flying vehicles usually do not turn precisely in place, so even a bit of x-y-z motion will help localize the features right away.

If you really wanted to have better tracking during turns, you could consider having a turns with non-zero radius, so that there is a bit of translation in addition to rotation, so that new features can be instantly localized.

You can find more information about feature point quality here : https://developer.qualcomm.com/sites/default/files/docs/machine-vision-sdk/api/v1.2.13/structmv_v_i_s_l_a_m_map_point.html.

Looking at qvio overlay function, it draws both in-state and not in-state features, so you can observe more closely what happens to the features as you rotate..

Aaky

@Alex-Kushleyev How can we create custom mask file for QVIO?

Alex Kushleyev

@Aaky ,

Please check the following thread for more information: https://forum.modalai.com/topic/3459/masking-qvio/

My suggestion:

download the test mask from here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/misc/mask_example_160x120.pgm . which has top half of image black (0), and bottom half white (255). values of 0 is where new features will be allowed to be detected
You can use gimp, to modify the mask and before saving you can threshold it so that the values are all 0 and 255 (nothing in between, just in case).
export from gimp as pgm (raw)
do a test with the original mask example, make sure no features are picked up in the bottom image. This is the baseline test
then try your new mask, you can also overlay the mask on top of the QVIO overlay, just like the other user did for checking.

Also please note (from the other thread) to pay attention to the fact that QVIO mask path is relative to where qvio server starts. So if you start it as a service, it's working directory will be / but if you run it manually, the working directory will be your current directory in the terminal - the other thread explains that issue.

Alex