Add mask file to QVIO algorithm
-
The mvVISLAM library supports a mask file to remove regions like legs, landing gear, etc. Adding that to the voxl-qvio-server would be very helpful. The file could live in /data/modalai/mask.pgm or something like that. If no file is present, then no mask is used.
A possible sticky point is that the Qualcomm API does not make it clear what resolution the mask needs to be if we are using anything other than a VGA sensor, for example, the OV9782 sensor.
-
The mvVISLAM library supports a mask file to remove regions like legs, landing gear, etc. Adding that to the voxl-qvio-server would be very helpful. The file could live in /data/modalai/mask.pgm or something like that. If no file is present, then no mask is used.
A possible sticky point is that the Qualcomm API does not make it clear what resolution the mask needs to be if we are using anything other than a VGA sensor, for example, the OV9782 sensor.
Hello @david-moro ,
mvVISLAM library supports a mask file in the following format : 8 bits per pixel PGM with white (255) being where NO features should be found and black (0) where it is OK to have features. For a frame of aspect ratio 4:3, typically a mask of size 160x120 pixels should be used, although you can have a higher resolution mask if you want to. The mask will be resized to the actual image size or the smaller resolution mask will be properly queried (i do not know which one, but it should not matter). I would just suggest keeping the aspect ratio of the mask the same as the actual image.
You can test this easily by creating a mask that splits the image in half and seeing if features get detected in masked out region. Please note that the mask only affects detection of new features -- if the feature was detected in unmasked region and then the feature ends up in the masked region, it will still be tracked.
It does not look like a mask file is an option in voxl-qvio-server at the moment. You could try to add it yourself, here is an example how it was used in another project : https://github.com/ATLFlight/snap_vio/blob/master/src/snap_vio.cpp#L345C19-L345C19 . If mvVISLAM is not able to open the mask file you provided (wrong path, etc), it should output an error message like this :
AR ERROR: arFileOpen(): Failed to open file: <path to mask file name>I hope this helps!
Alex
-
Hello @david-moro ,
mvVISLAM library supports a mask file in the following format : 8 bits per pixel PGM with white (255) being where NO features should be found and black (0) where it is OK to have features. For a frame of aspect ratio 4:3, typically a mask of size 160x120 pixels should be used, although you can have a higher resolution mask if you want to. The mask will be resized to the actual image size or the smaller resolution mask will be properly queried (i do not know which one, but it should not matter). I would just suggest keeping the aspect ratio of the mask the same as the actual image.
You can test this easily by creating a mask that splits the image in half and seeing if features get detected in masked out region. Please note that the mask only affects detection of new features -- if the feature was detected in unmasked region and then the feature ends up in the masked region, it will still be tracked.
It does not look like a mask file is an option in voxl-qvio-server at the moment. You could try to add it yourself, here is an example how it was used in another project : https://github.com/ATLFlight/snap_vio/blob/master/src/snap_vio.cpp#L345C19-L345C19 . If mvVISLAM is not able to open the mask file you provided (wrong path, etc), it should output an error message like this :
AR ERROR: arFileOpen(): Failed to open file: <path to mask file name>I hope this helps!
Alex
@Alex-Kushleyev . Just went through the process of compiling the code and pushing it to the VOXL2. I can confirm the mask works (i.e. no new points in the masked region, however, points can move into it if captured before). One thing to note is that the qvio server did not complain when the file was missing. Maybe I am not looking at the right output though.
Thanks a lot for the help!
-
A Alex Kushleyev referenced this topic on
-
@Alex-Kushleyev . Just went through the process of compiling the code and pushing it to the VOXL2. I can confirm the mask works (i.e. no new points in the masked region, however, points can move into it if captured before). One thing to note is that the qvio server did not complain when the file was missing. Maybe I am not looking at the right output though.
Thanks a lot for the help!
-
@david-moro I am trying to do the same thing, where did you locate the .pgm file in the voxl-qvio-server package? In the github example there is no pgm file.
Thank you in advance for your response@afdrus , there is no mask example in the voxl qvio server package, but here is an example which has half of image black (top) and half white (bottom). You can use it for testing and modify for your application.
edit: the forum is not recognizing the mask pgm file, i will post it another way.
update: the mask example is now located here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/misc/mask_example_160x120.pgm
-
@afdrus , there is no mask example in the voxl qvio server package, but here is an example which has half of image black (top) and half white (bottom). You can use it for testing and modify for your application.
edit: the forum is not recognizing the mask pgm file, i will post it another way.
update: the mask example is now located here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/misc/mask_example_160x120.pgm
@Alex-Kushleyev Thank you for the help! Can I locate the pgm file anywhere I want on the Voxl2 and specify the absolute path to the pgm in staticMaskFileName or do I have to put the file in a specific path in order for the mvSLAMinitialize function to find it?
Moreover, are there any parameters that can be tweaked to force the qvio to maximize the amount of features across the whole image? When yawing I noticed that the algorithm tends to stick to the features detected in the previous frames rather than picking new ones. As a result, the at the end of the yaw the drone has 20ish features on one side of the image and none on the other side (despite the same lighting conditions) and as a result the quality of the vio can drop below 10% quite easily. Basically I would like the algorithm to switch earlier on new feature candidates rather than keeping old ones in the buffer.
-
@Alex-Kushleyev Thank you for the help! Can I locate the pgm file anywhere I want on the Voxl2 and specify the absolute path to the pgm in staticMaskFileName or do I have to put the file in a specific path in order for the mvSLAMinitialize function to find it?
Moreover, are there any parameters that can be tweaked to force the qvio to maximize the amount of features across the whole image? When yawing I noticed that the algorithm tends to stick to the features detected in the previous frames rather than picking new ones. As a result, the at the end of the yaw the drone has 20ish features on one side of the image and none on the other side (despite the same lighting conditions) and as a result the quality of the vio can drop below 10% quite easily. Basically I would like the algorithm to switch earlier on new feature candidates rather than keeping old ones in the buffer.
@afdrus , just to be safe, you should put the absolute path of the .pgm file staticMaskFileName, otherwise the qvio executable will probably try to load from the path that is being executed from. Absolute path would look like
/etc/modalai/vio_mask.pgmor similar.Regarding feature allocation, I don't think that there is an option to tweak this directly. FYI here is a documentation for the all the parameters that qvio algorithm accepts:
Keep in mind that there are two types of features, what we typically call
in-stateandnot in-state.in-stateare the features that are used in the pose solution directly.not in-statefeatures are the features that are tracked but not used in the pose solution. If your vehicle is stationary and performs a yaw, you can detect new features, but it is impossible to figure out how far they are - you need to actually move around. So it is possible, that you are looking at thein-statefeatures only, so after a yaw maneuver, there will be no newin-statefeatures due to lack of motion. So the worst case scenario is if you do a 180 degree yaw without any motion, so you will pick up a lot of new features but would not know how far they are. Luckily, flying vehicles usually do not turn precisely in place, so even a bit of x-y-z motion will help localize the features right away.If you really wanted to have better tracking during turns, you could consider having a turns with non-zero radius, so that there is a bit of translation in addition to rotation, so that new features can be instantly localized.
You can find more information about feature point quality here : https://developer.qualcomm.com/sites/default/files/docs/machine-vision-sdk/api/v1.2.13/structmv_v_i_s_l_a_m_map_point.html.
Looking at qvio overlay function, it draws both
in-stateandnot in-statefeatures, so you can observe more closely what happens to the features as you rotate.. -
-
@afdrus , just to be safe, you should put the absolute path of the .pgm file staticMaskFileName, otherwise the qvio executable will probably try to load from the path that is being executed from. Absolute path would look like
/etc/modalai/vio_mask.pgmor similar.Regarding feature allocation, I don't think that there is an option to tweak this directly. FYI here is a documentation for the all the parameters that qvio algorithm accepts:
Keep in mind that there are two types of features, what we typically call
in-stateandnot in-state.in-stateare the features that are used in the pose solution directly.not in-statefeatures are the features that are tracked but not used in the pose solution. If your vehicle is stationary and performs a yaw, you can detect new features, but it is impossible to figure out how far they are - you need to actually move around. So it is possible, that you are looking at thein-statefeatures only, so after a yaw maneuver, there will be no newin-statefeatures due to lack of motion. So the worst case scenario is if you do a 180 degree yaw without any motion, so you will pick up a lot of new features but would not know how far they are. Luckily, flying vehicles usually do not turn precisely in place, so even a bit of x-y-z motion will help localize the features right away.If you really wanted to have better tracking during turns, you could consider having a turns with non-zero radius, so that there is a bit of translation in addition to rotation, so that new features can be instantly localized.
You can find more information about feature point quality here : https://developer.qualcomm.com/sites/default/files/docs/machine-vision-sdk/api/v1.2.13/structmv_v_i_s_l_a_m_map_point.html.
Looking at qvio overlay function, it draws both
in-stateandnot in-statefeatures, so you can observe more closely what happens to the features as you rotate.. -
@Alex-Kushleyev How can we create custom mask file for QVIO?
@Aaky ,
Please check the following thread for more information: https://forum.modalai.com/topic/3459/masking-qvio/
My suggestion:
- download the test mask from here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/misc/mask_example_160x120.pgm . which has top half of image black (0), and bottom half white (255). values of 0 is where new features will be allowed to be detected
- You can use gimp, to modify the mask and before saving you can threshold it so that the values are all 0 and 255 (nothing in between, just in case).
- export from gimp as pgm (raw)
- do a test with the original mask example, make sure no features are picked up in the bottom image. This is the baseline test
- then try your new mask, you can also overlay the mask on top of the QVIO overlay, just like the other user did for checking.
Also please note (from the other thread) to pay attention to the fact that QVIO mask path is relative to where qvio server starts. So if you start it as a service, it's working directory will be
/but if you run it manually, the working directory will be your current directory in the terminal - the other thread explains that issue.Alex
-
J Jetson Nano referenced this topic on
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login