@Jgaucin voxl-tflite-server is just a wrapper for standard TensorFlow Lite that has been compiled with the proper configurations to take advantage of QRB5165.
voxl-tflite-server code
So, if you can achieve what you are trying to do using TensorFlow Lite on the desktop, then you should be able to bring over to VOXL 2 in a straightforward manner.
The inference_worker function here is where the models are processed