TFLite multiple model execution
-
Hi ModalAI Team,
Can you provide guidance on running multiple TFLite models onboard the QRB5165 board (voxl-suite 1.1.3-1, tflite 2.8.0-2)? I've read the SDK documentation but the following is unclear to me:
1. Do you need to configure and run multiple instances of voxl-tflite-server?
I've tried this by adding additional service definitions into /etc/systemd/system (e.g., one called voxl-tflite-server-pose.service) and running via systemctl. Does each instance need to be pointed at a discrete config file (e.g., /etc/modalai/voxl-tflite-server-pose.conf)?
On testing this approach I was able to run several parallel instances of voxl-tflite-server, but they all seem to reference the default config file (/etc/modalai/voxl-tflite-server.conf), where I can only define a single model. I'm unable to specify a config file in the systemd.service file as the -c option seems to be reserved for config (i.e., it doesn't actually run the model).
2. Do you use a single config file and voxl-tflite-server service to run multiple models?
Given that the voxl-tflite-server service seems locked to the voxl-tflite-server.conf file, do you instead specify multiple models in the config file (setting allow_multiple and output_pipe_prefix accordingly) and let the service manage system resources etc? If this is the case, have you got an example of the syntax? (I tried adding a second model definition to the conf but it didn't work).
3. Am I missing something like the need to manually publish another pipe from the additional servers?
Thanks,
Nick
-
Will try to help you out as best as I can here!
Yes, you're on the right track with running them as independent services with systemctl. You'll likely notice that they do all read from the same conf file and so what I'd recommend here is modifying the source code to point to a different config file location for each. I can help you find the section of relevant code if needed. This will give you different services with different configuration files. You've also correctly guessed here that you'll need all to publish to different output pipes so that they aren't overwriting each other.
Hope this helps!
Thomas
-
Thanks @thomas, I had to edit a few of the files in the voxl-tflite-server repo, but now have it working nicely. This was necessary to ensure that once packaged and deployed the model-specific services each had different names, so didn't overwrite one another on installation. I now have a set of voxl-tflite-server-[model] services that work in parallel (at the cost of GPU heat!).
Nick
-
Awesome, glad to hear it! Yeah, we've actually mounted fans to our boards that run multiple models simultaneously because the thermals definitely get up there. Might be worth thinking about depending on your application.
All the best, let me know if I can help out in any other way.
Thomas