Here's are some requirements and compatibility:

1. RAM Requirements for Qwen2-0.5B

Base model (FP16): ~1GB of RAM
Quantized to INT8: ~500MB of RAM
Quantized to INT4: ~250MB of RAM

VOXL 2 Specifications

The VOXL 2 has approximately 4GB of RAM available
It uses a Qualcomm QRB5165 processor
Has AI acceleration capabilities via the Hexagon DSP
Primarily designed for computer vision tasks

Compatibility Assessment
Qwen2-0.5B could potentially run on the VOXL 2 with quantization to INT4 or INT8, the memory footprint would be manageable.

Inference speed would likely be slow, perhaps 1-2 seconds per token

Question:

Has ModalAI explored using TensorFlow Lite with the Hexagon delegate to run compact language models like Qwen2-0.5B on the VOXL 2 platform? I'm interested in whether you've experimented with leveraging the Qualcomm GPU for language model inference, even though I understand the VOXL 2 is primarily optimized for computer vision workloads. Have you conducted any experiments or performance testing with small LLMs on this hardware?
Thanks.
suvasis

In short, no the VOXL 2 VOA is not using the GPU.

The qrb5165-dfs-server runs on the CVP, which is a computer vision dsp.

The voxl-tflite-server can put different models on CPU, GPU or NPU based on how the model is configured for object recognition.