# Can anyone recommend a Tflite Colab Notebook for VOXL2 Training Source: https://forum.modalai.com/topic/2997/can-anyone-recommend-a-tflite-colab-notebook-for-voxl2-training Category: General Questions (https://forum.modalai.com/category/2/general-questions) Posted: 2024-01-07 00:11:12 UTC by sansoy Replies: 52 · Views: 30454 ## sansoy · 2024-01-07 00:11:12 UTC I’m a total loss with making my Tfiite models work on the VOXL2. I’ve followed every thread. I’ve successfully trained models using the recommended Tensorflow, i’ve quantized it down to 16 float. i can successfully run these on my linux box, my macbook pro, a raspberrypi and the nVidia Jetson Nano. But when uploaded to VOXL2 i do get video but absolutely no detection what so ever. Also per https://docs.modalai.com/voxl-tflite-server/ i followed the instructions for post training quantization on my frozen graph and on my saved models. I used Netron to find the input/output parameters. I also successfully converted a YOLOv8 model into a tFlite model and ran object detection perfectly on all the different platforms except for the VOXL2. Here's a link to one of my colab notebooks i've used to train an object detection model. https://colab.research.google.com/drive/1QdgpSl63OSQdLTnFwOyP8dxLQ7W0HtmW?usp=sharing ## Reply by sansoy · 2024-01-08 20:06:54 UTC I successfully trained a YOLOv5 model using the following instruction sets ![results.png](https://forum.modalai.com/assets/uploads/files/1704744164660-results-resized.png) https://docs.ultralytics.com/yolov5/ https://docs.ultralytics.com/yolov5/tutorials/train_custom_data/ https://docs.ultralytics.com/yolov5/tutorials/model_export/ Followed the directions to export the model to a tflite having FP16 half precision python export.py --weights best.pt --include tflite --half 💡 ProTip: Add --half to export models at FP16 half precision for smaller file sizes and when i bring over to voxl2 I get the following error in /var/logs/syslog Jan 7 18:19:21 m0054 systemd[1]: Started voxl-tflite-server. Jan 7 18:19:21 m0054 bash[18587]: WARNING: Unknown model type provided! Defaulting post-process to object detection. Jan 7 18:19:21 m0054 bash[18587]: INFO: Created TensorFlow Lite delegate for GPU. Jan 7 18:19:29 m0054 bash[18587]: received SIGTERM Jan 7 18:19:29 m0054 systemd[1]: Stopping voxl-tflite-server... Jan 7 18:19:39 m0054 bash[18587]: INFO: Initialized OpenCL-based API. Jan 7 18:19:39 m0054 bash[18587]: INFO: Created 1 GPU delegate kernels. Jan 7 18:19:39 m0054 bash[18587]: ------VOXL TFLite Server------ Jan 7 18:19:40 m0054 bash[18587]: Error in TensorData: should not reach here Jan 7 18:19:40 m0054 bash[18587]: Segmentation fault: Jan 7 18:19:40 m0054 bash[18587]: Fault thread: voxl-tflite-ser(tid: 18670) Jan 7 18:19:40 m0054 systemd[1]: voxl-tflite-server.service: Main process exited, code=killed, status=11/SEGV Jan 7 18:19:40 m0054 systemd[1]: voxl-tflite-server.service: Failed with result 'signal'. Jan 7 18:19:40 m0054 systemd[1]: Stopped voxl-tflite-server. Jan 7 18:19:40 m0054 systemd[1]: Started voxl-tflite-server. Jan 7 18:19:40 m0054 bash[18674]: WARNING: Unknown model type provided! Defaulting post-process to object detection. Jan 7 18:19:40 m0054 bash[18674]: INFO: Created TensorFlow Lite delegate for GPU. Jan 7 18:19:41 m0054 bash[1425]: ERROR in pipe_client_init_channel opening request pipe: No such device or address Jan 7 18:19:41 m0054 bash[1425]: Most likely the server stopped without cleaning up Jan 7 18:19:41 m0054 bash[1425]: Client is cleaning up pipes for the server ## Reply by Guest · 2024-01-09 01:08:45 UTC (in reply to sansoy) @sansoy Hey, happy to try and help resolve this! The first thing I notice is that the voxl-tflite-server is defaulting your model to object detection which seems incorrect as you have a YOLO model. This is because the tflite server does a string compare call to determine which model is being used as seen [here](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/main.cpp?ref_type=heads#L349). What this means is you'll need to rename your model to `yolov5_float16_quant.tflite` for the time being to get proper YOLO processing. Obviously this isn't ideal and we're working on making this functionality better in a future software release. However, I'm more curious about that "should not reach here" message. I've traced that back to `inference_helper.cpp` which is likely hitting [this line](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L56) or one of the other ones which are similar to it. So the TFLite Server is attempting to read in your Tensors in some expected format but it's differing from the type it's getting. What you should try first is the renaming suggestion I mentioned in the first paragraph. It's possible that because it's defaulting to an object detection model and not a YOLO model, that's causing the server to read in your Tensors as the wrong datatype. If that doesn't work, if you could provide me with the output of `cat /etc/modalai/voxl-tflite-server.conf` that might help me in better diagnosing your issue. If I can't help, I may need you to pass along your actual model file so that I can load in your exact configuration and do some debugging to find the issue. Sorry about this! Thomas Patton thomas.patton@modalai.com ## Reply by sansoy · 2024-01-09 15:09:27 UTC (in reply to Guest) @Thomas-Patton Thanks for your response. Didnt realize you were checking for exact names. I renamed my yolo model to be yolov5_float16_quant.tflite and updated the yolov5_labels.txt file but still getting an error. Jan 9 15:06:30 m0054 systemd[1]: Started voxl-tflite-server. Jan 9 15:06:30 m0054 bash[5690]: INFO: Created TensorFlow Lite delegate for GPU. Jan 9 15:06:47 m0054 bash[5690]: INFO: Initialized OpenCL-based API. Jan 9 15:06:47 m0054 bash[5690]: INFO: Created 1 GPU delegate kernels. Jan 9 15:06:47 m0054 bash[5690]: ------VOXL TFLite Server------ Jan 9 15:06:47 m0054 bash[5690]: Segmentation fault: Jan 9 15:06:47 m0054 bash[5690]: Fault thread: voxl-tflite-ser(tid: 5770) Jan 9 15:06:47 m0054 bash[5690]: Fault address: 0x656972623d3d206e Jan 9 15:06:47 m0054 bash[5690]: Unknown reason. Jan 9 15:06:47 m0054 bash[1410]: ERROR in pipe_client_init_channel opening request pipe: No such device or address Jan 9 15:06:47 m0054 bash[1410]: Most likely the server stopped without cleaning up Jan 9 15:06:47 m0054 bash[1410]: Client is cleaning up pipes for the server Jan 9 15:06:47 m0054 systemd[1]: voxl-tflite-server.service: Main process exited, code=killed, status=11/SEGV Jan 9 15:06:47 m0054 systemd[1]: voxl-tflite-server.service: Failed with result 'signal'. Jan 9 15:06:48 m0054 systemd[1]: voxl-tflite-server.service: Service hold-off time over, scheduling restart. Jan 9 15:06:48 m0054 systemd[1]: voxl-tflite-server.service: Scheduled restart job, restart counter is at 16. Jan 9 15:06:48 m0054 systemd[1]: Stopped voxl-tflite-server. Jan 9 15:06:48 m0054 systemd[1]: Started voxl-tflite-server. Jan 9 15:06:48 m0054 bash[5774]: INFO: Created TensorFlow Lite delegate for GPU. here's my voxl-tflite-server.conf /** * This file contains configuration that's specific to voxl-tflite-server. * * skip_n_frames - how many frames to skip between processed frames. For 30Hz * input frame rate, we recommend skipping 5 frame resulting * in 5hz model output. For 30Hz/maximum output, set to 0. * model - which model to use. Currently support mobilenet, fastdepth, * posenet, deeplab, and yolov5. * input_pipe - which camera to use (tracking, hires, or stereo). * delegate - optional hardware acceleration: gpu, cpu, or nnapi. If * the selection is invalid for the current model/hardware, * will silently fall back to base cpu delegate. * allow_multiple - remove process handling and allow multiple instances * of voxl-tflite-server to run. Enables the ability * to run multiples models simultaneously. * output_pipe_prefix - if allow_multiple is set, create output pipes using default * names (tflite, tflite_data) with added prefix. * ONLY USED IF allow_multiple is set to true. * */ { "skip_n_frames": 0, "model": "/usr/bin/dnn/yolov5_float16_quant.tflite", "input_pipe": "/run/mpa/hires_color", "delegate": "gpu", "allow_multiple": false, "output_pipe_prefix": "mobilenet" } ## Reply by Guest · 2024-01-09 16:24:55 UTC (in reply to sansoy) @sansoy Thanks for an informative response. One thing that's confusing me is the message `"ERROR in pipe_client_init_channel"` as the `pipe_client_init_channel` method is deprecated. Do you mind letting me know what version of the SDK you're on? If it isn't the most recent SDK, it's probably worth upgrading to see if it fixes anything. I know we've put out a lot of changes in `libmodal-pipe`. You can read how to flash the latest SDK [here](https://docs.modalai.com/flash-system-image/). Unfortunately just from these debug messages I can't pin down the issue and so I might need you to provide me with a model file to help out more. I can understand if you don't want to leak your trained model file, though. One thing you could do in this case would be to just train for a single epoch just as a means of creating a model through the same process. If I have a model file I can do some more rigorous debugging to determine the issue. Thanks and sorry about all of this! Thomas Patton thomas.patton@modalai.com ## Reply by sansoy · 2024-01-09 20:07:10 UTC (in reply to Guest) @Thomas-Patton voxl2:/$ voxl-version -------------------------------------------------------------------------------- system-image: 1.6.2-M0054-14.1a-perf kernel: #1 SMP PREEMPT Fri May 19 22:19:33 UTC 2023 4.19.125 -------------------------------------------------------------------------------- hw version: M0054 -------------------------------------------------------------------------------- voxl-suite: 1.0.0 ----------------------------------------------------------------------------- will update to 1.0.1 Can I email you my tflite and saved model for review? I'm doing a run right now that should be completed in a couple hours. Sabri ## Reply by tom (ModalAI staff) · 2024-01-09 20:08:50 UTC (in reply to sansoy) @sansoy You should upgrade to the latest SDK (1.1.2) ## Reply by sansoy · 2024-01-09 22:04:30 UTC (in reply to tom) @tom so i downloaded the upgrade and started the upgrade but its been stuck for about an hour. How long does it take to flash the upgrade? Sabri Flashing the following System Image: Build Name: 1.7.1-M0054-14.1a-perf-nightly-20231025 Build Date: 2023-10-25 Platform: M0054 System Image Version: 1.7.1 Installing the following version of voxl-suite: voxl-suite Version: 1.1.2 Would you like to continue with SDK install? 1) Yes 2) No #? yes [ERROR] invalid option #? 1 [INFO] adb installed [INFO] fastboot installed ---- Starting System Image Flash ---- ----./flash-system-image.sh ---- Detected OS: Linux Installer Version: 0.8 Image Version: 1.7.1 Please power off your VOXL, connect via USB, then power on VOXL. We will keep searching for an ADB or Fastboot device over USB [INFO] Found ADB device [INFO] Rebooting to fastboot . [INFO] Found fastboot device [WARNING] This system image flash is intended only for the following platform: VOXL2 (m0054) Make sure that the device that will be flashed is correct. Flashing a device with an incorrect system image will lead the device to be stuck in fastboot. Would you like to continue with the VOXL2 (m0054) system image flash? 1) Yes 2) No #? 1 ## Reply by tom (ModalAI staff) · 2024-01-09 22:26:11 UTC (in reply to sansoy) @sansoy It should start right away, I would power cycle your voxl2 and try again ## Reply by sansoy · 2024-01-09 22:39:01 UTC (in reply to tom) @tom i did all that and still stuck. could it be whats in the warning about being stuck in fastboot? it is the voxl2 and not the voxl2 mini. [WARNING] This system image flash is intended only for the following platform: VOXL2 (m0054) Make sure that the device that will be flashed is correct. Flashing a device with an incorrect system image will lead the device to be stuck in fastboot. ## Reply by tom (ModalAI staff) · 2024-01-09 23:00:09 UTC (in reply to sansoy) @sansoy As long as you are using the voxl2 SDK and are indeed flashing voxl2 hardware then that warning can be ignored. ## Reply by sansoy · 2024-01-10 00:32:42 UTC (in reply to tom) @tom hey Tom, i'm having absolutely no luck. i've tried 3 times and it still just hangs at Would you like to continue with the VOXL2 (m0054) system image flash? 1) Yes 2) No #? 1 I then followed the unbrick instructions and reinstalled everything per https://docs.modalai.com/voxl2-unbricking/#ubuntu-host Got the system back up and running and tried to install the latest SDK again with no luck. It just hangs. ## Reply by sansoy · 2024-01-10 00:46:00 UTC (in reply to sansoy) UPDATE: Got it working with "sudo" for the install. normally one would get a permission errors and thought maybe that was the issue and sure enough. recommend updating your docs to say sudo ./install.sh ## Reply by sansoy · 2024-01-10 04:49:10 UTC (in reply to sansoy) @tom so i trained on a new batch of AR15 images and got really good numbers in terms of losses and mAPs. Ran an unquantized and quantized version in voxl-tflite-server and again nothing is being recognized. Here's a link to the tflites, and saved_models with inference results on never before seen images. Any insight on how to make these models work in your environment would be awesomely appreciated. https://drive.google.com/drive/folders/1N1pU0jMRTb3rODSfIuETPrBf66m4ody7?usp=drive_link ## Reply by tom (ModalAI staff) · 2024-01-10 17:54:30 UTC (in reply to sansoy) @sansoy Interesting, sudo isn't normally required. I'm curious, what linux distro are you running? ## Reply by tom (ModalAI staff) · 2024-01-10 17:55:40 UTC (in reply to sansoy) @sansoy @Thomas-Patton is the ML expert here and I'll let him comment on that front ## Reply by sansoy · 2024-01-10 18:09:51 UTC (in reply to tom) @tom Ubuntu 22.04.3 LTS ## Reply by tom (ModalAI staff) · 2024-01-10 18:21:47 UTC (in reply to sansoy) @sansoy Huh, okay, that's what I run as well. What groups are your default user in? For example, here is mine: ``` ~ groups ok | 10:20:36 AM tom adm dialout cdrom sudo dip plugdev lpadmin lxd sambashare docker ``` ## Reply by sansoy · 2024-01-10 22:17:20 UTC (in reply to tom) @tom eve@eve:~$ groups eve adm cdrom sudo dip plugdev lpadmin lxd sambashare ## Reply by tom (ModalAI staff) · 2024-01-11 17:41:42 UTC (in reply to sansoy) @sansoy Can you try adding your user to the dialout group and seeing if that fixes the issue? `sudo usermod -a -G dialout $USER` ## Reply by sansoy · 2024-01-12 14:36:26 UTC (in reply to tom) @tom did that and still no inference. ![tflite_1705069939.png](https://forum.modalai.com/assets/uploads/files/1705070183730-tflite_1705069939.png) ## Reply by tom (ModalAI staff) · 2024-01-12 18:03:40 UTC (in reply to sansoy) @sansoy That was for fixing the fastboot issue, unrelated ## Reply by Guest · 2024-01-12 18:38:35 UTC (in reply to tom) @sansoy When I get some time today I'll try to download your .tflite models and see what's going on. The good news is that the server is at least running! It very well may just be an issue with how the tensor is being parsed. Thomas thomas.patton@modalai.com ## Reply by Guest · 2024-01-15 17:18:59 UTC (in reply to Guest) @sansoy Hey, they just gave me access to the Google Drive folder. Can you confirm that the `edgetpu.tflite` in the root directory is the file you want me to try and get working? Thomas ## Reply by sansoy · 2024-01-15 17:56:55 UTC (in reply to Guest) @thomas i just checked permissions and you have access to all the files and yes to edgetpu.tflit ssd-mobilenet-v2-fpnlite-640_quant.tflite ssd-mobilenet-v2-fpnlite-640.tflite saved_models/saved_model.pb ## Reply by Guest · 2024-01-15 21:45:35 UTC (in reply to sansoy) @sansoy Of these 4 models which is the custom trained YOLO model you mentioned above? I can't even get the `edgetpu.tflite` model to load so I need a little bit more information on how each of these files was generated. Thomas ## Reply by sansoy · 2024-01-15 22:50:57 UTC (in reply to Guest) @thomas here's a yolov5 trained model. https://drive.google.com/file/d/1wRbIXdylgx-EOGDLnuWnsytGd2DcTNeY/view?usp=drive_link I used this instruction set to train a yolov5 model which works well on my mac, rPI4, linux box and nvidia jetson. https://docs.ultralytics.com/yolov5/tutorials/train_custom_data/#23-organize-directories ## Reply by sansoy · 2024-01-16 12:58:58 UTC (in reply to sansoy) @thomas here's the colab notebook i used to create the tflites and quantized tflites https://colab.research.google.com/drive/1QdgpSl63OSQdLTnFwOyP8dxLQ7W0HtmW?usp=sharing ## Reply by Guest · 2024-01-16 18:24:35 UTC (in reply to sansoy) @sansoy I found the bug in our code that's causing the issue. You can see on [this line](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp?ref_type=heads#L743) that a for YOLO models a former employee hardcoded the number of classes to be 80 leading to a segfault when we only have a single class like in your case. Here's what your model looks like on our server with the fix: ![7b4e7252-e3f7-4204-be81-abacde3b5c4b-image.png](https://forum.modalai.com/assets/uploads/files/1705428986964-7b4e7252-e3f7-4204-be81-abacde3b5c4b-image.png) What I'm going to do is write a patch for `voxl-tflite-server` that more intelligently reads the number from the labels file. This will be available in our [SDK Nightlies](https://console.cloud.google.com/storage/browser/platform-nightlies/voxl2;tab=objects?organizationId=517175400245&project=modalai-core-services&prefix=&forceOnObjectsSortingFiltering=true&pageState=(%22StorageObjectListTable%22:(%22s%22:%5B(%22i%22:%22objectListDisplayFields%2FtimeCreated%22,%22s%22:%221%22),(%22i%22:%22displayName%22,%22s%22:%220%22)%5D)) ) and so you'll be able to use the fix tomorrow morning. If you'd like the fix sooner, what I can do is package the new voxl-tflite-server into a .deb so you can deploy it manually to your VOXL. I can help you out with this process if you're not familiar. Thanks, Thomas Patton thomas.patton@modalai.com ## Reply by Guest · 2024-01-16 20:20:56 UTC (in reply to Guest) @thomas https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/merge_requests/22 Here's the merge request with the fix, this will be published both in tonight's nightly and new SDK releases. Thomas ## Reply by sansoy · 2024-01-17 00:57:32 UTC (in reply to Guest) @thomas you rock! awesome! thanks for troubleshooting. ## Reply by sansoy · 2024-01-17 01:05:13 UTC (in reply to Guest) @thomas thanks thomas. will download tomorrow and test either in evening or next day. Really appreciate your help with this. Great job! Sabri ## Reply by Guest · 2024-01-17 01:10:14 UTC (in reply to sansoy) @sansoy Yeah of course, glad I could be of assistance! Be sure to let me know if you need any more help with this. Two things I was thinking about on my drive home that you should know - First, when you're on that page I linked above for the software nightlies make sure you sort the files by date descending so that you get the most recent one. I always make the mistake of getting the wrong one. Secondly, when you flash that nightly SDK it will overwrite anything not in the `/data/` directory and so you will need to put your model file and labels txt back into `/usr/bin/dnn/`. Thomas ## Reply by sansoy · 2024-01-18 12:36:24 UTC (in reply to Guest) @thomas Morning Thomas, just getting back to this. I couldnt find any SDK that had a 2024 stamp. Was this uploaded somewhere else? Sabri ## Reply by Guest · 2024-01-18 15:41:26 UTC (in reply to sansoy) @sansoy Here's the one from last night: https://console.cloud.google.com/storage/browser/_details/platform-nightlies/voxl2/voxl2_SDK_nightly_20240117.tar.gz?pageState=(%22StorageObjectListTable%22:(%22s%22:%5B(%22i%22:%22objectListDisplayFields%2FtimeLastModified%22,%22s%22:%221%22),(%22i%22:%22displayName%22,%22s%22:%220%22)%5D))&organizationId=517175400245&project=modalai-core-services You just have to make sure when you're on [this](https://console.cloud.google.com/storage/browser/platform-nightlies/voxl2;tab=objects?organizationId=517175400245&project=modalai-core-services&prefix=&forceOnObjectsSortingFiltering=true&pageState=(%22StorageObjectListTable%22:(%22s%22:%5B(%22i%22:%22objectListDisplayFields%2FtimeLastModified%22,%22s%22:%221%22),(%22i%22:%22displayName%22,%22s%22:%220%22)%5D))) page that you sort date modified by descending. Thanks, Thomas ## Reply by sansoy · 2024-01-18 17:23:54 UTC (in reply to Guest) @thomas installed voxl2_SDK_nightly_20240117.tar.gz per https://docs.modalai.com/flash-system-image/ still not getting any detection even with your default yolov5. ive tried with hires large color, small color and grey. ![tflite_1705597671.png](https://forum.modalai.com/assets/uploads/files/1705598564342-tflite_1705597671.png) ## Reply by Guest · 2024-01-18 17:30:34 UTC (in reply to sansoy) @sansoy Checking this out right now, will let you know when I have something. Thomas ## Reply by Guest · 2024-01-18 18:53:08 UTC (in reply to Guest) @thomas @sansoy So I was able to get our YOLO model running immediately, am about to try yours next. All I did was: 1. Flash the nightly SDK from last night using the steps you linked above 2. Edited `/etc/modalai/voxl-tflite-server.conf` so that the model was `/usr/bin/dnn/yolov5_float16_quant.tflite` and the input pipe was `/run/mpa/hires_small_color`. You should definitely try to reproduce this result as if you can't it might indicate a deeper problem. Then to use your custom model: 1. `adb shell` and then `cd /usr/bin/dnn` and then `cp yolov5_labels.txt yolov5_labels.txt.bak && cp yolov5_float16_quant.tflite yolov5_float16_quant.tflite.bak` to back up the existing model and label files. 2. `vi yolov5_labels.txt` and change to only have `ar15` as a class 3. From outside ADB, `adb push yolo5_fp16_quant.tflite /usr/bin/dnn/yolov5_float16_quant.tflite` 4. `voxl-tflite-server` ![e2ee78d8-8d17-4863-ad90-8917442727b5-image.png](https://forum.modalai.com/assets/uploads/files/1705603954658-e2ee78d8-8d17-4863-ad90-8917442727b5-image.png) Thomas thomas.patton@modalai.com ## Reply by sansoy · 2024-01-18 19:28:43 UTC (in reply to Guest) @thomas WORKED! i did most of what you stated except for renaming my file to your filename. !![tflite_1705605919.png](https://forum.modalai.com/assets/uploads/files/1705606018428-tflite_1705605919.png) in a future release would be able to use the original filename vs renaming it to something that is hardcoded? ## Reply by Guest · 2024-01-18 19:56:25 UTC (in reply to sansoy) @sansoy Glad you got it working! It's important to note that `voxl-tflite-server` isn't intended to be a high-grade tool for running any ML model with great configurations and optimizations; it's just an example of how you can use the VOXL ecosystem to run models. As much as I would enjoy building such a tool and implementing your request (and the others I get), there just isn't enough of a business demand for it right now. That being said, the code is open source and right [here](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/main.cpp?ref_type=heads#L349) is the line that does the string comparison. If you want to put out a merge request to add a new field for `model_type` to the `/etc/modalai/voxl-tflite-server.conf` JSON and have it select an inference method based on that, you can tag me and I'll review it. Thanks, Thomas Patton thomas.patton@modalai.com ## Reply by sansoy · 2024-01-18 23:27:18 UTC (in reply to Guest) @thomas said in [Can anyone recommend a Tflite Colab Notebook for VOXL2 Training](/post/14066): > It's important to note that voxl-tflite-server isn't intended to be a high-grade tool for running any ML model with great configurations and optimizations; it's just an example of how you can use the VOXL ecosystem to run models. As much as I would enjoy building such a tool and implementing your request (and the others I get), there just isn't enough of a business demand for it right now. Thanks for the info. how can i get access to the box coordinates in the image? ## Reply by Guest · 2024-01-19 18:37:15 UTC (in reply to sansoy) @sansoy Check out https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/include/ai_detection.h?ref_type=heads , this defines the detection output. This is streamed out to a pipe in `/run/mpa` Thomas ## Reply by Darshit Desai · 2024-01-28 02:34:51 UTC (in reply to Guest) @thomas @sansoy Has anyone tried integrating a yolov8n model, I have a tflite fp16 quantized model but this line in the modal ai documentation says that any other model architecture apart from the ones mentioned here cannot be used, ![6991e349-2c7a-4562-9033-649a3f0a604f-Screenshot from 2024-01-27 21-34-41.png](https://forum.modalai.com/assets/uploads/files/1706409287890-6991e349-2c7a-4562-9033-649a3f0a604f-screenshot-from-2024-01-27-21-34-41.png) file:///home/swarm_researchers/Pictures/Screenshots/Screenshot%20from%202024-01-27%2021-34-41.png ## Reply by Guest · 2024-01-28 02:36:18 UTC (in reply to Darshit Desai) @Darshit-Desai @sansoy has been able to get custom models working, but you'll have to follow the steps we took in the posts above. Let me know how I can best help! Thomas ## Reply by Darshit Desai · 2024-01-28 03:02:29 UTC (in reply to Guest) @thomas Also the model which @sansoy used was yolov5n, and I checked most of the code of inference helper it was related to yolov5, is the tflite server compatible to any yolo model? Actually in a separate post I asked about object tracking with yolo models, from the article I read here and referring to the individual tracker documentations, somehow I need to have access to the tflite model to run these trackers, is there a way to integrate the trackers into the voxl-tflite-server, as the documentation says it provides hardware acceleration. https://www.datature.io/blog/implementing-object-tracking-for-computer-vision https://learnopencv.com/understanding-multiple-object-tracking-using-deepsort/#DeepSORT-Implementation ## Reply by Darshit Desai · 2024-01-28 05:22:33 UTC (in reply to Darshit Desai) @thomas Edit about upgrading the sdk to nightly build: I tried it on my starling v2 drone, but somehow the nighly build is unable to write px4 parameters and the upgrade ends up failing ([firmware issue link](https://forum.modalai.com/post/14349) I reverted the sdk back to sdk 1.0 and the drone fortunately works again, but how would I avoid the number of classes bug which you highlighted above. I also have modified yolo to detect just one class. ## Reply by Guest · 2024-01-28 06:11:10 UTC (in reply to Darshit Desai) @Darshit-Desai The goal of `voxl-tflite-server` is to provide an example of how ML models can be used in the VOXL ecosystem, not as an in-depth, well-tested framework. Your custom YOLO model might work, it might not. At lot of the logic in the server for math regarding the raw tensor data that the model produces has been written specifically for the models we put onboard. A good example of this is how the module postprocesses our YoloV5 network [here](https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server/-/blob/master/src/inference_helper.cpp#L849); if your output tensor is the same format then you'll probably get good results out. If not, then you won't. So I would follow the steps we took above to get a custom YOLO model working. If it doesn't work, you'll need to write your own C++ module that subscribes to an input camera pipe using `libmodal-pipe`, creates a `TFLite::Interpreter` that works with your model, and then parses the output tensor from your YOLO model. I'm happy to help you out with whatever you need here. No idea what's going on with the PX4 params, that's outside of my domain. However if you're having problems with TFLite server regardless, it might just be best to code up your own approach as outlined above. Keep me in the loop, happy to help you out however you need! Thomas Patton ## Reply by Darshit Desai · 2024-01-28 06:44:02 UTC (in reply to Guest) @thomas I used my custom dataset to retrain yolo v5 model and I get similar performance on my desktop. About the sdk is it possible to get the changes you did to the tflite server in a build closer to sdk 1.1.2 for the starling v2. I was able to upgrade and fly the drone till that version of the sdk 1.1.2 that means that whatever nightly releases happened after that might have same px4 related bugs. About the Multi object tracking, if I have to rewrite the same thing which deepsort or any other algorithm did that would entirely have to be in C++? Also if I want to run any ML/AI model without the tflite server, lets say in a python script (running on docker container onboard voxl2) it would not get any of the gpu hardware acceleration? I ask this questions because writing a sdk level software is much difficult. Instead if we had drivers like in our desktop pcs (CUDA like drivers) which when invoked in a python script could readily make the acceleration available. It would make ML deployment much easier. ## Reply by Guest · 2024-01-29 05:41:21 UTC (in reply to Darshit Desai) @Darshit-Desai Check your other post for details on getting the nightly builds. VOXL can run any language you install on it, Python or C++ or whatever else. However if you want to read camera data in in real-time, our supported solution is through `libmodal-pipe` which is written in C++. As for the hardware acceleration, it may be possible to use it in Python but we only have it documented for C++ via the `nnapi_delegate` class in `TFLite` which you can see [here](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/nnapi/nnapi_delegate.cc). There might be other methods for acceleration but this is all we've used and tested so far. That is to say that we don't currently have hardware accelerators built into a general purpose library like CUDA. I would love to write a library like that but we simply don't have the business demand for it right now. Hope this helps, Thomas Patton ## Reply by Darshit Desai · 2024-01-29 21:07:35 UTC (in reply to Guest) @thomas what is the purpose of the quantization script mentioned in the voxl docs, Is it the same as model conversion given in yolov5 documentation https://docs.modalai.com/voxl-tflite-server/#using-your-own-models Can I use the model conversion script given over here? https://docs.ultralytics.com/yolov5/tutorials/model_export/ ## Reply by Guest · 2024-01-29 21:38:29 UTC (in reply to Darshit Desai) @Darshit-Desai I **think** that the yolov5 one will work as I'm pretty sure that's what I used to make a custom one at some point. I think the most important part is that the quantization is FP16 so make sure you add that `--half` arg. Definitely try it out and let me know how it goes. Thomas ## Reply by Darshit Desai · 2024-01-29 21:42:38 UTC (in reply to Guest) @thomas Actually I did it without the --half argument and it works onboard the voxl, here's the result ![0350ddea-b659-4949-af36-5fbef65a270a-image.png](https://forum.modalai.com/assets/uploads/files/1706564628438-0350ddea-b659-4949-af36-5fbef65a270a-image.png) Also I don't know why but when I checked in netron the input image size was give 3,640,640 while voxl shows that the image resolution is 1024x768. I also read somewhere in the tflite documentation that the image size if changed needs to be updated in inference helper. I didn't do that too ## Reply by Guest · 2024-01-29 21:57:16 UTC (in reply to Darshit Desai) @Darshit-Desai Yeah, for reference almost all of the `voxl-tflite-server` code and documentation was written by a former employee so I don't always have the source of ground truth for things. I think there may be some logic in there that deals with reshaping the output tensors but I'm not fully sure. Same thing with the `--half` argument, I think we advise it but if you can get your model to work without quantization that's awesome. Glad to see it's working!! The results look good :) Keep me posted on how I can help, Thomas Patton