Training custom yolov8 model
-
I am following the tutorial below to train a custom yolov8 model with 295 photos of a fuel canister. I'm training the model on a GCP instance with a NVIDIA Tesla P4. Instead of using the full COCO database, I'm only using 0=background and 1=fuel_canister. I am getting flooded with 'should not reach here' error logs when trying to run voxl-tflite-server. I have tried both configs below. Any ideas?
https://gitlab.com/voxl-public/support/voxl-train-yolov8.git
voxl2:~$ journalctl -u voxl-tflite-server -n 20 --no-pager -- Logs begin at Thu 2023-03-02 12:58:02 UTC, end at Tue 2025-05-13 00:18:20 UTC. -- May 13 00:18:15 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Current pipeline throughput: 4.64115 frames per second1) docker build -t yolov8-train:latest . 2) docker run --gpus all -it --shm-size=8g -v "$(pwd)":/app/ yolov8-train:latest /bin/bash 3) /app python3 train.py 4) /app yolo task=detect mode=val model=/app/runs/detect/fuel_canister_exp12/weights/best.pt data=/app/dataset.yaml split=test name=fuel_canister_test_eval 5) /app yolo export model=./runs/detect/fuel_canister_exp12/weights/best.pt format=tflite imgsz=640 nms=False half=Truecat fuel_canister_labels.txt 0 background 1 fuel_canistercat /etc/modalai/voxl-tflite-server /** * voxl-tflite-server Configuration File * * skip_n_frames - how many frames to skip between processed frames. For 30 Hz * input, skip 5 ^g^r 5 Hz inference. Set 0 for full rate. * model - which model to use. Bundled choices include mobilenet, * fastdepth, posenet, deeplab, yolov5, yolov8. * input_pipe - which camera pipe to read (tracking, hires, stereo, etc.). * delegate - hardware acceleration: gpu, cpu, or nnapi. "gpu" is best * on VOXL 2 for float16 models. * allow_multiple - if true, removes single-instance lock so multiple servers * can run (one per config file). * output_pipe_prefix - prefix added to the default output pipes when * allow_multiple is true. */ { "skip_n_frames": 0, "model": "/etc/modalai/tflite_models/fuel_canister_yolov8_custom.tflite", "input_pipe": "/run/mpa/hires_front_small_color", "delegate": "cpu", "requires_labels": true, "labels": "/etc/modalai/tflite_models/fuel_canister_labels.txt", "allow_multiple": false, "output_pipe_prefix": "fuel_canister", "output_meta_type": "YOLO_V8", "debug_en": false, "confidence_threshold": 0.1 }/** * voxl-tflite-server Configuration File * * skip_n_frames - how many frames to skip between processed frames. For 30 Hz * input, skip 5 ⇒ 5 Hz inference. Set 0 for full rate. * model - which model to use. Bundled choices include mobilenet, * fastdepth, posenet, deeplab, yolov5, yolov8. * input_pipe - which camera pipe to read (tracking, hires, stereo, etc.). * delegate - hardware acceleration: gpu, cpu, or nnapi. "gpu" is best * on VOXL 2 for float16 models. * allow_multiple - if true, removes single-instance lock so multiple servers * can run (one per config file). * output_pipe_prefix - prefix added to the default output pipes when * allow_multiple is true. */ { "skip_n_frames": 0, "model": "/etc/modalai/tflite_models/fuel_canister_yolov8_custom.tflite", "input_pipe": "/run/mpa/hires_front_small_color", "delegate": "cpu", "requires_labels": true, "labels": "/etc/modalai/tflite_models/fuel_canister_labels.txt", "allow_multiple": false, "output_pipe_prefix": "fuel_canister", "confidence_threshold": 0.5 }voxl2:~$ voxl-version ──────────────────────────────────────────────────────────────────────────────── system-image: 1.8.02-M0054-14.1a-perf kernel: #1 SMP PREEMPT Mon Nov 11 22:47:44 UTC 2024 4.19.125 ──────────────────────────────────────────────────────────────────────────────── hw platform: M0054 mach.var: 1.0.1 ──────────────────────────────────────────────────────────────────────────────── voxl-suite: 1.4.3 ──────────────────────────────────────────────────────────────────────────────── Packages: Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.4/binary-arm64/ Last Updated: 2025-04-17 20:21:35 List: kernel-module-voxl-fsync-mod-4.19.125 1.0-r0 kernel-module-voxl-gpio-mod-4.19.125 1.0-r0 kernel-module-voxl-platform-mod-4.19.125 1.0-r0 libfc-sensor 1.0.7 libmodal-cv 0.5.16 libmodal-exposure 0.1.3 libmodal-journal 0.2.3 libmodal-json 0.4.3 libmodal-pipe 2.10.6 libqrb5165-io 0.4.9 libvoxl-cci-direct 0.2.5 libvoxl-cutils 0.1.1 modalai-slpi 1.1.19 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-dfs-server 0.2.0 qrb5165-imu-server 1.1.3 qrb5165-rangefinder-server 0.1.5 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.3.5 qrb5165-tflite 2.8.0-2 voxl-bind-spektrum 0.1.1 voxl-camera-calibration 0.5.9 voxl-camera-server 2.1.2 voxl-ceres-solver 2:1.14.0-10 voxl-configurator 1.0.0 voxl-cpu-monitor 0.5.3 voxl-cross-template 0.0.1 voxl-docker-support 1.3.1 voxl-elrs 0.4.2 voxl-esc 1.5.1 voxl-feature-tracker 0.5.2 voxl-flow-server 0.3.6 voxl-fsync-mod 1.0-r0 voxl-gphoto2-server 0.0.10 voxl-gpio-mod 1.0-r0 voxl-io-server 0.0.5 voxl-jpeg-turbo 2.1.3-5 voxl-lepton-server 1.3.3 voxl-lepton-tracker 0.0.4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.5.0 voxl-mavcam-manager 0.5.8 voxl-mavlink 0.1.4 voxl-mavlink-server 1.4.5 voxl-modem 1.1.5 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.9 voxl-mpa-tools 1.3.7 voxl-open-vins 0.4.17 voxl-open-vins-server 0.3.0 voxl-opencv 4.5.5-2 voxl-osd 0.1.3 voxl-platform-mod 1.0-r0 voxl-portal 0.7.9 voxl-px4 1.14.0-2.0.98 voxl-px4-imu-server 0.1.2 voxl-px4-params 0.6.7 voxl-qvio-server 1.1.1 voxl-remote-id 0.0.9 voxl-reset-slpi 0.0.1 voxl-state-estimator 0.0.4 voxl-streamer 0.7.5 voxl-suite 1.4.3 voxl-tag-detector 0.0.4 voxl-tflite-server 0.3.9 voxl-utils 1.4.6 voxl-uvc-server 0.1.7 voxl-vision-hub 1.8.20 voxl-vtx 1.2.2 voxl2-io 0.0.3 voxl2-system-image 1.8.02-r0 voxl2-wlan 1.0-r0 -
I am following the tutorial below to train a custom yolov8 model with 295 photos of a fuel canister. I'm training the model on a GCP instance with a NVIDIA Tesla P4. Instead of using the full COCO database, I'm only using 0=background and 1=fuel_canister. I am getting flooded with 'should not reach here' error logs when trying to run voxl-tflite-server. I have tried both configs below. Any ideas?
https://gitlab.com/voxl-public/support/voxl-train-yolov8.git
voxl2:~$ journalctl -u voxl-tflite-server -n 20 --no-pager -- Logs begin at Thu 2023-03-02 12:58:02 UTC, end at Tue 2025-05-13 00:18:20 UTC. -- May 13 00:18:15 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Error in TensorData<float>: should not reach here May 13 00:18:16 m0054 bash[10642]: Current pipeline throughput: 4.64115 frames per second1) docker build -t yolov8-train:latest . 2) docker run --gpus all -it --shm-size=8g -v "$(pwd)":/app/ yolov8-train:latest /bin/bash 3) /app python3 train.py 4) /app yolo task=detect mode=val model=/app/runs/detect/fuel_canister_exp12/weights/best.pt data=/app/dataset.yaml split=test name=fuel_canister_test_eval 5) /app yolo export model=./runs/detect/fuel_canister_exp12/weights/best.pt format=tflite imgsz=640 nms=False half=Truecat fuel_canister_labels.txt 0 background 1 fuel_canistercat /etc/modalai/voxl-tflite-server /** * voxl-tflite-server Configuration File * * skip_n_frames - how many frames to skip between processed frames. For 30 Hz * input, skip 5 ^g^r 5 Hz inference. Set 0 for full rate. * model - which model to use. Bundled choices include mobilenet, * fastdepth, posenet, deeplab, yolov5, yolov8. * input_pipe - which camera pipe to read (tracking, hires, stereo, etc.). * delegate - hardware acceleration: gpu, cpu, or nnapi. "gpu" is best * on VOXL 2 for float16 models. * allow_multiple - if true, removes single-instance lock so multiple servers * can run (one per config file). * output_pipe_prefix - prefix added to the default output pipes when * allow_multiple is true. */ { "skip_n_frames": 0, "model": "/etc/modalai/tflite_models/fuel_canister_yolov8_custom.tflite", "input_pipe": "/run/mpa/hires_front_small_color", "delegate": "cpu", "requires_labels": true, "labels": "/etc/modalai/tflite_models/fuel_canister_labels.txt", "allow_multiple": false, "output_pipe_prefix": "fuel_canister", "output_meta_type": "YOLO_V8", "debug_en": false, "confidence_threshold": 0.1 }/** * voxl-tflite-server Configuration File * * skip_n_frames - how many frames to skip between processed frames. For 30 Hz * input, skip 5 ⇒ 5 Hz inference. Set 0 for full rate. * model - which model to use. Bundled choices include mobilenet, * fastdepth, posenet, deeplab, yolov5, yolov8. * input_pipe - which camera pipe to read (tracking, hires, stereo, etc.). * delegate - hardware acceleration: gpu, cpu, or nnapi. "gpu" is best * on VOXL 2 for float16 models. * allow_multiple - if true, removes single-instance lock so multiple servers * can run (one per config file). * output_pipe_prefix - prefix added to the default output pipes when * allow_multiple is true. */ { "skip_n_frames": 0, "model": "/etc/modalai/tflite_models/fuel_canister_yolov8_custom.tflite", "input_pipe": "/run/mpa/hires_front_small_color", "delegate": "cpu", "requires_labels": true, "labels": "/etc/modalai/tflite_models/fuel_canister_labels.txt", "allow_multiple": false, "output_pipe_prefix": "fuel_canister", "confidence_threshold": 0.5 }voxl2:~$ voxl-version ──────────────────────────────────────────────────────────────────────────────── system-image: 1.8.02-M0054-14.1a-perf kernel: #1 SMP PREEMPT Mon Nov 11 22:47:44 UTC 2024 4.19.125 ──────────────────────────────────────────────────────────────────────────────── hw platform: M0054 mach.var: 1.0.1 ──────────────────────────────────────────────────────────────────────────────── voxl-suite: 1.4.3 ──────────────────────────────────────────────────────────────────────────────── Packages: Repo: http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.4/binary-arm64/ Last Updated: 2025-04-17 20:21:35 List: kernel-module-voxl-fsync-mod-4.19.125 1.0-r0 kernel-module-voxl-gpio-mod-4.19.125 1.0-r0 kernel-module-voxl-platform-mod-4.19.125 1.0-r0 libfc-sensor 1.0.7 libmodal-cv 0.5.16 libmodal-exposure 0.1.3 libmodal-journal 0.2.3 libmodal-json 0.4.3 libmodal-pipe 2.10.6 libqrb5165-io 0.4.9 libvoxl-cci-direct 0.2.5 libvoxl-cutils 0.1.1 modalai-slpi 1.1.19 mv-voxl 0.1-r0 qrb5165-bind 0.1-r0 qrb5165-dfs-server 0.2.0 qrb5165-imu-server 1.1.3 qrb5165-rangefinder-server 0.1.5 qrb5165-slpi-test-sig 01-r0 qrb5165-system-tweaks 0.3.5 qrb5165-tflite 2.8.0-2 voxl-bind-spektrum 0.1.1 voxl-camera-calibration 0.5.9 voxl-camera-server 2.1.2 voxl-ceres-solver 2:1.14.0-10 voxl-configurator 1.0.0 voxl-cpu-monitor 0.5.3 voxl-cross-template 0.0.1 voxl-docker-support 1.3.1 voxl-elrs 0.4.2 voxl-esc 1.5.1 voxl-feature-tracker 0.5.2 voxl-flow-server 0.3.6 voxl-fsync-mod 1.0-r0 voxl-gphoto2-server 0.0.10 voxl-gpio-mod 1.0-r0 voxl-io-server 0.0.5 voxl-jpeg-turbo 2.1.3-5 voxl-lepton-server 1.3.3 voxl-lepton-tracker 0.0.4 voxl-libgphoto2 0.0.4 voxl-libuvc 1.0.7 voxl-logger 0.5.0 voxl-mavcam-manager 0.5.8 voxl-mavlink 0.1.4 voxl-mavlink-server 1.4.5 voxl-modem 1.1.5 voxl-mongoose 7.7.0-1 voxl-mpa-to-ros 0.3.9 voxl-mpa-tools 1.3.7 voxl-open-vins 0.4.17 voxl-open-vins-server 0.3.0 voxl-opencv 4.5.5-2 voxl-osd 0.1.3 voxl-platform-mod 1.0-r0 voxl-portal 0.7.9 voxl-px4 1.14.0-2.0.98 voxl-px4-imu-server 0.1.2 voxl-px4-params 0.6.7 voxl-qvio-server 1.1.1 voxl-remote-id 0.0.9 voxl-reset-slpi 0.0.1 voxl-state-estimator 0.0.4 voxl-streamer 0.7.5 voxl-suite 1.4.3 voxl-tag-detector 0.0.4 voxl-tflite-server 0.3.9 voxl-utils 1.4.6 voxl-uvc-server 0.1.7 voxl-vision-hub 1.8.20 voxl-vtx 1.2.2 voxl2-io 0.0.3 voxl2-system-image 1.8.02-r0 voxl2-wlan 1.0-r0@KnightHawk06 ok, I think I found my problem, yolov8 doesn't support exporting with nms. Can a yolov5 model be exported and used on the voxl2?
Reading along? Create a free ModalAI Forum account to join in.
With an account you can reply, ask your own question, get an email when a ModalAI engineer answers, and mark the reply that solved it. Your place in each thread is saved between visits.
Questions about VOXL, Flight Core, ESCs and ModalAI drones are answered here by the engineers who build them.
Register Login