Need help simulating .tflite yolo models on my linux machine.

Ajay Bidyarthy

@thomas
Okay, So i did run voxl-tflite-server -t alongside voxl-replay command but am still not getting anything.
For yolov5 I get no output after running voxl replay. it just outputs this:

For my custom yolov7 nd yolov8 I get these errors:

I even referred to this thread https://forum.modalai.com/topic/2997/can-anyone-recommend-a-tflite-colab-notebook-for-voxl2-training/3

and configured the main.cpp file by adding if else statement for custom model to use YOLO model instead of OBJECT_DETECTION as the default one before even running the above commands.

I'm not sure why am I still not getting any output.
Thanks

@Ajay-Bidyarthy I'll look into it today.

Thomas

@thomas @Ajay-Bidyarthy

Sadly I won't have time to look at this today and am out of office tomorrow so I can't fully debug this for you. Let's focus just on YOLOv5 right now since that's the one we have onboard by default and it isn't segfaulting. Make sure you're launching tflite-server in a terminal and then in a separate terminal running the voxl-replay command.

I apologize for this, sorry I didn't have time for it today.

Thomas

Ajay Bidyarthy

@thomas
No worries. Ok, but even for yolov5 I've been running both voxl replay and tflite server commands in separate windows/terminals but still facing the issue.

Let me know why is it so and how can we resolve this.
Thanks

@Ajay-Bidyarthy

Hey Ajay, working on it right now. Will let you know when I have an update.

Thomas

@thomas @Ajay-Bidyarthy

Okay so I've identified a problem in our timing code and am working on a fix. Here's a short-term solution for you:

Make sure /etc/modalai/voxl-tflite-server.conf has "input_pipe" set to "/run/mpa/tracking"
Get your VOXL's IP address with voxl-my-ip and enter this IP into your web browser to pull up voxl-portal in your browser.
Run voxl-tflite-server on VOXL
Run your voxl-logger log as voxl-logger -p /data/voxl-logger/log0009/ or whatever the path is saved as

Now in your web browser, from the "Cameras" dropdown click tflite. You may need to refresh the page to get this to pop up. The permalink to this page is <YOUR IP>/video.html?cam=tflite (e.g. http://192.168.0.198/video.html?cam=tflite) so you can optionally just go there. This overlay shows the ML model running and in the upper left of the frame shows a framerate. You can use this to estimate for the time being.

Am working on the fix now, will let you know when it's released.

Thomas

@thomas @Ajay-Bidyarthy

Okay, I've pushed up a change for our nightly builds to fix the timing printouts with the -t option. So starting tomorrow you can flash the latest nightly SDK build and have this fix. If you'd like it today let me know and I can push the .deb up to Google Drive so you can flash it directly. Here is the link for the nightly SDK builds.

Thomas

Ajay Bidyarthy

@thomas
Hi,
Thanks for this info. So I have tried accessing the portal by running the voxl-my-ip command but found nothing getting printed out when I run that command. I inspected to make sure my voxl-portal service is running perfectly but still cant seem to get the ip for portal.
Also, can you push the latest SDK as well or share the deb with me.
file:///home/bhavya/Downloads/WhatsApp%20Image%202024-04-09%20at%2010.46.27%20AM.jpeg

Thanks

@Ajay-Bidyarthy

Make sure you set up WiFi on VOXL! Here's the guide: https://docs.modalai.com/voxl-2-wifi-setup/

Thomas

Ajay Bidyarthy

@thomas
Hi thomas,
Could you please provide us the latest nightly sdk build for VOXL2 mini if you think we can work with that for now?

Regards,
OKSI Team

@Ajay-Bidyarthy

https://console.cloud.google.com/storage/browser/_details/platform-nightlies/voxl2/voxl2_SDK_nightly_20240411.tar.gz;tab=live_object?organizationId=517175400245&project=modalai-core-services

Ajay Bidyarthy

@thomas
Hi thomas,
Is this package for voxl2min or voxl2?
Regards

Ajay Bidyarthy

@thomas
Because I think this SDK is meant for M-0054 while our version is M0104. Let me know if we can go ahead and flash this SDK or not?
Thanks

Ajay Bidyarthy

@thomas
Also, Now that I have flashed the SDK onto my chip its showing voxl2 in the shell screen instead of voxl2-mini. And even if I try and go ahead and do the following steps. I still cant get the desired output.
2ab88265-965c-4de3-a187-2ecc2d5f3cce-Screenshot from 2024-04-15 14-29-02.png
file:///home/bhavya/Pictures/Screenshots/Screenshot%20from%202024-04-15%2014-29-02.png

@Ajay-Bidyarthy

Oh my apologies, here's the link for VOXL2 Mini nightly SDK. I almost never work with these so apologies for the confusion

https://console.cloud.google.com/storage/browser/_details/platform-nightlies/voxl2-mini/voxl2-mini_SDK_nightly_20240412.tar.gz;tab=live_object?organizationId=517175400245&project=modalai-core-services

Thomas

Ajay Bidyarthy

@thomas
Hey thomas,
So I tried with this new sdk. Unfortunately, I still dont see any output here as well

Have you also tried using this method? Or were there issues getting WiFi setup?

@thomas said in Need help simulating .tflite yolo models on my linux machine.:

Okay so I've identified a problem in our timing code and am working on a fix. Here's a short-term solution for you:
Make sure /etc/modalai/voxl-tflite-server.conf has "input_pipe" set to "/run/mpa/tracking"
Get your VOXL's IP address with voxl-my-ip and enter this IP into your web browser to pull up voxl-portal in your browser.
Run voxl-tflite-server on VOXL
Run your voxl-logger log as voxl-logger -p /data/voxl-logger/log0009/ or whatever the path is saved as
Now in your web browser, from the "Cameras" dropdown click tflite. You may need to refresh the page to get this to pop up. The permalink to this page is <YOUR IP>/video.html?cam=tflite (e.g. http://192.168.0.198/video.html?cam=tflite) so you can optionally just go there. This overlay shows the ML model running and in the upper left of the frame shows a framerate. You can use this to estimate for the time being.

Am working on the fix now, will let you know when it's released.

Thomas

Yeah I'm really not sure what's going on here, on that nightly SDK voxl-tflite-server -t is printing out timing information. I don't have a VOXL2 Mini to test on so if it's a specific issue there I'm really not sure. Let me try some stuff today and I'll get back to you.

Thomas

Ajay Bidyarthy

@thomas
Hi thomas,
Yes so right now we tried figuring out if maybe the issue is with the model not being specific to voxl2mini or maybe the logs are generated based on voxl2. But I dont think thats the case.
The thing which I'm concerned about is:

How do we identify or pinpoint the problem ...like are we supposed to see the output after the 'Disconnected from camera server' line or before? And if we are supposed to see it and are not able to... then where exactly is the problem happening in the src/main.cpp file of voxl-tflite-server. Can we debug that somehow?
Also did you check running the voxl-replay logger command directly without having any camera setup or camera-server configured.... similar to the way we are currently running on our end.

@Ajay-Bidyarthy

If voxl-tflite-server was running into an actual error you'd see some sort of an error message printout. We typically debug issues through voxl-portal but if you aren't able to get WiFi set up you won't be able to do so. I again reflashed a nightly SDK to confirm things are working on my end and they are. To reconfirm:

Flash the latest SDK nightly from the link shared above
Modify /etc/modalai/voxl-tflite-server.conf to have /run/mpa/tracking as its input. This file gets overwritten on flash so make sure you do this.
voxl-tflite-server -t in one terminal
voxl-replay -p /data/voxl-logger/log0009/ in another

I can tell you that replay doesn't care about camera server being configured, it should work regardless.

-Thomas

Ajay Bidyarthy

@thomas
Hi thomas,
So I followed just the 4 steps you shared and its working perfectly fine with the default models.
9cde7d30-94c5-4b41-9fff-179560d17050-Screenshot from 2024-04-17 12-28-53.png
But when i deploy my custom voxl-tflite-server deb package onto the chip it says warning downgrading from voxl-tflite-server 0.3.3 to 0.3.2. So maybe I feel the issue is with the deb package of voxl-tflite-server I'm building with my custom tflite models and building the package out of it and deploying.
Maybe you can update the gitlab repo so that I can build my custom voxl-tflite-server deb package and deploy it.
Let me know if you can do this.

Thanks
file:///home/bhavya/Pictures/Screenshots/Screenshot%20from%202024-04-17%2012-28-53.png