Upgrade Tensorflow Version tflite-server
-
@Matt-Turi thank you for the fast reply again. I will check the apm and Camera server output format tomorrow and also send you a picture of the setup.
-
@Matt-Turi I got it working with changing the apm. Thank you for the suggestion. So i looked up my Camera Supports and it should support nv12 and in voxl-uvc-server it shows the following:
uvc_get_stream_ctrl_format_size succeeded for format 2after looking at the voxl-uvc-server:
https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-uvc-server/-/blob/dev/src/main.c
L: 307 There is a for loop looking for a format to choose and format 2 should be NV12 but for some reason when looking at the stream over voxl-portal i get green stripes in the ouput. Do you know what the problem could be?The Camera is a Boson 640 Black-White Thermal Camera.
-
Here is a Picture of the green stripes:
-
@Philemon-Benner Can you post the output of
lsusb,voxl-uvc-server -s, andvoxl-uvc-server -d -m -r 640x512? -
@Eric-Katzfey
lsusb:Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 002: ID 09cb:4007 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hubvoxl-uvc-server -s:
*** START DEVICE LIST *** Found device 1 Got device descriptor for 09cb:4007 134275 Found device 09cb:4007 DEVICE CONFIGURATION (09cb:4007/134275) --- Status: idle VideoControl: bcdUVC: 0x0100 VideoStreaming(1): bEndpointAddress: 129 Formats: UncompressedFormat(1) bits per pixel: 12 GUID: 4934323000001000800000aa00389b71 (I420) default frame: 1 aspect ratio: 0x0 interlace flags: 00 copy protect: 00 FrameDescriptor(1) capabilities: 02 size: 640x512 bit rate: 10592000-235929600 max frame size: 491520 default interval: 1/60 interval[0]: 1/60 interval[1]: 1/30 UncompressedFormat(2) bits per pixel: 16 GUID: 5931362000001000800000aa00389b71 (Y16 ) default frame: 1 aspect ratio: 0x0 interlace flags: 00 copy protect: 00 FrameDescriptor(1) capabilities: 02 size: 640x512 bit rate: 10592000-629145600 max frame size: 655360 default interval: 1/60 interval[0]: 1/60 interval[1]: 1/30 UncompressedFormat(3) bits per pixel: 12 GUID: 4e56313200001000800000aa00389b71 (NV12) default frame: 1 aspect ratio: 0x0 interlace flags: 00 copy protect: 00 FrameDescriptor(1) capabilities: 02 size: 640x512 bit rate: 10592000-235929600 max frame size: 491520 default interval: 1/60 interval[0]: 1/60 interval[1]: 1/30 UncompressedFormat(4) bits per pixel: 12 GUID: 4e56323100001000800000aa00389b71 (NV21) default frame: 1 aspect ratio: 0x0 interlace flags: 00 copy protect: 00 FrameDescriptor(1) capabilities: 02 size: 640x512 bit rate: 10592000-235929600 max frame size: 491520 default interval: 1/60 interval[0]: 1/60 interval[1]: 1/30 END DEVICE CONFIGURATION *** END DEVICE LIST ***voxl-uvc-server -d -m -r 640x512:
Enabling debug messages Enabling MPA debug messages voxl-uvc-server starting Image resolution 640x512, 30 fps chosen UVC initialized Device found Device opened uvc_get_stream_ctrl_format_size succeeded for format 2 Streaming starting Got frame callback! frame_format = 17, width = 640, height = 512, length = 491520, ptr = (nil) making new fifo /run/mpa/uvc/voxl-tflite-server0 opened new pipe for writing after 3 attempt(s) write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 1 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 * got image 30 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 40 errno: 0 previous client state was 2 write to ch: 0 id: 0 result: 491520 errno: 0 -
Just checked the camera on my laptop. Camera it self is working fine
It's also working fine with nv12 format without the green lines:ffplay /dev/video4 -pixel_format nv12Output:
Input #0, video4linux2,v4l2, from '/dev/video4':B sq= 0B f=0/0 Duration: N/A, start: 33575.082223, bitrate: 235929 kb/s Stream #0:0: Video: rawvideo (NV12 / 0x3231564E), nv12, 640x512, 235929 kb/s, 60 fps, 60 tbr, 1000k tbn, 1000k tbc -
@Philemon-Benner It looks like voxl-uvc-server is working fine. The error messages seem to be related to voxl-tflite-server and not voxl-uvc-server. Can you use voxl-streamer to stream the output of voxl-uvc-server to VLC and see if there are still green lines? What are you currently using to stream the output of voxl-uvc-server?
-
@Philemon-Benner I'm guessing that there may be a version mismatch somewhere in there.
-
@Eric-Katzfey Yeah i will try that today. I'm currently showing the stream with voxl Portal but with the voxl-tflite-server pipe. So voxl-uvc-server --> voxl-tflite-server --> voxl-portal(green lines output).
-
@Eric-Katzfey what a exactly do you mean with version mismatch, and where. The thing i could think of is tflite server 0.2.0 as seen above in the first try of installation i had some errors because of existing dependencies.
-
@Eric-Katzfey Ok so i tried using it with voxl-streamer. It's working completely fine. Any suggestions?
-
I also tried changing the model but still the same result.
-
Update:
So i stepped back to TF1 and trained the ssdlite_mobilenet_v2. It's working great on the drone with tflite-server 0.1.8. Thanks for all the suggestions ·@Matt-Turi . But i am looking forward to using the tflite-server 0.2.0. If you still have suggestions for fixing the green stripes in the new version please let me know, because for custom models the new tflite-version is way easier to integrate and the code is more understandable for me. -
@Philemon-Benner Thanks for the follow up on this! We'll take a look.
-
@Matt-Turi is there a way to change small stuff in tflite-server code without building from source? Or is there a way to do it over ssh?. I can't access the usb slot from the voxl of the drone. I just wanna change the box color and confidence threshhold.
-
@Philemon-Benner If you make any changes to the code you will need to rebuild it from source. Once built, you can push this package over ssh either manually or using the deploy_to_voxl.sh script that is up on dev, which has an arg for ssh and send ip. Lines 106 and 107 of the script have the scp and opkg install commands that are used.
-
@Matt-Turi Thank you for the fast answer. Then i will build it from source.
But could be a cool feature in the future, if Box Color, Box Thickness and Confidence Threshhold would be in the config file. -
@Philemon-Benner great suggestion, I'll add that in soon. As for the green stripes issue seen with the dev version using a Flir Boson camera and voxl-uvc-server, I was able to successfully start and run tflite-server (mobilenetv2 w/gpu) with the same setup and only saw a few green "flickers" every few seconds due to the high input rate (60 fps) of the boson camera. I will work on some handling for this case, but have you tried your latest ssdlite_mobilenet_v2 model with the dev version of voxl-tflite-server (0.2.0)?
-
@Matt-Turi No i think because if you just update with opkg just the package is updatet and not the folders like the ones in /usr/bin/dnn where the models are stored. But yeah i will have a look at it. And also is the flickering happening on the in- or output? Because if it's happening in the input the inference results obviously will be less accurate, because of the green stripes.
-
@Philemon-Benner when you update with opkg, all included files will also be updated including the /usr/bin/dnn/ directory. In regards to v0.2.0, I pushed up a patch yesterday that should fix the flickering (was only on output). As long as the skip_n_frames parameter is set to at least 1 with the boson camera (since it comes in at a fixed 60fps), you should be good to go!
A note on inference with the Boson 640 Black-White Thermal Camera, I had some interesting results as the included mobilenet/most general models are not trained on thermal datasets, so keep that in mind when evaluating inference.
