# Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d Source: https://forum.modalai.com/topic/3579/sentinel-segmentation-fault-after-running-voxl-uvc-server-v-camera-id-d Category: Development Drones (https://forum.modalai.com/category/44/development-drones) Tags: sentinel Posted: 2024-06-24 14:19:01 UTC by Róbert Dévényi Replies: 29 · Views: 8515 ## Róbert Dévényi · 2024-06-24 14:19:01 UTC I have bought the SIYI A8 mini gimbal camera and Sentinel perfectly detects it and I was able to stream video from it. Then, the next day something happened and since then I have been receiving this error: (the camera id is 534d, this is for certain) voxl2:~$ voxl-uvc-server -v 534d -d loading config file Enabling debug messages ================================================================= width: 640 height: 480 fps: 30 pipe_name: uvc ================================================================= voxl-uvc-server starting Image resolution 640x480, 30 fps chosen Vendor ID 0x534d chosen UVC initialized Device 534d:0000 found Device opened uvc_get_stream_ctrl_format_size succeeded for format YUYV Streaming starting Segmentation fault: Fault thread: voxl-uvc-server(tid: 17803) Fault address: 0x7f9d661000 Access to this address is not allowed. Segmentation fault So it seems that the camera is recognized perfectly fine, streaming is started but then a segmentation fault occurs. I would appreciate any help in this matter. I tried restarting the drone, the camera, killing the camera processing etc. hundreds of times, it seems that restarting / rebooting does not work and this is some persistent problem. Thanks for any help in advance. ## Reply by Róbert Dévényi · 2024-06-24 14:22:17 UTC @Róbert-Dévényi the SDK version is 1.2.0 and voxl-uvc-server version is 0.1.6. ## Reply by Eric Katzfey (ModalAI staff) · 2024-06-24 15:12:39 UTC @Róbert-Dévényi That isn't a camera that we have available to try unfortunately. In our experience bringing up a new UVC camera usually involves a bit of work. Perhaps try a lower frame rate and see if that changes anything. Another idea is to stop all other services and just run voxl-uvc-server. Does it still crash? ## Reply by Róbert Dévényi · 2024-06-24 15:27:36 UTC (in reply to Eric Katzfey) @Eric-Katzfey when I run voxl-uvc-server -f 15, 20, ... all lower frame rates give me the same seg fault. How can I stop other services from running? ## Reply by Eric Katzfey (ModalAI staff) · 2024-06-24 15:30:59 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi You can use the `voxl-inspect-services` utility to see what services have been enabled. For each of those services you can either stop them or disable them. If you just stop them they will still start automatically on next reboot. If you disable them then you will have to re-enable them once you are done debugging. Stopping a service: `systemctl stop `, disabling a service `systemctl disable `. ## Reply by Róbert Dévényi · 2024-06-24 15:36:56 UTC (in reply to Eric Katzfey) @Eric-Katzfey Thanks. Which services are the most likely to cause a segfault? ## Reply by Róbert Dévényi · 2024-06-24 15:41:34 UTC (in reply to Eric Katzfey) @Eric-Katzfey it seems that for some reason the voxl-uvc-server service just stops running by itself even after I run systemctl start voxl-uvc-server or systemctl enable voxl-uvc-server. Probably that causes the segfault, that the service is not running. ## Reply by Eric Katzfey (ModalAI staff) · 2024-06-24 15:48:04 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi Nothing should be causing a segfault. I'm just offering suggestions on how to debug what is going on. ## Reply by Eric Katzfey (ModalAI staff) · 2024-06-24 15:49:06 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi The service just runs the application. ## Reply by Róbert Dévényi · 2024-06-24 15:51:38 UTC (in reply to Eric Katzfey) @Eric-Katzfey so I stopped all services except for voxl-uvc-server: Scanning services... Service Name | Enabled | Running | CPU Usage ------------------------------------------------------------------- docker-autorun | Disabled | Not Running | modallink-relink | Disabled | Not Running | voxl-camera-server | Enabled | Not Running | voxl-cpu-monitor | Enabled | Not Running | voxl-dfs-server | Disabled | Not Running | voxl-feature-tracker | Disabled | Not Running | voxl-flow-server | Disabled | Not Running | voxl-imu-server | Enabled | Not Running | voxl-lepton-server | Disabled | Not Running | voxl-mavcam-manager | Enabled | Not Running | voxl-mavlink-server | Enabled | Not Running | voxl-modem | Enabled | Not Running | voxl-neopixel-manager | Disabled | Not Running | voxl-open-vins-server | Disabled | Not Running | voxl-portal | Enabled | Not Running | voxl-px4-imu-server | Disabled | Not Running | voxl-px4 | Enabled | Not Running | voxl-qvio-server | Enabled | Not Running | voxl-rangefinder-server | Disabled | Not Running | voxl-remote-id | Disabled | Not Running | voxl-softap | Disabled | Not Running | voxl-static-ip | Disabled | Not Running | voxl-streamer | Enabled | Not Running | voxl-tag-detector | Disabled | Not Running | voxl-tflite-server | Disabled | Not Running | voxl-time-sync | Disabled | Not Running | voxl-uvc-server | Enabled | Running | 0.0% voxl-vision-hub | Enabled | Not Running | voxl-wait-for-fs | Enabled | Completed | And I am still getting the exact same segfault. ## Reply by Róbert Dévényi · 2024-06-24 15:52:15 UTC (in reply to Róbert Dévényi) loading config file Enabling debug messages ================================================================= width: 640 height: 480 fps: 30 pipe_name: uvc ================================================================= voxl-uvc-server starting Image resolution 640x480, 30 fps chosen Vendor ID 0x534d chosen UVC initialized Device 534d:0000 found Device opened uvc_get_stream_ctrl_format_size succeeded for format YUYV Streaming starting Segmentation fault: Fault thread: voxl-uvc-server(tid: 16852) Fault address: 0x7f7dda3000 Access to this address is not allowed. Segmentation fault ## Reply by Róbert Dévényi · 2024-06-24 15:55:36 UTC (in reply to Eric Katzfey) @Eric-Katzfey I don't know if it helps but whenever I run voxl-uvc-server, the segfault happens and after that the voxl-uvc-server service stops running, but is still enabled. ## Reply by Eric Katzfey (ModalAI staff) · 2024-06-24 16:01:38 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi The service just runs the application. If the application stops then the service stops. Are you familiar with Linux and systemd? It's just a standard Linux startup mechanism. But you want to stop all services, even for voxl-uvc-server, then try to start voxl-uvc-server on the command line and see if it still segfaults. There are ways to debug segfaults. But you would need to understand how to get Linux to generate a core file after the segfault and how to load the core into gdb. ## Reply by Róbert Dévényi · 2024-06-24 16:05:06 UTC (in reply to Eric Katzfey) @Eric-Katzfey I would appreciate it if you could tell me what commands to use to stop all the services and then restart voxl-uvc-server or any resource that helps with debugging segfaults in general. ## Reply by Róbert Dévényi · 2024-06-24 17:12:37 UTC (in reply to Eric Katzfey) @Eric-Katzfey I reset all mpa services to factory settings and I keep getting the same segfault. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-06-25 03:36:54 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi , You can stop the systemd service for `voxl-uvc-server` using the following command: ```systemctl stop voxl-uvc-server``` this will stop the service / server but will not prevent it from auto-starting during next reboot Then you can run the `voxl-uvc-server` in foreground. Just run it with the normal arguments: ``` voxl-uvc-server ``` Confirm that this still crashes.. then run the same executable in gdb: ``` > gdb /usr/bin/voxl-uvc-server (gdb) r ... wait for crash... ... bt ``` `bt` command in gdb will print the backtrace of the call stack and will tell you which function the seg fault happened in (and you will see the call stack). Maybe this will help with debugging. Alex ## Reply by Róbert Dévényi · 2024-06-25 15:09:09 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev Thank you very much hopefully I can figure it out. ## Reply by Róbert Dévényi · 2024-06-25 15:10:55 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev I can probably show you the backtrace log so that you can take a look at what causes the segfault. ## Reply by AndriiHlyvko · 2024-08-27 18:08:39 UTC (in reply to Róbert Dévényi) @Róbert-Dévényi @Eric-Katzfey @Alex-Kushleyev I am getting the same seg fault with a boson+ camera. I determined that the seg fault occurs when the uvc_start_streaming function is called (I am running a modified version of voxl-uvc-server). I have the stack trace from gdb. If I figure out the cause of the problem I will post it here. ``` (gdb) run Starting program: /usr/bin/voxl-uvc-server [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1". VOXL UVC Server loading config file VOXL UVC Server parsed argument voxl-uvc-server starting Image resolution 640x512, 30 fps chosen [New Thread 0x7ff7a7e1c0 (LWP 2189)] UVC initialized Device found [New Thread 0x7ff727d1c0 (LWP 2190)] Device opened Found desired frame format: 1 uvc_get_stream_ctrl_format_size succeeded for format NV12 bmHint: 0001 bFormatIndex: 2 bFrameIndex: 1 dwFrameInterval: 333333 wKeyFrameRate: 1 wPFrameRate: 0 wCompQuality: 0 wCompWindowSize: 0 wDelay: 0 dwMaxVideoFrameSize: 491520 dwMaxPayloadTransferSize: 983044 bInterfaceNumber: 1 [New Thread 0x7ff6a7c1c0 (LWP 2191)] [New Thread 0x7ff627b1c0 (LWP 2192)] [New Thread 0x7ff5a7a1c0 (LWP 2193)] Mavlink Onboard server Connected [New Thread 0x7fe70ef1c0 (LWP 2194)] Streaming starting [New Thread 0x7feffff1c0 (LWP 2195)] Thread 3 "voxl-uvc-server" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ff727d1c0 (LWP 2190)] __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185 185 ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory. (gdb) bt #0 __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185 #1 0x0000007ff7ed32a0 in _uvc_process_payload () from /usr/lib/libuvc.so.0 Backtrace stopped: previous frame identical to this frame (corrupt stack?) ``` ## Reply by Alex Kushleyev (ModalAI staff) · 2024-08-28 06:06:38 UTC (in reply to AndriiHlyvko) @AndriiHlyvko , we have gotten another report about this recently. It seem there is a bug in libuvc which has been fixed, but we are using older version of the library. Inserting the following check at the line below has been reported to fix the issue: ``` if (strmh->got_bytes + data_len > strmh->cur_ctrl.dwMaxVideoFrameSize) { UVC_DEBUG("Transfer too large and will overflow the buffer. Dropping data!"); _uvc_swap_buffers(strmh); return; } ``` here: https://github.com/modalai/libuvc/blob/3fb2d2fd31f215f76bd5d6782785385fd0927ce5/src/stream.c#L757 It seems the fix in the official libuvc release is here: https://github.com/libuvc/libuvc/blob/master/src/stream.c#L779 https://github.com/libuvc/libuvc/blob/master/src/stream.c#L787 We have not fixed this officially, please look at the code snippet i provided official fix and implement something similar until we fix it. Alex ## Reply by AndriiHlyvko · 2024-08-28 18:02:08 UTC (in reply to Alex Kushleyev) Thanks @Alex-Kushleyev . Do you know if there are any differences between using the voxl fork of libuvc vs compiling the master libuvc? I was thinking of just building the master libuvc but I am not sure if modalai did any changes to it. ## Reply by Alex Kushleyev (ModalAI staff) · 2024-08-28 19:19:25 UTC (in reply to AndriiHlyvko) @AndriiHlyvko , you can see the changes that we made to the original fork : https://github.com/modalai/libuvc/commits/modalai-master (just the three top commits). You may definitely try to use the latest master from the official repo, but we have not tested it. I believe there was some discussion recently about moving our code to latest, but we have not done it yet. Alex ## Reply by AndriiHlyvko · 2024-08-28 21:03:09 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev just tested with latest master. The seg fault is gone and it seems to be working fine. One new problem that I have after moving to libuvc master is that the video stream from the boson+ looks wrong (the video is green :/ ). Its only wrong for boson+ and not the boson. I'll keep debugging the issue. I was thinking maybe the boson+ sends incomplete frames in the uvc callback but I'm not sure. ## Reply by AndriiHlyvko · 2024-08-28 22:32:01 UTC (in reply to AndriiHlyvko) @Alex-Kushleyev figured it out. I had a bug that I introduced. I broke the mapping between UVC_FRAME_FORMAT enum and the voxl IMAGE_FORMAT enum in my fork of the uvc-server. Right now uvc streaming is working well with the boson+ ## Reply by Alex Kushleyev (ModalAI staff) · 2024-08-30 16:34:08 UTC (in reply to AndriiHlyvko) @AndriiHlyvko , that's great! If you are willing to help out and share your changes on top of libuvc master, it should help us get this updated more quickly :) Thank you Alex ## Reply by AndriiHlyvko · 2024-09-03 15:58:29 UTC (in reply to Alex Kushleyev) @Alex-Kushleyev sure. Should I make a merge request for it and tag you in it? ## Reply by Alex Kushleyev (ModalAI staff) · 2024-09-03 20:09:15 UTC (in reply to AndriiHlyvko) @AndriiHlyvko , yes and please tag @Eric-Katzfey who will probably review and merge. Thank you! Alex ## Reply by AndriiHlyvko · 2024-09-04 18:22:51 UTC (in reply to Alex Kushleyev) Hey @Alex-Kushleyev and @Eric-Katzfey here is the MR https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-libuvc/-/merge_requests/3#59f81c123b3abbcb97274545796dd18706c3e106 ## Reply by Alex Kushleyev (ModalAI staff) · 2024-09-11 03:53:51 UTC (in reply to AndriiHlyvko) @AndriiHlyvko , Thank you! I will ask @Eric-Katzfey to review at his convenience. ## Reply by Alex Kushleyev (ModalAI staff) · 2025-01-14 04:07:21 UTC (in reply to Alex Kushleyev) @AndriiHlyvko, I just ran into the same issue while testing Boson (in my case I was testing the 16-bit Boson streaming mode). The libuvc issue caused a crash, since i did not use a fixed version of libuvc. However, what actually caused the buffer overflow was the fact that Boson was reporting smaller buffer requirements for RAW16 during query (it was reporting `width * height * 1.5` , just like for YUV, but actual frame size was `width * height * 2`, so uvc pipeline allocated smaller buffer and overflow happened with the full frame came in. Was this the same issue with your Boson tests or did you experience something else? Thanks! Alex