Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d

Eric Katzfey

@Róbert-Dévényi The service just runs the application.

Róbert Dévényi

@Eric-Katzfey so I stopped all services except for voxl-uvc-server:
Scanning services...

Service Name | Enabled | Running | CPU Usage

And I am still getting the exact same segfault.

Róbert Dévényi

loading config file
Enabling debug messages

width: 640
height: 480
fps: 30
pipe_name: uvc

voxl-uvc-server starting
Image resolution 640x480, 30 fps chosen
Vendor ID 0x534d chosen
UVC initialized
Device 534d:0000 found
Device opened
uvc_get_stream_ctrl_format_size succeeded for format YUYV
Streaming starting

Segmentation fault:
Fault thread: voxl-uvc-server(tid: 16852)
Fault address: 0x7f7dda3000
Access to this address is not allowed.
Segmentation fault

Róbert Dévényi

@Eric-Katzfey I don't know if it helps but whenever I run voxl-uvc-server, the segfault happens and after that the voxl-uvc-server service stops running, but is still enabled.

Eric Katzfey

@Róbert-Dévényi The service just runs the application. If the application stops then the service stops. Are you familiar with Linux and systemd? It's just a standard Linux startup mechanism. But you want to stop all services, even for voxl-uvc-server, then try to start voxl-uvc-server on the command line and see if it still segfaults. There are ways to debug segfaults. But you would need to understand how to get Linux to generate a core file after the segfault and how to load the core into gdb.

Róbert Dévényi

@Eric-Katzfey I would appreciate it if you could tell me what commands to use to stop all the services and then restart voxl-uvc-server or any resource that helps with debugging segfaults in general.

Róbert Dévényi

@Eric-Katzfey I reset all mpa services to factory settings and I keep getting the same segfault.

Alex Kushleyev

@Róbert-Dévényi ,

You can stop the systemd service for voxl-uvc-server using the following command:

systemctl stop voxl-uvc-server
this will stop the service / server but will not prevent it from auto-starting during next reboot

Then you can run the voxl-uvc-server in foreground. Just run it with the normal arguments:

voxl-uvc-server <your additional arguments here>

Confirm that this still crashes.. then run the same executable in gdb:

> gdb /usr/bin/voxl-uvc-server
(gdb) r <your additional arguments here>
...
wait for crash...
...
bt

bt command in gdb will print the backtrace of the call stack and will tell you which function the seg fault happened in (and you will see the call stack). Maybe this will help with debugging.

Alex

Róbert Dévényi

@Alex-Kushleyev Thank you very much hopefully I can figure it out.

Róbert Dévényi

@Alex-Kushleyev I can probably show you the backtrace log so that you can take a look at what causes the segfault.

AndriiHlyvko

@Róbert-Dévényi @Eric-Katzfey @Alex-Kushleyev I am getting the same seg fault with a boson+ camera. I determined that the seg fault occurs when the uvc_start_streaming function is called (I am running a modified version of voxl-uvc-server). I have the stack trace from gdb. If I figure out the cause of the problem I will post it here.

(gdb) run
Starting program: /usr/bin/voxl-uvc-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
VOXL UVC Server
loading config file
VOXL UVC Server parsed argument
voxl-uvc-server starting
Image resolution 640x512, 30 fps chosen
[New Thread 0x7ff7a7e1c0 (LWP 2189)]
UVC initialized
Device found
[New Thread 0x7ff727d1c0 (LWP 2190)]
Device opened
Found desired frame format: 1
uvc_get_stream_ctrl_format_size succeeded for format NV12
bmHint: 0001
bFormatIndex: 2
bFrameIndex: 1
dwFrameInterval: 333333
wKeyFrameRate: 1
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 0
dwMaxVideoFrameSize: 491520
dwMaxPayloadTransferSize: 983044
bInterfaceNumber: 1
[New Thread 0x7ff6a7c1c0 (LWP 2191)]
[New Thread 0x7ff627b1c0 (LWP 2192)]
[New Thread 0x7ff5a7a1c0 (LWP 2193)]
Mavlink Onboard server Connected 
[New Thread 0x7fe70ef1c0 (LWP 2194)]
Streaming starting
[New Thread 0x7feffff1c0 (LWP 2195)]

Thread 3 "voxl-uvc-server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff727d1c0 (LWP 2190)]
__memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
185     ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory.
(gdb) bt
#0  __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
#1  0x0000007ff7ed32a0 in _uvc_process_payload () from /usr/lib/libuvc.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Alex Kushleyev

@AndriiHlyvko , we have gotten another report about this recently. It seem there is a bug in libuvc which has been fixed, but we are using older version of the library.

Inserting the following check at the line below has been reported to fix the issue:

if (strmh->got_bytes + data_len > strmh->cur_ctrl.dwMaxVideoFrameSize) {
    UVC_DEBUG("Transfer too large and will overflow the buffer. Dropping data!");
    _uvc_swap_buffers(strmh);
    return;
  }

here: https://github.com/modalai/libuvc/blob/3fb2d2fd31f215f76bd5d6782785385fd0927ce5/src/stream.c#L757

It seems the fix in the official libuvc release is here:
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L779
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L787

We have not fixed this officially, please look at the code snippet i provided official fix and implement something similar until we fix it.

Alex

AndriiHlyvko

Thanks @Alex-Kushleyev . Do you know if there are any differences between using the voxl fork of libuvc vs compiling the master libuvc? I was thinking of just building the master libuvc but I am not sure if modalai did any changes to it.

Alex Kushleyev

@AndriiHlyvko , you can see the changes that we made to the original fork : https://github.com/modalai/libuvc/commits/modalai-master (just the three top commits).

You may definitely try to use the latest master from the official repo, but we have not tested it. I believe there was some discussion recently about moving our code to latest, but we have not done it yet.

Alex

AndriiHlyvko

@Alex-Kushleyev just tested with latest master. The seg fault is gone and it seems to be working fine. One new problem that I have after moving to libuvc master is that the video stream from the boson+ looks wrong (the video is green ). Its only wrong for boson+ and not the boson. I'll keep debugging the issue. I was thinking maybe the boson+ sends incomplete frames in the uvc callback but I'm not sure.

AndriiHlyvko

@Alex-Kushleyev figured it out. I had a bug that I introduced. I broke the mapping between UVC_FRAME_FORMAT enum and the voxl IMAGE_FORMAT enum in my fork of the uvc-server. Right now uvc streaming is working well with the boson+

Alex Kushleyev

@AndriiHlyvko , that's great!

If you are willing to help out and share your changes on top of libuvc master, it should help us get this updated more quickly

Thank you

Alex

AndriiHlyvko

@Alex-Kushleyev sure. Should I make a merge request for it and tag you in it?

Alex Kushleyev

@AndriiHlyvko , yes and please tag @Eric-Katzfey who will probably review and merge.

Thank you!

Alex

AndriiHlyvko

Hey @Alex-Kushleyev and @Eric-Katzfey here is the MR https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-libuvc/-/merge_requests/3#59f81c123b3abbcb97274545796dd18706c3e106

Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d

Service Name | Enabled | Running | CPU Usage

loading config file Enabling debug messages

width: 640 height: 480 fps: 30 pipe_name: uvc

loading config file
Enabling debug messages

width: 640
height: 480
fps: 30
pipe_name: uvc