Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d

Róbert Dévényi

@Eric-Katzfey I would appreciate it if you could tell me what commands to use to stop all the services and then restart voxl-uvc-server or any resource that helps with debugging segfaults in general.

Róbert Dévényi

@Eric-Katzfey I reset all mpa services to factory settings and I keep getting the same segfault.

Alex Kushleyev

@Róbert-Dévényi ,

You can stop the systemd service for voxl-uvc-server using the following command:

systemctl stop voxl-uvc-server
this will stop the service / server but will not prevent it from auto-starting during next reboot

Then you can run the voxl-uvc-server in foreground. Just run it with the normal arguments:

voxl-uvc-server <your additional arguments here>

Confirm that this still crashes.. then run the same executable in gdb:

> gdb /usr/bin/voxl-uvc-server
(gdb) r <your additional arguments here>
...
wait for crash...
...
bt

bt command in gdb will print the backtrace of the call stack and will tell you which function the seg fault happened in (and you will see the call stack). Maybe this will help with debugging.

Alex

Róbert Dévényi

@Alex-Kushleyev Thank you very much hopefully I can figure it out.

Róbert Dévényi

@Alex-Kushleyev I can probably show you the backtrace log so that you can take a look at what causes the segfault.

AndriiHlyvko

@Róbert-Dévényi @Eric-Katzfey @Alex-Kushleyev I am getting the same seg fault with a boson+ camera. I determined that the seg fault occurs when the uvc_start_streaming function is called (I am running a modified version of voxl-uvc-server). I have the stack trace from gdb. If I figure out the cause of the problem I will post it here.

(gdb) run
Starting program: /usr/bin/voxl-uvc-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
VOXL UVC Server
loading config file
VOXL UVC Server parsed argument
voxl-uvc-server starting
Image resolution 640x512, 30 fps chosen
[New Thread 0x7ff7a7e1c0 (LWP 2189)]
UVC initialized
Device found
[New Thread 0x7ff727d1c0 (LWP 2190)]
Device opened
Found desired frame format: 1
uvc_get_stream_ctrl_format_size succeeded for format NV12
bmHint: 0001
bFormatIndex: 2
bFrameIndex: 1
dwFrameInterval: 333333
wKeyFrameRate: 1
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 0
dwMaxVideoFrameSize: 491520
dwMaxPayloadTransferSize: 983044
bInterfaceNumber: 1
[New Thread 0x7ff6a7c1c0 (LWP 2191)]
[New Thread 0x7ff627b1c0 (LWP 2192)]
[New Thread 0x7ff5a7a1c0 (LWP 2193)]
Mavlink Onboard server Connected 
[New Thread 0x7fe70ef1c0 (LWP 2194)]
Streaming starting
[New Thread 0x7feffff1c0 (LWP 2195)]

Thread 3 "voxl-uvc-server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff727d1c0 (LWP 2190)]
__memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
185     ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory.
(gdb) bt
#0  __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
#1  0x0000007ff7ed32a0 in _uvc_process_payload () from /usr/lib/libuvc.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Alex Kushleyev

@AndriiHlyvko , we have gotten another report about this recently. It seem there is a bug in libuvc which has been fixed, but we are using older version of the library.

Inserting the following check at the line below has been reported to fix the issue:

if (strmh->got_bytes + data_len > strmh->cur_ctrl.dwMaxVideoFrameSize) {
    UVC_DEBUG("Transfer too large and will overflow the buffer. Dropping data!");
    _uvc_swap_buffers(strmh);
    return;
  }

here: https://github.com/modalai/libuvc/blob/3fb2d2fd31f215f76bd5d6782785385fd0927ce5/src/stream.c#L757

It seems the fix in the official libuvc release is here:
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L779
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L787

We have not fixed this officially, please look at the code snippet i provided official fix and implement something similar until we fix it.

Alex

AndriiHlyvko

Thanks @Alex-Kushleyev . Do you know if there are any differences between using the voxl fork of libuvc vs compiling the master libuvc? I was thinking of just building the master libuvc but I am not sure if modalai did any changes to it.

Alex Kushleyev

@AndriiHlyvko , you can see the changes that we made to the original fork : https://github.com/modalai/libuvc/commits/modalai-master (just the three top commits).

You may definitely try to use the latest master from the official repo, but we have not tested it. I believe there was some discussion recently about moving our code to latest, but we have not done it yet.

Alex

AndriiHlyvko

@Alex-Kushleyev just tested with latest master. The seg fault is gone and it seems to be working fine. One new problem that I have after moving to libuvc master is that the video stream from the boson+ looks wrong (the video is green ). Its only wrong for boson+ and not the boson. I'll keep debugging the issue. I was thinking maybe the boson+ sends incomplete frames in the uvc callback but I'm not sure.

AndriiHlyvko

@Alex-Kushleyev figured it out. I had a bug that I introduced. I broke the mapping between UVC_FRAME_FORMAT enum and the voxl IMAGE_FORMAT enum in my fork of the uvc-server. Right now uvc streaming is working well with the boson+

Alex Kushleyev

@AndriiHlyvko , that's great!

If you are willing to help out and share your changes on top of libuvc master, it should help us get this updated more quickly

Thank you

Alex

AndriiHlyvko

@Alex-Kushleyev sure. Should I make a merge request for it and tag you in it?

Alex Kushleyev

@AndriiHlyvko , yes and please tag @Eric-Katzfey who will probably review and merge.

Thank you!

Alex

AndriiHlyvko

Hey @Alex-Kushleyev and @Eric-Katzfey here is the MR https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-libuvc/-/merge_requests/3#59f81c123b3abbcb97274545796dd18706c3e106

Alex Kushleyev

@AndriiHlyvko , Thank you! I will ask @Eric-Katzfey to review at his convenience.

Alex Kushleyev

@AndriiHlyvko,

I just ran into the same issue while testing Boson (in my case I was testing the 16-bit Boson streaming mode). The libuvc issue caused a crash, since i did not use a fixed version of libuvc. However, what actually caused the buffer overflow was the fact that Boson was reporting smaller buffer requirements for RAW16 during query (it was reporting width * height * 1.5 , just like for YUV, but actual frame size was width * height * 2, so uvc pipeline allocated smaller buffer and overflow happened with the full frame came in.

Was this the same issue with your Boson tests or did you experience something else?

Thanks!

Alex