Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d

Róbert Dévényi

@Eric-Katzfey when I run voxl-uvc-server -f 15, 20, ... all lower frame rates give me the same seg fault. How can I stop other services from running?

Eric Katzfey

@Róbert-Dévényi You can use the voxl-inspect-services utility to see what services have been enabled. For each of those services you can either stop them or disable them. If you just stop them they will still start automatically on next reboot. If you disable them then you will have to re-enable them once you are done debugging. Stopping a service: systemctl stop <service name>, disabling a service systemctl disable <service name>.

Róbert Dévényi

@Eric-Katzfey Thanks. Which services are the most likely to cause a segfault?

Róbert Dévényi

@Eric-Katzfey it seems that for some reason the voxl-uvc-server service just stops running by itself even after I run systemctl start voxl-uvc-server or systemctl enable voxl-uvc-server. Probably that causes the segfault, that the service is not running.

Eric Katzfey

@Róbert-Dévényi Nothing should be causing a segfault. I'm just offering suggestions on how to debug what is going on.

Eric Katzfey

@Róbert-Dévényi The service just runs the application.

Róbert Dévényi

@Eric-Katzfey so I stopped all services except for voxl-uvc-server:
Scanning services...

Service Name | Enabled | Running | CPU Usage

And I am still getting the exact same segfault.

Róbert Dévényi

loading config file
Enabling debug messages

width: 640
height: 480
fps: 30
pipe_name: uvc

voxl-uvc-server starting
Image resolution 640x480, 30 fps chosen
Vendor ID 0x534d chosen
UVC initialized
Device 534d:0000 found
Device opened
uvc_get_stream_ctrl_format_size succeeded for format YUYV
Streaming starting

Segmentation fault:
Fault thread: voxl-uvc-server(tid: 16852)
Fault address: 0x7f7dda3000
Access to this address is not allowed.
Segmentation fault

Róbert Dévényi

@Eric-Katzfey I don't know if it helps but whenever I run voxl-uvc-server, the segfault happens and after that the voxl-uvc-server service stops running, but is still enabled.

Eric Katzfey

@Róbert-Dévényi The service just runs the application. If the application stops then the service stops. Are you familiar with Linux and systemd? It's just a standard Linux startup mechanism. But you want to stop all services, even for voxl-uvc-server, then try to start voxl-uvc-server on the command line and see if it still segfaults. There are ways to debug segfaults. But you would need to understand how to get Linux to generate a core file after the segfault and how to load the core into gdb.

Róbert Dévényi

@Eric-Katzfey I would appreciate it if you could tell me what commands to use to stop all the services and then restart voxl-uvc-server or any resource that helps with debugging segfaults in general.

Róbert Dévényi

@Eric-Katzfey I reset all mpa services to factory settings and I keep getting the same segfault.

Alex Kushleyev

@Róbert-Dévényi ,

You can stop the systemd service for voxl-uvc-server using the following command:

systemctl stop voxl-uvc-server
this will stop the service / server but will not prevent it from auto-starting during next reboot

Then you can run the voxl-uvc-server in foreground. Just run it with the normal arguments:

voxl-uvc-server <your additional arguments here>

Confirm that this still crashes.. then run the same executable in gdb:

> gdb /usr/bin/voxl-uvc-server
(gdb) r <your additional arguments here>
...
wait for crash...
...
bt

bt command in gdb will print the backtrace of the call stack and will tell you which function the seg fault happened in (and you will see the call stack). Maybe this will help with debugging.

Alex

Róbert Dévényi

@Alex-Kushleyev Thank you very much hopefully I can figure it out.

Róbert Dévényi

@Alex-Kushleyev I can probably show you the backtrace log so that you can take a look at what causes the segfault.

AndriiHlyvko

@Róbert-Dévényi @Eric-Katzfey @Alex-Kushleyev I am getting the same seg fault with a boson+ camera. I determined that the seg fault occurs when the uvc_start_streaming function is called (I am running a modified version of voxl-uvc-server). I have the stack trace from gdb. If I figure out the cause of the problem I will post it here.

(gdb) run
Starting program: /usr/bin/voxl-uvc-server 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
VOXL UVC Server
loading config file
VOXL UVC Server parsed argument
voxl-uvc-server starting
Image resolution 640x512, 30 fps chosen
[New Thread 0x7ff7a7e1c0 (LWP 2189)]
UVC initialized
Device found
[New Thread 0x7ff727d1c0 (LWP 2190)]
Device opened
Found desired frame format: 1
uvc_get_stream_ctrl_format_size succeeded for format NV12
bmHint: 0001
bFormatIndex: 2
bFrameIndex: 1
dwFrameInterval: 333333
wKeyFrameRate: 1
wPFrameRate: 0
wCompQuality: 0
wCompWindowSize: 0
wDelay: 0
dwMaxVideoFrameSize: 491520
dwMaxPayloadTransferSize: 983044
bInterfaceNumber: 1
[New Thread 0x7ff6a7c1c0 (LWP 2191)]
[New Thread 0x7ff627b1c0 (LWP 2192)]
[New Thread 0x7ff5a7a1c0 (LWP 2193)]
Mavlink Onboard server Connected 
[New Thread 0x7fe70ef1c0 (LWP 2194)]
Streaming starting
[New Thread 0x7feffff1c0 (LWP 2195)]

Thread 3 "voxl-uvc-server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff727d1c0 (LWP 2190)]
__memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
185     ../sysdeps/aarch64/multiarch/../memcpy.S: No such file or directory.
(gdb) bt
#0  __memcpy_generic () at ../sysdeps/aarch64/multiarch/../memcpy.S:185
#1  0x0000007ff7ed32a0 in _uvc_process_payload () from /usr/lib/libuvc.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Alex Kushleyev

@AndriiHlyvko , we have gotten another report about this recently. It seem there is a bug in libuvc which has been fixed, but we are using older version of the library.

Inserting the following check at the line below has been reported to fix the issue:

if (strmh->got_bytes + data_len > strmh->cur_ctrl.dwMaxVideoFrameSize) {
    UVC_DEBUG("Transfer too large and will overflow the buffer. Dropping data!");
    _uvc_swap_buffers(strmh);
    return;
  }

here: https://github.com/modalai/libuvc/blob/3fb2d2fd31f215f76bd5d6782785385fd0927ce5/src/stream.c#L757

It seems the fix in the official libuvc release is here:
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L779
https://github.com/libuvc/libuvc/blob/master/src/stream.c#L787

We have not fixed this officially, please look at the code snippet i provided official fix and implement something similar until we fix it.

Alex

AndriiHlyvko

Thanks @Alex-Kushleyev . Do you know if there are any differences between using the voxl fork of libuvc vs compiling the master libuvc? I was thinking of just building the master libuvc but I am not sure if modalai did any changes to it.

Alex Kushleyev

@AndriiHlyvko , you can see the changes that we made to the original fork : https://github.com/modalai/libuvc/commits/modalai-master (just the three top commits).

You may definitely try to use the latest master from the official repo, but we have not tested it. I believe there was some discussion recently about moving our code to latest, but we have not done it yet.

Alex

AndriiHlyvko

@Alex-Kushleyev just tested with latest master. The seg fault is gone and it seems to be working fine. One new problem that I have after moving to libuvc master is that the video stream from the boson+ looks wrong (the video is green ). Its only wrong for boson+ and not the boson. I'll keep debugging the issue. I was thinking maybe the boson+ sends incomplete frames in the uvc callback but I'm not sure.

Sentinel segmentation fault after running voxl-uvc-server -v [camera id] -d

Service Name | Enabled | Running | CPU Usage

loading config file Enabling debug messages

width: 640 height: 480 fps: 30 pipe_name: uvc

loading config file
Enabling debug messages

width: 640
height: 480
fps: 30
pipe_name: uvc