Odd conditions resulting in voxl-camera-server crash

hmlow

Hi devs

So im still trying to troubleshoot the voxl-camera-server crashes that i encountered as described here:
https://forum.modalai.com/post/11480

While doing so i discovered that there are odd conditions that would trigger the voxl-camera-server to crash.

Moving the putty window
-SSH into VOXL2 (im using the Windows ver of putty)
-run voxl-camera-server in debug voxl-camera-server -d 0
-Click and drag the putty window to move it around
-voxl-camera-server crashes

Delaying the startup of voxl-streamer
-Add the following directives to voxl-streamer's systemd unit file:
Type=oneshot
ExecStartPre=/bin/sleep 45
-voxl-camera-server crashes randomly - sometimes upon bootup, sometimes several mins after
-changing the Type to simple and removing the sleep duration solves the problem

Im not sure if they are related by here (blocking processes?)

Thanks

hmlow

@hmlow

no one?
am i really the only one encountering these crashes?
once in a blue moon the crash still happens during bootup for no specific reason (that i could tell)

Final few log prints before it crashes:

VERBOSE: Camera: tof processing preview frame
VERBOSE: finished sending request for frame 1477 for camera tof
VERBOSE: returning from SendOneCaptureRequest for frame 1478 for camera tof
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 1478 for camera tof for 1 streams
VERBOSE: Received result from HAl3 for frame number 1477
VERBOSE: Received 0 buffers from camera tof, partial result:1
VERBOSE: Sent tof data to royale for processing
VERBOSE: Received result from HAl3 for frame number 8847
VERBOSE: Received 0 buffers from camera tracking, partial result:2
VERBOSE: Received metadata for frame 8847 from camera tracking
VERBOSE:        Timestamp: 1277332818009
VERBOSE:        Gain: 169
VERBOSE:        Exposure: 5184081
VERBOSE: Received result from HAl3 for frame number 8847
VERBOSE: Received 1 buffers from camera tracking, partial result:0
VERBOSE: Received output buffer 8847 from camera tracking
VERBOSE: tracking procesing new buffer
VERBOSE: Camera: tracking processing preview frame
VERBOSE: Preview format HAL_PIXEL_FORMAT_RAW10
VERBOSE: finished sending request for frame 8852 for camera tracking
VERBOSE: returning from SendOneCaptureRequest for frame 8853 for camera tracking
VERBOSE: added request for preview stream
VERBOSE: Sending request for frame 8853 for camera tracking for 1 streams
VERBOSE: Received result from HAl3 for frame number 8852
VERBOSE: Received 0 buffers from camera tracking, partial result:1
VERBOSE: Received royale data for camera: tof
ERROR:   Received "Buffer" error from camera: tracking
ERROR:   Received "Result" error from camera: tracking

I encountered the same crash on a completely new and different set of hardware (VOXL2, cameras, etc) as well.
Surely it cant be a h/w problem?

@hmlow

Hello, sorry to hear you've been having issues with this. Looking through our code those error messages are coming directly from our camera module so I can't say much as to why it's happening.

If you're using a completely new set of hardware then the problem seems software related. Do you have any way of reproducing this issue on a non-Windows system? Ubuntu 20.04 would be preferable as it's our only officially supported operating system.

I'm not an expert in this area so I've pinged someone more experienced who should be able to provide more help.

Best of luck!

Thomas

hmlow

@Thomas-Patton thanks for looking into this.

The random crash that I described in my second message in this thread is not related to the OS on which I run putty on. The crash happens on the VOXL2, sometimes even before I SSH in.