Camera server crashes with any use of stereo cameras.

[mud za]

Hello. I have found a problem when trying to calibrate the stereo cameras, or when I try to utilize them at all.
As soon as any service tries to pull a frame from the stereo cameras this happens (for example, when starting up voxl-portal to begin the calibration procedure):

Initialize
Initialize
Received raw10 frame, checking t o se if is actually raw8
Frame was actually 8 bit, sending as is
Internal AE new val:exposure 0 gain 0
	Port		: port_33
	Name		: stereo
	Enabled		: 1
	Type		: stereo
	Api		: hal3
	p-W		: 1280
	p-H		: 480
	p-Fmt		: nv21
	FPS		: 30
	OverrideId	: -1
	AEAlgo	: modalai
...
Internal AE new val:exposure 234 gain 234
Internal AE new val:exposure 221 gain 234
Internal AE new val:exposure 221 gain 221
------ voxl-camera-server INFO: Client: voxl-portal0 connected to channel: 1
voxl-camera-server ERROR: Error sending request 2, ErrorCode: -19

voxl-camera-server FATAL: Recieved Fatal error from camera: stereo
                          Camera server will be stopped
------ voxl-camera-server WARNING: Thread: stereo request thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: tracking result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: hires result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: stereo result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: hires request thread recieved ESTOP

------ voxl-camera-server INFO: Camera server is now stopping
                There is a chance that it may segfault here, this is a mmqcamera bug, ignore it

------ voxl-camera-server INFO: Stopping tracking camera
voxl-camera-server ERROR: Framenumber: 3676 ErrorCode: 2
voxl-camera-server ERROR: Framenumber: 3677 ErrorCode: 2
voxl-camera-server ERROR: Framenumber: 3678 ErrorCode: 2
------ voxl-camera-server WARNING: Thread: tracking request thread recieved ESTOP
------ voxl-camera-server INFO: tracking camera stopped successfully

------ voxl-camera-server INFO: Stopping hires camera
------ voxl-camera-server INFO: hires camera stopped successfully
------ voxl-camera-server INFO: Stopping stereo camera
------ voxl-camera-server INFO: stereo camera stopped successfully

------ voxl-camera-server INFO: Camera server exited gracefully

From what I see in the camera HAL3 API, -ENODEV just refers to an internal, and is a rather non-descriptive error.
The camera server has already been configured.
The camera's physical connection seems fine.
The rest of the cameras operate correctly in voxl-portal and have been calibrated.

voxl-version outputs:

cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory
--------------------------------------------------------------------------------
system-image:    ModalAI 3.3.0 BUILDER: ekatzfey BUILD_TIME: 2021-06-06_19:28
kernel:          #1 SMP PREEMPT Sun Jun 6 19:41:01 UTC 2021 3.18.71-perf
factory-bundle:  1.0.1 (Yocto installation)
--------------------------------------------------------------------------------
architecture:    aarch64
processor:       apq8096
os:              GNU/Linux
--------------------------------------------------------------------------------
voxl-suite:
Package: voxl-suite
Version: 0.5.0
Depends: libjpeg_turbo (>= 9.0.4), libmodal_exposure (>= 0.0.2), libmodal_json (>= 0.3.6), libmodal_pipe (>= 2.1.1), librc_math (>= 1.1.5), libvoxl_cutils (>= 0.0.2), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.2), mongoose (>= 7.3.0), opencv (>= 4.5.2-2), openmp (>= 10.0.2), voxl-camera-calibration (>= 0.1.1), voxl-camera-server (>= 0.8.1), voxl-cpu-monitor (>= 0.2.0), voxl-dfs-server (>= 0.2.2), voxl-docker-support (>= 1.1.3), voxl-gphoto2 (>= 0.0.5), voxl-imu-server (>= 0.9.1), voxl-mavlink (>= 0.0.2), voxl-modem (>= 0.12.0), voxl-mpa-tools (>= 0.3.6), voxl-nodes (>= 0.2.0), voxl-portal (>= 0.1.2), voxl-qvio-server (>= 0.3.4), voxl-streamer (>= 0.2.6), voxl-tag-detector (>= 0.0.2), voxl-tflite (>= 2.2.3), voxl-tflite-server (>= 0.1.5), voxl-utils (>= 0.8.4), voxl-vision-px4 (>= 0.9.5), voxl-vpn (>= 0.0.3)
Status: install user installed
Section: base
Architecture: all
Maintainer: james@modalai.com
MD5Sum: f55ec020942e92ff12bc137c2aa8d2a5
Size: 1956
Filename: voxl-suite_0.5.0.ipk
Description: meta-package for voxl-suite software collection
Installed-Time: 1640891859

Package: voxl-suite
Version: 0.2.0
Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
Status: unknown ok not-installed
Section: base
Architecture: armv7a
Maintainer: james@modalai.com
MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
Size: 870
Filename: voxl-suite_0.2.0.ipk
Description: meta-package to install all of the voxl-suite

--------------------------------------------------------------------------------

[A Former User]

This issue pops up from time to time and it is, without fail, a hardware issue. Can you check not just the connectors for your stereo cameras but also the cables themselves. They can develop kinks or get scratched and expose the traces if they go through any number of unfriendly environments, which will regularly lead to hal3 being unable to see the device.