Camera server crashes with any use of stereo cameras.
-
Hello. I have found a problem when trying to calibrate the stereo cameras, or when I try to utilize them at all.
As soon as any service tries to pull a frame from the stereo cameras this happens (for example, when starting up voxl-portal to begin the calibration procedure):Initialize Initialize Received raw10 frame, checking t o se if is actually raw8 Frame was actually 8 bit, sending as is Internal AE new val:exposure 0 gain 0 Port : port_33 Name : stereo Enabled : 1 Type : stereo Api : hal3 p-W : 1280 p-H : 480 p-Fmt : nv21 FPS : 30 OverrideId : -1 AEAlgo : modalai ... Internal AE new val:exposure 234 gain 234 Internal AE new val:exposure 221 gain 234 Internal AE new val:exposure 221 gain 221 ------ voxl-camera-server INFO: Client: voxl-portal0 connected to channel: 1 voxl-camera-server ERROR: Error sending request 2, ErrorCode: -19 voxl-camera-server FATAL: Recieved Fatal error from camera: stereo Camera server will be stopped ------ voxl-camera-server WARNING: Thread: stereo request thread recieved ESTOP ------ voxl-camera-server WARNING: Thread: tracking result thread recieved ESTOP ------ voxl-camera-server WARNING: Thread: hires result thread recieved ESTOP ------ voxl-camera-server WARNING: Thread: stereo result thread recieved ESTOP ------ voxl-camera-server WARNING: Thread: hires request thread recieved ESTOP ------ voxl-camera-server INFO: Camera server is now stopping There is a chance that it may segfault here, this is a mmqcamera bug, ignore it ------ voxl-camera-server INFO: Stopping tracking camera voxl-camera-server ERROR: Framenumber: 3676 ErrorCode: 2 voxl-camera-server ERROR: Framenumber: 3677 ErrorCode: 2 voxl-camera-server ERROR: Framenumber: 3678 ErrorCode: 2 ------ voxl-camera-server WARNING: Thread: tracking request thread recieved ESTOP ------ voxl-camera-server INFO: tracking camera stopped successfully ------ voxl-camera-server INFO: Stopping hires camera ------ voxl-camera-server INFO: hires camera stopped successfully ------ voxl-camera-server INFO: Stopping stereo camera ------ voxl-camera-server INFO: stereo camera stopped successfully ------ voxl-camera-server INFO: Camera server exited gracefullyFrom what I see in the camera HAL3 API, -ENODEV just refers to an internal, and is a rather non-descriptive error.
The camera server has already been configured.
The camera's physical connection seems fine.
The rest of the cameras operate correctly in voxl-portal and have been calibrated.voxl-version outputs:
cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory -------------------------------------------------------------------------------- system-image: ModalAI 3.3.0 BUILDER: ekatzfey BUILD_TIME: 2021-06-06_19:28 kernel: #1 SMP PREEMPT Sun Jun 6 19:41:01 UTC 2021 3.18.71-perf factory-bundle: 1.0.1 (Yocto installation) -------------------------------------------------------------------------------- architecture: aarch64 processor: apq8096 os: GNU/Linux -------------------------------------------------------------------------------- voxl-suite: Package: voxl-suite Version: 0.5.0 Depends: libjpeg_turbo (>= 9.0.4), libmodal_exposure (>= 0.0.2), libmodal_json (>= 0.3.6), libmodal_pipe (>= 2.1.1), librc_math (>= 1.1.5), libvoxl_cutils (>= 0.0.2), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.2), mongoose (>= 7.3.0), opencv (>= 4.5.2-2), openmp (>= 10.0.2), voxl-camera-calibration (>= 0.1.1), voxl-camera-server (>= 0.8.1), voxl-cpu-monitor (>= 0.2.0), voxl-dfs-server (>= 0.2.2), voxl-docker-support (>= 1.1.3), voxl-gphoto2 (>= 0.0.5), voxl-imu-server (>= 0.9.1), voxl-mavlink (>= 0.0.2), voxl-modem (>= 0.12.0), voxl-mpa-tools (>= 0.3.6), voxl-nodes (>= 0.2.0), voxl-portal (>= 0.1.2), voxl-qvio-server (>= 0.3.4), voxl-streamer (>= 0.2.6), voxl-tag-detector (>= 0.0.2), voxl-tflite (>= 2.2.3), voxl-tflite-server (>= 0.1.5), voxl-utils (>= 0.8.4), voxl-vision-px4 (>= 0.9.5), voxl-vpn (>= 0.0.3) Status: install user installed Section: base Architecture: all Maintainer: james@modalai.com MD5Sum: f55ec020942e92ff12bc137c2aa8d2a5 Size: 1956 Filename: voxl-suite_0.5.0.ipk Description: meta-package for voxl-suite software collection Installed-Time: 1640891859 Package: voxl-suite Version: 0.2.0 Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3) Status: unknown ok not-installed Section: base Architecture: armv7a Maintainer: james@modalai.com MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d Size: 870 Filename: voxl-suite_0.2.0.ipk Description: meta-package to install all of the voxl-suite -------------------------------------------------------------------------------- -
This issue pops up from time to time and it is, without fail, a hardware issue. Can you check not just the connectors for your stereo cameras but also the cables themselves. They can develop kinks or get scratched and expose the traces if they go through any number of unfriendly environments, which will regularly lead to hal3 being unable to see the device.
