Camera server crashes with any use of stereo cameras.



  • Hello. I have found a problem when trying to calibrate the stereo cameras, or when I try to utilize them at all.
    As soon as any service tries to pull a frame from the stereo cameras this happens (for example, when starting up voxl-portal to begin the calibration procedure):

    Initialize
    Initialize
    Received raw10 frame, checking t o se if is actually raw8
    Frame was actually 8 bit, sending as is
    Internal AE new val:exposure 0 gain 0
    	Port		: port_33
    	Name		: stereo
    	Enabled		: 1
    	Type		: stereo
    	Api		: hal3
    	p-W		: 1280
    	p-H		: 480
    	p-Fmt		: nv21
    	FPS		: 30
    	OverrideId	: -1
    	AEAlgo	: modalai
    ...
    Internal AE new val:exposure 234 gain 234
    Internal AE new val:exposure 221 gain 234
    Internal AE new val:exposure 221 gain 221
    ------ voxl-camera-server INFO: Client: voxl-portal0 connected to channel: 1
    voxl-camera-server ERROR: Error sending request 2, ErrorCode: -19
    
    voxl-camera-server FATAL: Recieved Fatal error from camera: stereo
                              Camera server will be stopped
    ------ voxl-camera-server WARNING: Thread: stereo request thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: tracking result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: hires result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: stereo result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: hires request thread recieved ESTOP
    
    ------ voxl-camera-server INFO: Camera server is now stopping
                    There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
    
    ------ voxl-camera-server INFO: Stopping tracking camera
    voxl-camera-server ERROR: Framenumber: 3676 ErrorCode: 2
    voxl-camera-server ERROR: Framenumber: 3677 ErrorCode: 2
    voxl-camera-server ERROR: Framenumber: 3678 ErrorCode: 2
    ------ voxl-camera-server WARNING: Thread: tracking request thread recieved ESTOP
    ------ voxl-camera-server INFO: tracking camera stopped successfully
    
    ------ voxl-camera-server INFO: Stopping hires camera
    ------ voxl-camera-server INFO: hires camera stopped successfully
    ------ voxl-camera-server INFO: Stopping stereo camera
    ------ voxl-camera-server INFO: stereo camera stopped successfully
    
    ------ voxl-camera-server INFO: Camera server exited gracefully
    

    From what I see in the camera HAL3 API, -ENODEV just refers to an internal, and is a rather non-descriptive error.
    The camera server has already been configured.
    The camera's physical connection seems fine.
    The rest of the cameras operate correctly in voxl-portal and have been calibrated.

    voxl-version outputs:

    cat: /etc/modalai/voxl-software-bundle-version.txt: No such file or directory
    --------------------------------------------------------------------------------
    system-image:    ModalAI 3.3.0 BUILDER: ekatzfey BUILD_TIME: 2021-06-06_19:28
    kernel:          #1 SMP PREEMPT Sun Jun 6 19:41:01 UTC 2021 3.18.71-perf
    factory-bundle:  1.0.1 (Yocto installation)
    --------------------------------------------------------------------------------
    architecture:    aarch64
    processor:       apq8096
    os:              GNU/Linux
    --------------------------------------------------------------------------------
    voxl-suite:
    Package: voxl-suite
    Version: 0.5.0
    Depends: libjpeg_turbo (>= 9.0.4), libmodal_exposure (>= 0.0.2), libmodal_json (>= 0.3.6), libmodal_pipe (>= 2.1.1), librc_math (>= 1.1.5), libvoxl_cutils (>= 0.0.2), libvoxl_io (>= 0.5.4), mavlink-camera-manager (>= 0.0.2), mongoose (>= 7.3.0), opencv (>= 4.5.2-2), openmp (>= 10.0.2), voxl-camera-calibration (>= 0.1.1), voxl-camera-server (>= 0.8.1), voxl-cpu-monitor (>= 0.2.0), voxl-dfs-server (>= 0.2.2), voxl-docker-support (>= 1.1.3), voxl-gphoto2 (>= 0.0.5), voxl-imu-server (>= 0.9.1), voxl-mavlink (>= 0.0.2), voxl-modem (>= 0.12.0), voxl-mpa-tools (>= 0.3.6), voxl-nodes (>= 0.2.0), voxl-portal (>= 0.1.2), voxl-qvio-server (>= 0.3.4), voxl-streamer (>= 0.2.6), voxl-tag-detector (>= 0.0.2), voxl-tflite (>= 2.2.3), voxl-tflite-server (>= 0.1.5), voxl-utils (>= 0.8.4), voxl-vision-px4 (>= 0.9.5), voxl-vpn (>= 0.0.3)
    Status: install user installed
    Section: base
    Architecture: all
    Maintainer: james@modalai.com
    MD5Sum: f55ec020942e92ff12bc137c2aa8d2a5
    Size: 1956
    Filename: voxl-suite_0.5.0.ipk
    Description: meta-package for voxl-suite software collection
    Installed-Time: 1640891859
    
    Package: voxl-suite
    Version: 0.2.0
    Depends: docker, imu_app (= 0.0.6), libvoxl_io (= 0.5.2), voxl-cam-manager (= 0.2.2), voxl-docker-support (= 1.1.1), voxl-hal3-tof-cam-ros (= 0.0.2), voxl-modem (= 0.10.0), voxl-nodes (= 0.0.8), voxl-rtsp (= 1.0.2), voxl-utils (= 0.5.2), voxl-vision-px4 (= 0.6.8), voxl_imu (= 0.0.4), voxl-time-sync (= 0.0.1), voxl-vpn (= 0.0.2), librc_math (= 1.1.2), libmodal_pipe (= 1.2.2), modalai-vl (= 0.1.3)
    Status: unknown ok not-installed
    Section: base
    Architecture: armv7a
    Maintainer: james@modalai.com
    MD5Sum: af706cd3c1ea59f274f2ed9b93141f1d
    Size: 870
    Filename: voxl-suite_0.2.0.ipk
    Description: meta-package to install all of the voxl-suite
    
    --------------------------------------------------------------------------------
    

  • Dev Team

    This issue pops up from time to time and it is, without fail, a hardware issue. Can you check not just the connectors for your stereo cameras but also the cables themselves. They can develop kinks or get scratched and expose the traces if they go through any number of unfriendly environments, which will regularly lead to hal3 being unable to see the device.


Log in to reply