Intermittent crashing of voxl-camera-server during VOA testing



  • We have calibrated our stereo cameras per these instructions: https://docs.modalai.com/calibrate-cameras/#calibrate-stereo-cameras.

    We are using the dev version of the voxl-vision-px4 (that fixed a previous bug) and which also tightened the range of accurate obstacle detections from 0.2 -> 20 meters to 0.5 -> 8 meters. In our experience, detections within this range are still quite noisy (sometimes swinging 2-3 meters in tolerance), but they do serve as coarse-grained estimates. In our VOXL, we have enabled the voxl-camera-server and voxl-dfs-server. We confirm this is working by looking at the OBSTACLE_DISTANCE field in the MAVLink Inspector in QGC.

    In QGC -> Vehicle Setup -> Safety -> Collision Avoidance, we have enabled Obstacle Prevention. We have also set the MPC_COL_PREV_D distance to 6.0 meters. We have taken the drone outside, gotten GPS sync, turned on Position mode, and manually attempted to fly it into a large tree with dense foliage. As advertised, the VOA will kick in and prevent us from flying into the tree. (This is great!)

    During this testing, when we fly the drone towards the tree, we see a spike in the CPU usage of the voxl-dfs-server. (We are not surprised by this, but share it as a data point along with our belief that this service is operating as expected and uses more CPU to compute points as it approaches obstacles.)

    Sometimes, when we fly the drone towards the tree, the voxl-camera-server will crash. One time, we saw the voxl-dfs-server crash as well. The crash message from the voxl-camera-server contains an ESTOP message on a single camera stream, although the exact camera stream varies (e.g., hires, tracking, or stereo). When one of these services crash, QGC oscillates between Position Mode and Hold Mode. On our controller, we lose the ability to pitch and roll. We can yaw, ascend and descend, but nothing more. Without being able to fly in any direction, we eventually land.

    We suspect that when the voxl-camera-server crashes, the voxl-dfs-server is unable to get data and does not function. We do not know if this fact is communicated to QGC and is the mechanism that triggers the Hold mode.

    We believe this intermittent crashing is likely due to some bug (perhaps due in some part to the interaction between the voxl-camera-server and voxl-dfs-server in the midst of a VOA event). Please advise if this is likely the case and how we might handle this.


  • Dev Team

    Can you paste the log from journalctl -u voxl-camera-server here so that we can see exactly what it's saying. Camera server doesn't know or care about what its clients do with the data it provides, it only knows if someone is listening for each camera. These camera server crashes are almost always caused by hardware issues with regards to the camera cables, though it is unusual that the errors you're seeing are coming from different cameras each time. You can also run camera server manually with debug flags voxl-camera-server -d 0 to have it print additional outputs about every single thing that it does. If you can show me a log either from journalctl or a manually run camera server instance that'd be very helpful.


  • Dev Team

    Hi Jared,

    Quick follow up to our call earlier, I've published the latest version of camera server(0.7.5) to stable, this has the increased pipe sizes for the hires camera and will print out the available resolutions for each camera when run with voxl-camera-server -d 1 (or -d 0, though this will be VERY verbose and will start printing each new exposure/gain).



  • Awesome, thank you, @Alex-Gardner. We will try this when we are back in the office and let you know what we find.



  • @Alex-Gardner The available resolutions do print out accordingly, about the same as the previous list you send us. Thanks!



  • Thanks for printing out those resolutions. Very helpful!

    yocto:~$ voxl-camera-server -d 1
    exising instance of voxl-camera-server found, attempting to stop it
    ------ voxl-camera-server INFO: Port J2
    
    ------ voxl-camera-server: Done configuring hires camera
    	 Port       : port_J2
    	 Name       : hires
    	 Enabled    : 1
    	 Type       : hires
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : nv21
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : isp
    
    ------ voxl-camera-server INFO: Port J3
    
    ------ voxl-camera-server: Done configuring stereo camera
    	 Port       : port_J3
    	 Name       : stereo
    	 Enabled    : 1
    	 Type       : stereo
    	 Api        : hal3
    	 P-W        : 1280
    	 P-H        : 480
    	 P-Fmt      : nv21
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 100
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    ------ voxl-camera-server INFO: Port J4
    
    ------ voxl-camera-server: Done configuring tracking camera
    	 Port       : port_J4
    	 Name       : tracking
    	 Enabled    : 1
    	 Type       : tracking
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : raw8
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 100
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    Writing new configuration to /etc/modalai/voxl-camera-server.conf
    Created pipe: hires_preview channel: 0
    Created pipe: stereo channel: 1
    Created pipe: tracking channel: 2
    Starting Camera: tracking
    SUCCESS: Camera module opened
    ----------- Number of cameras: 3
    
    ------ voxl-camera-server INFO: Client: qvio-server0 connected to channel: 2
    ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
    ------ voxl-camera-server ERROR: Client connected to invalid channel: 1
    ----------- HiRes  camera id: 0
    ----------- Stereo camera id: 2
    ----------- Mono   camera id: 1
    ------ voxl-camera-server INFO: Client: voxl-streamer0 connected to channel: 2
    Available resolutions for camera: tracking:
    640 x 480
    320 x 240
    Resolution: 640 x 480 Found!
    ------ voxl-camera-server: Preview buffer allocations
    Internal Exposure for: tracking 2
    /builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface
    Creating ModalAI exposure interface
    	 Port       : port_J4
    	 Name       : tracking
    	 Enabled    : 1
    	 Type       : tracking
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : raw8
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 100
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    Starting Camera: hires
    Available resolutions for camera: hires:
    4160 x 3120
    4056 x 3040
    4000 x 3000
    3040 x 3040
    3016 x 3016
    3840 x 2160
    3648 x 2736
    3264 x 2448
    3200 x 2400
    2976 x 2976
    3044 x 1720
    2704 x 2028
    2704 x 1520
    2592 x 1944
    2688 x 1512
    2028 x 1144
    2160 x 2160
    1920 x 2160
    1920 x 1920
    1880 x 1880
    2048 x 1536
    1920 x 1440
    1920 x 1080
    1600 x 1600
    1600 x 1200
    1520 x 1520
    1440 x 1080
    1080 x 1080
    1280 x 960
    1340 x 760
    1280 x 768
    1280 x 720
    1200 x 1200
    1280 x 640
    1280 x 480
    1040 x 780
    1024 x 768
    960 x 960
    720 x 720
    800 x 600
    960 x 720
    848 x 480
    858 x 480
    864 x 480
    800 x 480
    720 x 480
    640 x 480
    640 x 240
    640 x 360
    480 x 640
    480 x 480
    480 x 360
    480 x 320
    432 x 240
    352 x 288
    320 x 240
    Resolution: 640 x 480 Found!
    ------ voxl-camera-server: Preview buffer allocations
    	 Port       : port_J2
    	 Name       : hires
    	 Enabled    : 1
    	 Type       : hires
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : nv21
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : isp
    
    Starting Camera: stereo
    Available resolutions for camera: stereo:
    1280 x 480
    848 x 480
    858 x 480
    864 x 480
    800 x 480
    720 x 480
    640 x 480
    640 x 240
    640 x 360
    480 x 480
    480 x 360
    480 x 320
    432 x 240
    352 x 288
    320 x 240
    Resolution: 1280 x 480 Found!
    ------ voxl-camera-server: Preview buffer allocations
    Internal Exposure for: stereo 1
    /builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface
    Creating ModalAI exposure interface
    Received raw10 frame, checking to see if is actually raw8
    Frame was actually 8 bit, sending as is
    	 Port       : port_J3
    	 Name       : stereo
    	 Enabled    : 1
    	 Type       : stereo
    	 Api        : hal3
    	 P-W        : 1280
    	 P-H        : 480
    	 P-Fmt      : nv21
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 100
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    ------ voxl-camera-server: Camera server is now running
    

Log in to reply