TOF sensor not working with latest VOXL SW



  • I recently worked through an issue with @Matt-Turi where the voxl-tflite-server was not not working with the latest VOXL SW.

    In that forum post, I was working with the hires + tracking cameras. With voxl-portal, I was able to see the hires, tflite-overlay, tracking, and qvio-overlay images.

    Since I have a TOF sensor, I want to enable and use that sensor as well. Note that a week ago (with older VOXL SW) the VOXL I have was working properly with this sensor.

    I ran voxl-configure-cameras to change the configuration to option 6: Hires + TOF + Tracking. After doing that, the voxl-camera-server crashes. If I restart it, it dies again. If I change the camera configuration to hires, tracking, or hires+tracking, the voxl-camera-server works properly. Something seems amiss with the TOF sensor.

    I ran journalctl -u voxl-camera-server and got this output:

    Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-camera-server.
    Jan 01 00:02:25 apq8096 bash[2130]: ------ voxl-camera-server INFO: Camera server exited gracefully
    Jan 01 00:02:25 apq8096 systemd[1]: Stopped voxl-camera-server.
    Jan 01 00:02:25 apq8096 systemd[1]: Started voxl-camera-server.
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Camera server is now stopping
    Jan 01 00:02:26 apq8096 bash[3798]: There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Stopping tracking camera
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: tracking camera stopped successfully
    Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Stopping tof camera
    Jan 01 00:02:26 apq8096 bash[3798]: Fault address: 0x68
    Jan 01 00:02:26 apq8096 bash[3798]: Address not mapped.
    Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
    Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
    Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m
    Jan 01 00:03:04 apq8096 systemd[1]: Stopped voxl-camera-server.
    Jan 01 00:03:04 apq8096 systemd[1]: Started voxl-camera-server.
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 3
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Camera server is now stopping
    Jan 01 00:03:04 apq8096 bash[4165]: There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Stopping tracking camera
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: tracking camera stopped successfully
    Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Stopping tof camera
    Jan 01 00:03:04 apq8096 bash[4165]: Fault address: 0x68
    Jan 01 00:03:04 apq8096 bash[4165]: Address not mapped.
    Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
    Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
    Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m
    

    The line with the error about the client connecting to an invalid channel: 0 seems relevant. Could this be another SW version mismatch (similar to the forum post mentioned at the outset)?

    Any guidance on how to fix this?


  • Dev Team

    Hey Jared,

    Could you share the output of running:

    voxl-camera-server -d 0
    

    Also, can you confirm that the sensors are plugged into the correct ports per our docs?



  • Here's the output you asked for:

    yocto:/data$ voxl-camera-server -d 0
    ------ voxl-camera-server INFO: Port J2
    
    ------ voxl-camera-server: Done configuring hires camera
    	 Port       : port_J2
    	 Name       : hires
    	 Enabled    : 1
    	 Type       : hires
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : nv21
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : isp
    
    ------ voxl-camera-server INFO: Port J3
    
    ------ voxl-camera-server: Done configuring tof camera
    	 Port       : port_J3
    	 Name       : tof
    	 Enabled    : 1
    	 Type       : tof
    	 Api        : hal3
    	 P-W        : 224
    	 P-H        : 1557
    	 P-Fmt      : blob
    	 FPS        : 15
    	 Tof Mode   : 9
    	 OverrideId : -1
    	 AEAlgo     : isp
    
    ------ voxl-camera-server INFO: Port J4
    
    ------ voxl-camera-server: Done configuring tracking camera
    	 Port       : port_J4
    	 Name       : tracking
    	 Enabled    : 1
    	 Type       : tracking
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : raw8
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 20
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    Writing new configuration to /etc/modalai/voxl-camera-server.conf
    Created pipe: hires channel: 0
    Created pipe: tof channel: 6
    Created pipe: tracking channel: 7
    Starting Camera: tracking
    SUCCESS: Camera module opened
    ----------- Number of cameras: 2
    
    ----------- HiRes  camera id: 0
    ----------- Mono   camera id: 1
    Available resolutions for camera: tracking:
    640 x 480
    320 x 240
    Resolution: 640 x 480 Found!
    ------ voxl-camera-server: Preview buffer allocations
    Internal Exposure for: tracking 7
    /builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface
    Creating ModalAI exposure interface
    Initialize
    Initialize
    	 Port       : port_J4
    	 Name       : tracking
    	 Enabled    : 1
    	 Type       : tracking
    	 Api        : hal3
    	 P-W        : 640
    	 P-H        : 480
    	 P-Fmt      : raw8
    	 FPS        : 30
    	 OverrideId : -1
    	 AEAlgo     : modalai
    =================MODALAI Auto Exposure Settings==================
    gain_min:                        0
    gain_max:                        1000
    exposure_min_us:                 20
    exposure_max_us:                 33000
    desired_msv:                     58.000000
    k_p_ns:                          32000.000000
    k_i_ns:                          20.000000
    max_i:                           250.000000
    p_good_thresh:                   3
    exposure_period:                 1
    gain_period:                     2
    display_debug:                   no
    exposure_offset_for_gain_calc:   8000
    =================================================================
    
    Starting Camera: tof
    
    ------ voxl-camera-server INFO: Camera server is now stopping
    		There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
    
    ------ voxl-camera-server INFO: Stopping tracking camera
    ------ Result thread on camera: tracking recieved stop command, exiting
    ------ voxl-camera-server INFO: tracking camera stopped successfully
    
    ------ voxl-camera-server INFO: Stopping tof camera
    Fault address: 0x68
    Address not mapped.
    Segmentation fault
    

    My VOXL is buried deep in a drone. I'd have to disassemble the drone to view the ports. That said, the cables/connections haven't changed since last week when it was working. I presume they were connected properly then.


  • Dev Team

    It seems that the TOF module is not detected:

    SUCCESS: Camera module opened
    ----------- Number of cameras: 2
    
    ----------- HiRes  camera id: 0
    ----------- Mono   camera id: 1
    

    VOXL is only detecting two cameras, the hires and tracking, which is why those configurations are working correctly and the selected one fails. If the TOF module was setup properly, there would be a third camera listed, as in:

    SUCCESS: Camera module opened
    ----------- Number of cameras: 3
    
    ----------- ToF    camera id: 0
    ----------- Stereo camera id: 2
    ----------- Mono   camera id: 1
    
    

    The TOF module may have been disconnected or a cable could have been damaged since you last tested, as the camera should be detected even if voxl-camera-server is configured incorrectly.



  • I see. 😕

    Looks like I will need to take apart the drone to see if it somehow became disconnected. If that doesn't solve it, I'll try replacing it. I'll post an update (in a few days) once I have done both of those things.

    Thanks again, @Matt-Turi.



  • @Matt-Turi, I check whether any connections were loose. There were none. I re-seated all the connectors anyway, but it made no difference.

    I ended up purchasing a new TOF sensor. After I installed it, it worked! I am able to see the TOF outputs in voxl-portal.

    I have one last issue, which could be related to the VOXL SW, so I'm keeping it in this thread instead of starting another one. After confirming the TOF sensor was working properly in voxl-portal, I wanted to see the pointcloud in rviz.

    I followed the steps described here to view the TOF sensor data in rviz. On a companion linux PC, I would connect to the VOXL over wifi. I could successfully ping going both ways. I ran the appropriate export statements (as described in the TOF sensor User Guide). But when I tried to run rviz or rostopic list, I would get the error message ERROR: Unable to communicate with master!. I tried this with two companion Linux PC's (Ubuntu 18.04 with ROS melodic and Ubuntu 20.04 with ROS noetic) and got the same results on both.

    It seems very much a networking issue. In searching ofther ROS forums, many attribute this error to a firewall being enabled. I've disabled the firewall on both companion Linux PC's but that didn't resolve the issue.

    I'm scratching my head because it appears that everything is fine network-wise, but ROS is unable to connect. It seems very much like a ROS/networking issue (and I still think that is what it is), but I am at the point where I figure it wouldn't hurt to ask the following question: is there anything in the latest ModalAI SW that could be affecting this? If not, is there anything obvious I could be missing? Thanks in advance!



  • I figured out my issue.

    On the VOXL, I had ROS_HOSTNAME=localhost. When I ran unset ROS_HOSTNAME, the companion PC was able to connect properly.


  • Dev Team

    Yep,

    This was an bug with voxl-utils v0.8.4 sourcing ros files in the wrong order, I just pushed a patch (0.8.5) up that fixes this, should be available in a few minutes.


Log in to reply