TOF sensor not working with latest VOXL SW

jaredjohansen

I recently worked through an issue with @Matt-Turi where the voxl-tflite-server was not not working with the latest VOXL SW.

In that forum post, I was working with the hires + tracking cameras. With voxl-portal, I was able to see the hires, tflite-overlay, tracking, and qvio-overlay images.

Since I have a TOF sensor, I want to enable and use that sensor as well. Note that a week ago (with older VOXL SW) the VOXL I have was working properly with this sensor.

I ran voxl-configure-cameras to change the configuration to option 6: Hires + TOF + Tracking. After doing that, the voxl-camera-server crashes. If I restart it, it dies again. If I change the camera configuration to hires, tracking, or hires+tracking, the voxl-camera-server works properly. Something seems amiss with the TOF sensor.

I ran journalctl -u voxl-camera-server and got this output:

Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:02:25 apq8096 bash[2130]: ------ voxl-camera-server INFO: Camera server exited gracefully
Jan 01 00:02:25 apq8096 systemd[1]: Stopped voxl-camera-server.
Jan 01 00:02:25 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Camera server is now stopping
Jan 01 00:02:26 apq8096 bash[3798]: There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Stopping tracking camera
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: tracking camera stopped successfully
Jan 01 00:02:26 apq8096 bash[3798]: ------ voxl-camera-server INFO: Stopping tof camera
Jan 01 00:02:26 apq8096 bash[3798]: Fault address: 0x68
Jan 01 00:02:26 apq8096 bash[3798]: Address not mapped.
Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Jan 01 00:02:26 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m
Jan 01 00:03:04 apq8096 systemd[1]: Stopped voxl-camera-server.
Jan 01 00:03:04 apq8096 systemd[1]: Started voxl-camera-server.
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 3
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Camera server is now stopping
Jan 01 00:03:04 apq8096 bash[4165]: There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Stopping tracking camera
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server ERROR: Client connected to invalid channel: 0
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: tracking camera stopped successfully
Jan 01 00:03:04 apq8096 bash[4165]: ------ voxl-camera-server INFO: Stopping tof camera
Jan 01 00:03:04 apq8096 bash[4165]: Fault address: 0x68
Jan 01 00:03:04 apq8096 bash[4165]: Address not mapped.
Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Jan 01 00:03:04 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m

The line with the error about the client connecting to an invalid channel: 0 seems relevant. Could this be another SW version mismatch (similar to the forum post mentioned at the outset)?

Any guidance on how to fix this?

Hey Jared,

Could you share the output of running:

voxl-camera-server -d 0

Also, can you confirm that the sensors are plugged into the correct ports per our docs?

jaredjohansen

Here's the output you asked for:

yocto:/data$ voxl-camera-server -d 0
------ voxl-camera-server INFO: Port J2

------ voxl-camera-server: Done configuring hires camera
	 Port       : port_J2
	 Name       : hires
	 Enabled    : 1
	 Type       : hires
	 Api        : hal3
	 P-W        : 640
	 P-H        : 480
	 P-Fmt      : nv21
	 FPS        : 30
	 OverrideId : -1
	 AEAlgo     : isp

------ voxl-camera-server INFO: Port J3

------ voxl-camera-server: Done configuring tof camera
	 Port       : port_J3
	 Name       : tof
	 Enabled    : 1
	 Type       : tof
	 Api        : hal3
	 P-W        : 224
	 P-H        : 1557
	 P-Fmt      : blob
	 FPS        : 15
	 Tof Mode   : 9
	 OverrideId : -1
	 AEAlgo     : isp

------ voxl-camera-server INFO: Port J4

------ voxl-camera-server: Done configuring tracking camera
	 Port       : port_J4
	 Name       : tracking
	 Enabled    : 1
	 Type       : tracking
	 Api        : hal3
	 P-W        : 640
	 P-H        : 480
	 P-Fmt      : raw8
	 FPS        : 30
	 OverrideId : -1
	 AEAlgo     : modalai
=================MODALAI Auto Exposure Settings==================
gain_min:                        0
gain_max:                        1000
exposure_min_us:                 20
exposure_max_us:                 33000
desired_msv:                     58.000000
k_p_ns:                          32000.000000
k_i_ns:                          20.000000
max_i:                           250.000000
p_good_thresh:                   3
exposure_period:                 1
gain_period:                     2
display_debug:                   no
exposure_offset_for_gain_calc:   8000
=================================================================

Writing new configuration to /etc/modalai/voxl-camera-server.conf
Created pipe: hires channel: 0
Created pipe: tof channel: 6
Created pipe: tracking channel: 7
Starting Camera: tracking
SUCCESS: Camera module opened
----------- Number of cameras: 2

----------- HiRes  camera id: 0
----------- Mono   camera id: 1
Available resolutions for camera: tracking:
640 x 480
320 x 240
Resolution: 640 x 480 Found!
------ voxl-camera-server: Preview buffer allocations
Internal Exposure for: tracking 7
/builds/voxl-public/modal-pipe-architecture/voxl-camera-server/src/expgain_interface/expgain_interface_factory.cpp 45 Creating exposure interface
Creating ModalAI exposure interface
Initialize
Initialize
	 Port       : port_J4
	 Name       : tracking
	 Enabled    : 1
	 Type       : tracking
	 Api        : hal3
	 P-W        : 640
	 P-H        : 480
	 P-Fmt      : raw8
	 FPS        : 30
	 OverrideId : -1
	 AEAlgo     : modalai
=================MODALAI Auto Exposure Settings==================
gain_min:                        0
gain_max:                        1000
exposure_min_us:                 20
exposure_max_us:                 33000
desired_msv:                     58.000000
k_p_ns:                          32000.000000
k_i_ns:                          20.000000
max_i:                           250.000000
p_good_thresh:                   3
exposure_period:                 1
gain_period:                     2
display_debug:                   no
exposure_offset_for_gain_calc:   8000
=================================================================

Starting Camera: tof

------ voxl-camera-server INFO: Camera server is now stopping
		There is a chance that it may segfault here, this is a mmqcamera bug, ignore it

------ voxl-camera-server INFO: Stopping tracking camera
------ Result thread on camera: tracking recieved stop command, exiting
------ voxl-camera-server INFO: tracking camera stopped successfully

------ voxl-camera-server INFO: Stopping tof camera
Fault address: 0x68
Address not mapped.
Segmentation fault

My VOXL is buried deep in a drone. I'd have to disassemble the drone to view the ports. That said, the cables/connections haven't changed since last week when it was working. I presume they were connected properly then.

It seems that the TOF module is not detected:

SUCCESS: Camera module opened
----------- Number of cameras: 2

----------- HiRes  camera id: 0
----------- Mono   camera id: 1

VOXL is only detecting two cameras, the hires and tracking, which is why those configurations are working correctly and the selected one fails. If the TOF module was setup properly, there would be a third camera listed, as in:

SUCCESS: Camera module opened
----------- Number of cameras: 3

----------- ToF    camera id: 0
----------- Stereo camera id: 2
----------- Mono   camera id: 1

The TOF module may have been disconnected or a cable could have been damaged since you last tested, as the camera should be detected even if voxl-camera-server is configured incorrectly.

jaredjohansen

I see.

Looks like I will need to take apart the drone to see if it somehow became disconnected. If that doesn't solve it, I'll try replacing it. I'll post an update (in a few days) once I have done both of those things.

Thanks again, @Matt-Turi.

jaredjohansen

@Matt-Turi, I check whether any connections were loose. There were none. I re-seated all the connectors anyway, but it made no difference.

I ended up purchasing a new TOF sensor. After I installed it, it worked! I am able to see the TOF outputs in voxl-portal.

I have one last issue, which could be related to the VOXL SW, so I'm keeping it in this thread instead of starting another one. After confirming the TOF sensor was working properly in voxl-portal, I wanted to see the pointcloud in rviz.

I followed the steps described here to view the TOF sensor data in rviz. On a companion linux PC, I would connect to the VOXL over wifi. I could successfully ping going both ways. I ran the appropriate export statements (as described in the TOF sensor User Guide). But when I tried to run rviz or rostopic list, I would get the error message ERROR: Unable to communicate with master!. I tried this with two companion Linux PC's (Ubuntu 18.04 with ROS melodic and Ubuntu 20.04 with ROS noetic) and got the same results on both.

It seems very much a networking issue. In searching ofther ROS forums, many attribute this error to a firewall being enabled. I've disabled the firewall on both companion Linux PC's but that didn't resolve the issue.

I'm scratching my head because it appears that everything is fine network-wise, but ROS is unable to connect. It seems very much like a ROS/networking issue (and I still think that is what it is), but I am at the point where I figure it wouldn't hurt to ask the following question: is there anything in the latest ModalAI SW that could be affecting this? If not, is there anything obvious I could be missing? Thanks in advance!

jaredjohansen

I figured out my issue.

On the VOXL, I had ROS_HOSTNAME=localhost. When I ran unset ROS_HOSTNAME, the companion PC was able to connect properly.

Yep,

This was an bug with voxl-utils v0.8.4 sourcing ros files in the wrong order, I just pushed a patch (0.8.5) up that fixes this, should be available in a few minutes.