ModalAI Forum

Hi Alex,

Thank you for the reply!

I've attached the stereo camera view, as well as disparity images. I think it shows enough features, but now I'm wondering why the distances array that gets calculated in voxl-vision-px4 sends out false positives, stating that there's an obstacle between 1.6m and 1.9m ahead, when clearly there isn't. This is what I see in terms of distances generated from the stereo camera view I have attached below:

Disparity raw (image saved by voxl-dfs-server -s):

Disparity screenshots from web portal:
scaled:

raw:

Following is my camera calibration on unit M20000005FW:

Extrinsics:

%YAML:1.0
---
R: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 9.9766949807073035e-01, 3.2726419179402305e-02,
       5.9871145863353382e-02, -3.3135402553510745e-02,
       9.9943374643317273e-01, 5.8507767150416416e-03,
       -5.9645768642156684e-02, -7.8209959881417272e-03,
       9.9818896723257755e-01 ]
T: !!opencv-matrix
   rows: 3
   cols: 1
   dt: d
   data: [ -8.0702806185923404e-02, 1.3414138137096483e-03,
       7.4901499926949659e-03 ]
reprojection_error: 3.1717501660388558e-01
calibration_time: "2021-11-28 23:55:40"

Intrinsics:

%YAML:1.0
---
M1: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 4.9801509489639710e+02, 0., 3.4535141859544910e+02, 0.,
       4.9660210124907837e+02, 2.5153003783439996e+02, 0., 0., 1. ]
D1: !!opencv-matrix
   rows: 5
   cols: 1
   dt: d
   data: [ -1.7398718943898814e-01, 6.8113519993759170e-02,
       -7.1388044381347557e-05, -2.4636080589428406e-03, 0. ]
reprojection_error1: 3.9142503079362151e-01
M2: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 4.9189021103330248e+02, 0., 2.5268459489748463e+02, 0.,
       4.9027463109482443e+02, 2.3132872578584897e+02, 0., 0., 1. ]
D2: !!opencv-matrix
   rows: 5
   cols: 1
   dt: d
   data: [ -1.6866195998342057e-01, 4.2438171238534744e-02,
       1.8663371564379684e-04, 3.5401519495983939e-03, 0. ]
reprojection_error2: 4.0183396708155583e-01
width: 640
height: 480
distortion_model: plumb_bob
calibration_time: "2021-11-28 23:55:40"

Thanks again!

@Alex-Gardner Hi Alex, we still haven't had any luck getting this to work, and our last resort is to create our own OpenCV environment and attempt to regenerate the depth pointcloud, as a means to determine where the issue lies.
Our impression was that this feature worked out of the box, but we have 4 vehicles that are facing the same problem, so we're wondering if it's a simple configuration change, a bug in the system, or simply the wrong versions of code deployed.

@Chad-Sweet I referenced the forum post linked here, and after some more investigation I found that it's not possible to add an extra tracking OV7251 camera onto the system. However, I'm wondering if it's possible to add a different model of camera alongside the stereo and tracking cameras

@Alex-Gardner Hi Alex, any updates on this? Am I able to see a valid point cloud when everything is working properly? Or are my results the best version of the point cloud I'll be getting?

That makes sense, thank you! We will try to tweak the exposure settings before making major changes to the unit.

Another idea we were discussing that kind of relates back to our problem, is to add another tracking camera to the voxl system for redundancy. Technically and physically speaking, is there anything that would stop us from achieving that?
I'm thinking we would need to run another instance of voxl-vision-px4 and create a component that will consolidate the results of both, falling back on one if another fails.

Any thoughts on that?

We are using the M500 in poorly-lit hallways with a dark void. Even though we've installed lighting on the drone, we still weren't able to get a stable position control status, it kept falling back to stabilized mode. If we fly the drone sideways facing the walls of the hallway, it flies much better (obviously, since it gets more odometry that way). So we're thinking of possible solutions:

• Rotate the tracking camera 90 degrees to face the walls of the hallways instead of the front, or move it to the side. That would keep the stereo cameras facing forward for offboard mode.
• Switch the RC channels so that our forward is the drone's side, but that means the stereo cameras would also face the side. This wouldn't allow us to use offboard mode, so we would either need to move the stereo cameras to the front, or use a little LIDAR instead.

Any thoughts or technical notes on the previous points?

Update

I was able to recalibrate the cameras, but the proximity data still seems off. Here's what I'm seeing:

Note that the white pointcloud is the DFS disparity pointcloud. The red points are the points that were considered the closest to the drone when calculating the closest obstacle in voa_manager. I'm simply doing the following:

// calculate distance in cm to send to px4
double x = T_point_wrt_level.d[0];
double y = T_point_wrt_level.d[1];
double z = T_point_wrt_stereo.d[2];

.....

// record if this point is closer than previously detected obstacle in this bin
if(dist_cm < distances[bin]) {
	 distances[bin] = dist_cm;

	if(send_linescan) {
		point_cloud_points[bin] = x;
		point_cloud_points[bin+1] = y;
		point_cloud_points[bin+2] = z;
	}
}

And then sending the point_cloud_points data over a new pipe that gets recognised automatically by voxl_mpa_to_ros

Any thoughts?

Hi Alex,

Thank you for your response! I tried 2 things:

1. I attempted to calibrate the stereo extrinsics, and got the following error:

voxl-calibrate-camera stereo -s 9x6 -l 0.029 -e
Sampling overlapping images now
Enter an empty line to take a sample or "stop" to finish sampling

Taking Frame

Taking Frame
stop
Stopping sampling
Pulling existing intrinsics from: /data/modalai/opencv_stereo_intrinsics.yml

Matrix L
[497.7006450187309, 0, 295.8446445528053;
 0, 496.3585448760351, 231.712917372481;
 0, 0, 1]
Distortion L
[-0.1865369023976547;
 0.1016650425512336;
 0.0001161942987197076;
 0.004382065643093006;
 0]
Matrix R
[490.0228530745731, 0, 373.8964558150473;
 0, 489.1449020535995, 247.3517837095496;
 0, 0, 1]
Distortion R
[-0.2086578047150433;
 0.151676525578168;
 -0.001749483285398663;
 -0.00269335419013983;
 0]

Calibrating Extrinsics
2 frames will be processed
R
[0.9913194969599594, 0.03829151497302103, -0.1257752552298109;
 -0.04342180569409582, 0.9983222939882923, -0.038303317300436;
 0.1240975492800618, 0.04343221393188008, 0.9913190410032776]
T
[-0.1452994050346482;
 -0.007246953931032494;
 -0.0001531663907812685]
Re-projection error reported by stereoCalibrate: 5.086737
Extrinsics Calibration Failed
Max reprojection error:  0.5

Exiting Cleanly

Here are my intrinsics:

%YAML:1.0
---
M1: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 4.9770064501873094e+02, 0., 2.9584464455280528e+02, 0.,
       4.9635854487603513e+02, 2.3171291737248100e+02, 0., 0., 1. ]
D1: !!opencv-matrix
   rows: 5
   cols: 1
   dt: d
   data: [ -1.8653690239765469e-01, 1.0166504255123363e-01,
       1.1619429871970763e-04, 4.3820656430930060e-03, 0. ]
M2: !!opencv-matrix
   rows: 3
   cols: 3
   dt: d
   data: [ 4.9002285307457311e+02, 0., 3.7389645581504732e+02, 0.,
       4.8914490205359948e+02, 2.4735178370954961e+02, 0., 0., 1. ]
D2: !!opencv-matrix
   rows: 5
   cols: 1
   dt: d
   data: [ -2.0865780471504333e-01, 1.5167652557816802e-01,
       -1.7494832853986628e-03, -2.6933541901398303e-03, 0. ]
distortion_model: plumb_bob
width: 640.
height: 480.

2. I also played with the dfs configs some more and looked at the depth pointcloud again, and I know this might be fixed when calibration is done, but it seems very wrong as there are way too many points close to the drone.

Here's my updated dfs config:

{
	"skip_n_frames":	5,
	"n_disparity_levels":	24,
	"sobel_cutoff":	5,
	"first_median_filter_size":	7,
	"en_second_median_filter":	false,
	"second_median_filter_size":	7
}

Here's what I'm seeing:

Any thoughts on this?

Thanks!

I'm trying to work back on why our collision prevention does not work as expected, and I noticed that we're not getting back a valid depth from stereo point cloud.

I've saved some more images from DFS server, that I've attached here. The weird thing is that I can see disparity points in the raw image, but they all seem to get filtered out, leaving little to no features to measure. I'm wondering how normal that is, what I'm expected to see in disparity images, and whether my environment is in good condition for this experiment.

I can follow up with the ROS pointcloud recording for your reference, but they're basically all inf or -inf values as such:

4684.26830914,-inf,inf,10000.0

Thanks in advance

Stereo right:

Stereo left:

Disparity raw:

Disparity scaled after blur:

Disparity filtered after blur:

Seemed like a weird fluke... I reinstalled the system image + voxl suite this time from the install script, then removed the packages individually and reinstalled my built versions, and it worked...

Thank you though!

Lynn

I installed the latest voxl-suite and system image 3.3, and I noticed that voxl-dfs-server is failing to run. I'm getting the following output:

yocto:/# opkg list-installed | grep opencl
opencl - 0.0.2
yocto:/# systemctl status voxl-dfs-server -l
● voxl-dfs-server.service - voxl-dfs-server
   Loaded: loaded (/usr/bin/voxl-dfs-server; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 1970-01-01 00:00:07 UTC; 51 years 9 months ago
  Process: 2114 ExecStart=/usr/bin/voxl-dfs-server (code=exited, status=127)
 Main PID: 2114 (code=exited, status=127)

Jan 01 00:00:07 apq8096 systemd[1]: Started voxl-dfs-server.
Jan 01 00:00:07 apq8096 systemd[1]: voxl-dfs-server.service: Main process exited, code=exited, status=127/n/a
Jan 01 00:00:07 apq8096 systemd[1]: voxl-dfs-server.service: Unit entered failed state.
Jan 01 00:00:07 apq8096 systemd[1]: voxl-dfs-server.service: Failed with result 'exit-code'.
Jan 01 00:00:08 apq8096 voxl-dfs-server[2114]: /usr/bin/voxl-dfs-server: error while loading shared libraries: libOpenCL.so: cannot open shared object file: No such file or directory
yocto:/# 

any thoughts as to what could be the problem?

Thanks

I resolved the problem by installing openmp from the voxl-packages site.

Re: Installing OpenCV-3.4.6.ipk fails on VOXL Emulator Docker Image (1.0.0)

I tried following the instructions listed here/in the voxl-opencv README, and I get an error stating that:

satisfy_dependencies_for: Cannot satisfy the following dependencies for voxl-opencv:
 * 	openmp * 
 * opkg_install_cmd: Cannot install package voxl-opencv.

I'm building voxl-opencv locally using VOXL Cross.

Any ideas on what I could try doing to resolve this?

Thanks

@Liping I'm also curious as to where you got voxl_imu from... I don't think I've ever seen that. The correct package is voxl-imu-server and you can find it here. You would have to uninstall voxl_imu

@Liping I just realized that you don't have voxl-camera-server installed. You will need that. I also found that having both voxl-camera-server and voxl-cam-manager has given me some issues so I would recommend uninstalling voxl-cam-manager

@Liping that's a curious problem... try checking the status of the imu and camera servers:
systemctl status voxl-camera-server
systemctl status voxl-imu-server

If you check the status of voxl-qvio again, does it say it failed? or it waits forever?

You should have the following calibration files:

etc/snav/calibration.stereo.xml
home/root/.ros/camera_info/left.yaml
home/root/.ros/camera_info/right.yaml
etc/snav/calibration.downward.xml
home/root/.ros/camera_info/downward.yaml

But I'm pretty sure you're going to need

/data/modalai/opencv_stereo_intrinsics.yml
/data/modalai/opencv_stereo_extrinsics.yml

I got those by running the calibration process you referenced.

@Liping Is voxl-qvio actually running? You can check by calling systemctl status voxl-qvio-server it might be that you need calibration files for the tracking camera

@Liping My issue was resolved by installing voxl-qvio. You can reference the diagram here to learn what services are needed for the different interactions.

I'm on the latest version of voxl-dfs-server, and I'm attempting to get collision prevention working, but I went through the code and it seems like the disparity computation is resulting in an empty matrix. In result, it's detecting no objects.

I've also tried placing a board infront of the camera and moved it around, but no luck.

I've attached the dsp image, as well as the input from the stereo cameras. I've calibrated the cameras today.

Thanks!

@Alex-Kushleyev I see. It works and I was able to get the calibration files so it's all good!

Thanks!