@eric
Hey Eric, happy to try and help you with this!
Firstly, voxl-inspect-extrinsics
should be helpful in at least getting some of the transformation matrices for the different cameras with respect to the body. If you need to configure extrinsics, check out the guide here.
When we generate the pointcloud, it's based purely off the values in the disparity map. And we obtain the disparity map from a lower-level API that doesn't explicitly specify which pixel coordinate system it generates the disparity map with respect to. So I dug through the documentation and all I could find was a brief hint that it might be the left image which lines up with your observed outputs.
Your results do look off, though, and I think I might know why: Inside of voxl-dfs-server
we do an undistortion process on the left and right images before computing DFS. If you get points in undistorted space and try to project them back onto a distorted image the result will definitely be a little off.
I understand the need to go from the undistorted space back to distorted space for extrinsics lookup, but I'm not sure I have a great method to do so. One thing you could try -- we undistort our images using a distortion grid specified in:
typedef struct mcv_undistort_config_t{
int width;
int height;
int image_mode;
mcv_cvp_undistort_grid_t* undistort_grid;
/* Unused */
mcvWarpMode warp_mode; ///< currently unused
mcvBorderMode border_mode; ///< currently unused
float transform[9]; ///< currently unused
} mcv_undistort_config_t;
where
/**
* @brief General structure for an undistortion grid
*/
typedef struct mcv_cvp_undistort_grid_t {
double* x;
double* y;
} mcv_cvp_undistort_grid_t;
This undistortion grid gets populated before the DFS process starts, I wonder if you could extract it and somehow invert the result to get a grid that projects from undistorted space back to distorted space?
I apologize, I'm definitely not an expert here. Will forward this thread to James when he gets in, he'll likely have a better insight.
Hope this at least helps somewhat!
Thomas Patton