Posts made by James Strawson

Hi Griffin,

DFS stands for depth-from-stereo and is only used on the M500 and Sentinel platforms that have stereo cameras.

Starlings use the PMD TOF camera which outputs 3D point points directly from voxl-camera-server. That data is in the reference frame of the camera such that Z points out the lens, X to the right of the image, and Y pointing downwards.

voxl-vision-hub then consumes that data and rotates it based on the mounting location described in /etc/modalai/extrinsics.conf and the current position/orientation of the drone as reported by VIO.

voxl-mapper is not installed by default as it's not part of the core SDK, however it is locally available and can be installed with apt install voxl-mapper without needing an internet connection.

Best,
James

@giladWDS

it looks like very few services are running. If you run voxl-inspect-services you can see what's enabled. You will want to enable voxl-vision-hub, voxl-mavlink-server and voxl-px4

What kind of sensors and data are you dealing with? voxl-vision-hub can subscribe to rangefinder, generic points clouds, and tof data. Since all the logic for compensating for roll/pitch to level out the point cloud data then slice the data into pie-slices to send to PX4 is done already in voxl-vision-hub I would suggest leveraging this. I think the most simple solution would be option 2 and then edit the voxl-vision-hub config file to point to your new server's pipe.

@TOMPLA

Starling 2 Max comes out of the box with PX4's EKF2 configured for GPS outdoor flight.

To fly indoors with VIO only you need to modify PX4's EKF2 parameters in QGC or with the voxl-configure-px4-params utility. In the EKF2_helpers menu, select "indoor_vio.params"

voxl2:/$ voxl-configure-px4-params -w
[INFO] Voxl-px4 service found and active.
Select an option:
[1] Quit
[2] Reset ALL PX4 params to default
[3] experimental_do_not_use
[4] EKF2_helpers
[5] battery_helpers
[6] ci_helpers
[7] voxl2_io_helpers
[8] other_helpers
[9] platforms
[10] radio_helpers
[11] voa_helpers
[12] calibration files
4

Select a EKF2 helper:
[1] Quit
[2] Back
[3] outdoor_flow.params
[4] outdoor_gps.params
[5] outdoor_gps_baro.params
[6] vio_gps_baro.params
[7] indoor_vio.params
[8] indoor_vio_missing_gps.params
7

[INFO] Loading file: /usr/share/modalai/px4_params/v1.14/EKF2_helpers/indoor_vio.params
[INFO] Scanning currently loaded parameters...
The following parameters about to be loaded differ from those currently loaded:
	NAME    		CURRENT    	NEW
[1]	EKF2_MAG_TYPE    	0.0		5.0
[2]	EKF2_EV_CTRL    	0.0		15.0
[3]	EKF2_HGT_REF    	1.0		3.0
[4]	EKF2_GPS_CTRL    	7.0		0.0
[5]	EKF2_EV_QMIN    	0.0		1.0

Would you like to continue with the parameter differences listed above? (y/n)
y

[INFO] Setting params...
[INFO] PX4 parameter configuration successful!
Select an option:
[1] Quit
[2] Reset ALL PX4 params to default
[3] experimental_do_not_use
[4] EKF2_helpers
[5] battery_helpers
[6] ci_helpers
[7] voxl2_io_helpers
[8] other_helpers
[9] platforms
[10] radio_helpers
[11] voa_helpers
[12] calibration files
1
voxl2:/$ 

Then reboot the Starling and PX4 should be configured to use the VIO data.

@LucaVertiq Great work!! I'm going to be OOO for a while but as we discussed before I'm happy to review it when I'm back and integrate into the SDK.

@Jetson-Nano If you are using a stereo-pair that are being published together on the same pipe like on a Seeker, M500, or Sentinel, then you should just need to set up the following config files:

/etc/modalai/voxl-qvio-server.conf:

    "cam_name":     "stereo", (or stereo_front for sentinel)
    "lens_cal_path":        "/data/modalai/opencv_stereo_intrinsics.yml", (or stereo_front_intrinsics for Sentinel)
    "cam_extrinsics_name":  "stereo_l", (or stereo_front_l for sentinel)
    "use_second_cam_if_stereo":     false,

To use the right camera, you can swap "use_second_cam_is_stereo" to true, and qvio will pull in instrinsic lens params from the right camera which is also part of the same intrinsics file. Note that we only have camera locations in the extrinsics files for the left camera, but you can add to /etc/modalai/extrinsics.conf an entry for the right camera if you wish.

@rogerli vibration and lead to the lenses rotating. I often use a dab of hot-glue to hold the lens in place after focusing. On the new Starling 2 and Starling 2 Max drones we now use a clamping ring around the entire camera and lens assembly to lock them in place.

@wilkinsaf voxl-vision-hub's configuration file can be modified to subscribe to multiple sources of relevant data. By default it it set up to consume point cloud data from voxl-dfs-server (stereo cameras on Sentinel and M500), TOF data from PMD TOF sensors (Seeker, Starling, and Starling 2), as well as VL53L1CX mini rangefinders. Data from all concurrent sources are merged together and passed to the autopilot as an obstacle_distance message over mavlink.

Multiple of the VL53L1CX rangefinders can be set up using the open-source voxl-rangefinder-server https://docs.modalai.com/rangefinders/

However, you can integrate any sensor you like by publishing one of the above 3 MPA types that voxl-vision-hub can consume.

@Kris I found and fixed a bug in voxl-camera-server that seems to be the culprit. I flashed my seeker with SDK1.2.0 and sideloaded the following voxl-camera-server with the bugfix to re-enable ingestion of TOF data into voxl-vision-hub

http://voxl-packages.modalai.com/dists/apq8096/dev/binary-arm64/voxl-camera-server_1.9.7_202407111658.ipk

Note VOA simply does not work in the default PX4 acceleration-based position mode so MPC_POS_MODE needs to be set to 0 (velocity control) as done by the voa_enable px4 param helper file here:

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-px4-params/-/blob/dev/params/v1.14/voa_helpers/voa_enable_indoor.params?ref_type=heads

@Jetson-Nano

In SDK 1.2.0 we did start pushing the default config of voxl-mapper a big harder with smaller voxl sizes, longer ray lengths, and double the UI update rate to better complement the newer TOF sensors on the Starling 2. Part of this update was also to lock some of the heavier parts of the mapper processing to cores 4,5,6 to make sure VIO wasn't affected too much. However, if you are still seeing a difference in VIO performance between mapper being on and off then there may still be some optimization to be done.

Just to confirm you are seeing worse VIO performance when mapper is enabled, correct?

If so, please try increasing the voxl_size parameter in /etc/modalai/voxl-mapper.conf from the default 0.1 to 0.25, and set the cpu into performance mode on bootup by changing the normal_cpu_mode field to "performance" in /etc/modalai/voxl-cpu-monitor.conf. The map will be very course and ugly with such a big voxel size but I'm just curious if the power cpu use helps resolve the issue.

For reference, here is the commit history for voxl-mapper showing the recent tweaks for SDK 1.2 and newer that turn up some of the knobs.

Hi @Jetson-Nano

The I2C port on J19 is connected to the Sensors DSP and therefore can't be accessed from the main applications processor.

Here is a new documentation page describing how to hook up and use these sensors.

https://docs.modalai.com/rangefinders/

That's a typo on the website we've just corrected. The Starling 2 Max does not include a TOF sensor, and this sensor is not intended for outdoor use.

I spent some time tuning the TOF point cloud processing filter params on Friday to reduce noise. These changes are in the dev branch now: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/919c24bef21cbea2eb7be33853884f5f606a0763

These these changes are likely to help in the outdoor use case too so I suggest giving it a go.

@Jetson-Nano

It's interesting that you see a difference in performance between SDK1.0 and SDK1.1, between those releases was a change in the way IMU timestamps were calculated that now provide more consistent and accurate timestamps which should have improved vio performance.

Do you mind trying SDK 1.1.3 with both the newest qrb5165_imu_server (1.0.1) and the old qrb5165-imu-server (0.6.0) from the old SDK to see if the problem is tied to the IMU server or something else?

http://voxl-packages.modalai.com/dists/qrb5165/sdk-1.0/binary-arm64/qrb5165-imu-server_0.6.0_arm64.deb
http://voxl-packages.modalai.com/dists/qrb5165/sdk-1.1/binary-arm64/qrb5165-imu-server_1.0.1_arm64.deb

Thank you!

@frafrat

Hi. The sensor is integrated into the VOXL SDK and raw data can be viewed through ADB or SSH with the voxl-inspect-rangefinders command line tool. All PX4 Autonomy Dev kids have this sensor. That data is sent from qrb5165-rangefinder-server to PX4 via voxl-mavlink-server. To enable in PX4 set the PX4 EKF2_RNG_CTRL param to 1. You will likely need to fiddle with PX4's other rangefinder params to achieve the behavior you want.

Best,
James

Hi Eric,

The stereo camera pair intrinsics and extrinsics are saved to /data/modalai/opencv_stereo_intrinsics.yaml and /data/modalai/opencv_stereo_extrinsics.yaml

We then use the following opencv calls to generate the rectification maps for each left and right image

cv::Mat R1, P1, R2, P2;
	cv::Mat map11, map12, map21, map22;
	cv::Mat Q; // Output 4×4 disparity-to-depth mapping matrix, to be used later
	double alpha = -1; // default scaling, similar to our "zoom" parameter
	cv::stereoRectify(M1, D1, M2, D2, img_size, R, T, R1, R2, P1, P2, Q, cv::CALIB_ZERO_DISPARITY, alpha, img_size);

	// make sure this uses CV_32_FC2 map format so we can read it and convert to our own
	// MCV undistortion map format, map12 and 22 are empty/unused!!
	cv::initUndistortRectifyMap(M1, D1, R1, P1, img_size, CV_32FC2, map11, map12);
	cv::initUndistortRectifyMap(M2, D2, R2, P2, img_size, CV_32FC2, map21, map22);

Those maps then get converted into a format that can be used by the Qualcomm hardware acceleration blocks for image dewarping, but the fundamental matrices and undistortion coefficients stay the same. I think the above lines is what you are after.

You are correct that the disparity map is centered about the left camera, I believe this is typical in OpenCV land too.

Note that we do not do a precise calibration between the Stereo and Hires Cameras from the factory, depending on how precise you need to alignment to be you may need to use a tool like Kalibr to do that.

@Chase-Riley Glad to hear. Does it also perform as expected at lower values such as 100 or 200? a value as high as 500 will increase latency when only sending small mavlink messages.

0.9765625 = 32/32.768

The ICM42688 will run at roughly 32khz +- a few percent when running off its own internal oscillator. Optionally, you can drive it from a standard 32.768khz crystal which is what we opted to do on VOXL2 so the sample rate is consistent between boards.

@Caio-Licínio-Vasconcelos-Pantarotto said in High Res Snapshot Corrupted:

@Caio-Licínio-Vasconcelos-Pantarotto
Unrelated to this... also noticed a dark ring border effect on the bottom of the image.. is this a common effect for this camera ?
Screenshot 2023-10-10 at 11.49.15.png

We are aware of the issue and are working on it.

between SDK 0.9.5 and SDK 1.0 we disabled the mavlink packet aggregation in voxl-mavlink-server which used to be on by default. This was originally a patch to reduce wifi traffic on VOXL1 + Flight Core systems that forwarded full offboard-rate traffic through WiFi since the implementation of PX4 on VOXL2's DSP only sends slow data rate mavlink telemetry to the ground control station, we didn't find this feature necessary anymore and disabled it by default.

You can re-enable the mavlink aggregation feature with the udp_mtu field in /etc/modalai/voxl-mavlink-server.conf, the JSON file contains the following description of the field:

 * udp_mtu - maximum transfer unit for UDP packets back to GCS. voxl-mavlink-server
 *           will bundle up packets for the GCS into a single UDP packet with 
 *           a maximum size of this. This might help with some radios.
 *           This feature is off by default to let the network stack handle aggregation.
 *           Set to 0 to disable this feature and send one UDP packet per msg.
 *           Set to something like 500 to bundle a handful of packets together.

I suspect doodlelabs might not be well optimized for very small mavlink messages or is not capable of doing its own aggregation. Try a few different values such as 100, 200, 300 etc and see if you get any improvement.

Best,
James

@peterkrull

Yes, the yaw estimate will track VIO unless you turn off the Yaw bit in EKF2_EV_CTRL and have EKF2_MAG_TYPE set to 0 (which is it on Starling by default). If you are doing mostly indoor flights you can completely disable the mag by setting EKF2_MAG_TYPE to 5 and SYS_HAS_MAG to 0.

The default EKF2 setup for starling is focused on indoor flight. It enables the magnetometer for calibration and logging but does not use the data. Every time I power on a starling QGC reports 0 degree yaw unless I rotate it. This ekf2 setup is described by the following file that's loaded during the flashing process

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-px4-params/-/blob/master/params/v1.14/EKF2_helpers/indoor_vio.params?ref_type=heads

Note that we are very close to the tip of px4 development and PX4's integration of EKF2 is undergoing a lot of rework right now, including restructuring the EKF2_AID_MASK param into several other EKF2_XYZ_CTRL params.

Please put the CAL_MAG0_ROT rotation parameter back to the default of 0. The driver handles rotating the sensor data from the IC into the frame of reference of the GPS/MAG module, the CAL_MAG0_ROT parameter exits for the case where you mount the gps/mag module in an unusual orientation.

Finally, looking through the logs it seems you experienced a typical vio failure, mostly likely due to lack of visible features on takeoff. Starling is a very small drone with the tracking camera close to the ground, so I recommend trying again by taking off on a feature-rich surface or off of a stand that lets the camera see more of the room during the critical takeoff stage. Once it's up in the air with a good view it will lock in.

@afdrus

in /etc/modalai/voxl-camera-server.conf you will see the following segment for the stereo pairs,

			"type":	"ov7251",
			"name":	"stereo",
			"enabled":	true,
			"camera_id":	1,
			"camera_id_second":	2,

If you swap the ID numbers for camera_id and camera_id_second it will reverse the left and right cameras in the stereo stream.

Best,
James