That's a typo on the website we've just corrected. The Starling 2 Max does not include a TOF sensor, and this sensor is not intended for outdoor use.

I spent some time tuning the TOF point cloud processing filter params on Friday to reduce noise. These changes are in the dev branch now: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/919c24bef21cbea2eb7be33853884f5f606a0763

These these changes are likely to help in the outdoor use case too so I suggest giving it a go.

Hello Aks,

I'm afraid there is no clean way in software to rotate the debug images other than copying the image byte-by-byte.

Typically we only use the tof_depth and tof_confidence image pipes for quick debug. voxl-vision-px4 makes real use of the sensor by subscribing to its full point cloud pipe and then projecting the 3D points into space using the known location and orientation of the TOF sensor as specified in /etc/modalai/extrinsics.conf

Best,
James

Hi Matthew,

When you try to run voxl-logger with the hires camera it should have given the following error message:

voxl:/$ voxl-logger --cam hires --samples 1
connected to ch 0 cam server: /run/mpa/hires/
ERROR only support RAW8, RAW16, and FLOAT32 images right now
got NV21 instead

The reason these are the only formats currently supported is because it uses opencv to save/load images. OpenCV is quick enough for black and white images, but simply too slow to be practical for large color images.

As an experiment I just made a new branch of voxl-mpa-tools with a slow but functional switch-case for the NV21 images produced by the hires camera. This will work for saving one or two images, but won't be able to keep up with a steady 30fps stream.

Here is the code if you want to try:
https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-mpa-tools/-/commit/633adfdaf8162548d5df4218243a32e06d2ea7b1

Migrating voxl-logger to use a faster image compression library and/or hardware acceleration is in the pipeline but I can't promise a release date for that I'm afraid.

I hope this helps,
James

@peterkrull

Yes, the yaw estimate will track VIO unless you turn off the Yaw bit in EKF2_EV_CTRL and have EKF2_MAG_TYPE set to 0 (which is it on Starling by default). If you are doing mostly indoor flights you can completely disable the mag by setting EKF2_MAG_TYPE to 5 and SYS_HAS_MAG to 0.

The default EKF2 setup for starling is focused on indoor flight. It enables the magnetometer for calibration and logging but does not use the data. Every time I power on a starling QGC reports 0 degree yaw unless I rotate it. This ekf2 setup is described by the following file that's loaded during the flashing process

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-px4-params/-/blob/master/params/v1.14/EKF2_helpers/indoor_vio.params?ref_type=heads

Note that we are very close to the tip of px4 development and PX4's integration of EKF2 is undergoing a lot of rework right now, including restructuring the EKF2_AID_MASK param into several other EKF2_XYZ_CTRL params.

Please put the CAL_MAG0_ROT rotation parameter back to the default of 0. The driver handles rotating the sensor data from the IC into the frame of reference of the GPS/MAG module, the CAL_MAG0_ROT parameter exits for the case where you mount the gps/mag module in an unusual orientation.

Finally, looking through the logs it seems you experienced a typical vio failure, mostly likely due to lack of visible features on takeoff. Starling is a very small drone with the tracking camera close to the ground, so I recommend trying again by taking off on a feature-rich surface or off of a stand that lets the camera see more of the room during the critical takeoff stage. Once it's up in the air with a good view it will lock in.

This is a known bug that was fixed in voxl-vision-px4 v0.8.8 which is currently on the development repo, scheduled to be merged into the stable repo in a week or so.

The bug is not caused by PX4, it was due the the handling of the specific mavlink packet containing console data. It should behave normally when not using the PX4 console.

Yes, It seems your installed voxl-vision-px4 and libmodal_pipe versions are from SDK 0.8 wheras your voxl-qvio-server is from SDK 0.7, that is the source of the mismatch.

You can upgrade qvio server with opkg install voxl-qvio-server to fix the mismatch quickly but I would recommend just flashing back to the SDK 0.7 platform release since SDK 0.8 does not support TOF on VOXL1. The platform releases are tested as a whole an include a set of packages all known to work together. When individual packages get upgraded instead of the whole SDK together then things tend to break.

Hi Griffin,

DFS stands for depth-from-stereo and is only used on the M500 and Sentinel platforms that have stereo cameras.

Starlings use the PMD TOF camera which outputs 3D point points directly from voxl-camera-server. That data is in the reference frame of the camera such that Z points out the lens, X to the right of the image, and Y pointing downwards.

voxl-vision-hub then consumes that data and rotates it based on the mounting location described in /etc/modalai/extrinsics.conf and the current position/orientation of the drone as reported by VIO.

voxl-mapper is not installed by default as it's not part of the core SDK, however it is locally available and can be installed with apt install voxl-mapper without needing an internet connection.

Best,
James

Models must contain only instructions from a limited set and be quantized properly to run on the GPU. Please refer to the TensorFlow docs for details:

https://www.tensorflow.org/lite/performance/gpu

Hi Steve,

You would likely want your own separate project that uses libmodal_pipe to create the client interface. That can be a systemd service if you want it on boot. MPA encourages lots of separate microservices and projects to all be compiled and run independently to avoid monolithic programs that grow too big and become hard to maintain. Examples such as all the voxl-inspect-*** tools in voxl-mpa-tools also serve as good starting points along with the modal-hello-client example you already found.

I hope this helps,
James

@Kashish-Garg-0

Yes, everything in the aerospace industry including PX4 and ardupilot use the standard NED coordinate frame with Z pointing down. A significant portion of the code in MAVROS seems to be flipping the direction of the Y and Z axis back and forth when converting from mavlink to ros topics.

I suggest using the vvhub_body_wrt_local pipe instead of the qvio pipe as the qvio data is centered around the imu whereas voxl-vision-hub transforms this to be centered around the body (COG) and corrects for gravity alignment.

You can inspect the local pose on VOXL natively with

voxl-inspect-pose vvhub_body_wrt_local
OR
voxl-inspect-pose --local

Note this pipe was called vvpx4_body_wrt_local in SDK0.9 and older.

I don't think ROS libraries like TF2 care which way gravity points. A transform is a transform, you can have it oriented however you prefer when you do your math. You should only need to change coordinate frames for the sake of visualization if RVIS doesn't let you define a default orientation or rotate the ground plane around in the UI.

Hi Griffin,

DFS stands for depth-from-stereo and is only used on the M500 and Sentinel platforms that have stereo cameras.

Starlings use the PMD TOF camera which outputs 3D point points directly from voxl-camera-server. That data is in the reference frame of the camera such that Z points out the lens, X to the right of the image, and Y pointing downwards.

voxl-vision-hub then consumes that data and rotates it based on the mounting location described in /etc/modalai/extrinsics.conf and the current position/orientation of the drone as reported by VIO.

voxl-mapper is not installed by default as it's not part of the core SDK, however it is locally available and can be installed with apt install voxl-mapper without needing an internet connection.

Best,
James

@giladWDS

it looks like very few services are running. If you run voxl-inspect-services you can see what's enabled. You will want to enable voxl-vision-hub, voxl-mavlink-server and voxl-px4

What kind of sensors and data are you dealing with? voxl-vision-hub can subscribe to rangefinder, generic points clouds, and tof data. Since all the logic for compensating for roll/pitch to level out the point cloud data then slice the data into pie-slices to send to PX4 is done already in voxl-vision-hub I would suggest leveraging this. I think the most simple solution would be option 2 and then edit the voxl-vision-hub config file to point to your new server's pipe.

@TOMPLA

Starling 2 Max comes out of the box with PX4's EKF2 configured for GPS outdoor flight.

To fly indoors with VIO only you need to modify PX4's EKF2 parameters in QGC or with the voxl-configure-px4-params utility. In the EKF2_helpers menu, select "indoor_vio.params"

voxl2:/$ voxl-configure-px4-params -w
[INFO] Voxl-px4 service found and active.
Select an option:
[1] Quit
[2] Reset ALL PX4 params to default
[3] experimental_do_not_use
[4] EKF2_helpers
[5] battery_helpers
[6] ci_helpers
[7] voxl2_io_helpers
[8] other_helpers
[9] platforms
[10] radio_helpers
[11] voa_helpers
[12] calibration files
4

Select a EKF2 helper:
[1] Quit
[2] Back
[3] outdoor_flow.params
[4] outdoor_gps.params
[5] outdoor_gps_baro.params
[6] vio_gps_baro.params
[7] indoor_vio.params
[8] indoor_vio_missing_gps.params
7

[INFO] Loading file: /usr/share/modalai/px4_params/v1.14/EKF2_helpers/indoor_vio.params
[INFO] Scanning currently loaded parameters...
The following parameters about to be loaded differ from those currently loaded:
	NAME    		CURRENT    	NEW
[1]	EKF2_MAG_TYPE    	0.0		5.0
[2]	EKF2_EV_CTRL    	0.0		15.0
[3]	EKF2_HGT_REF    	1.0		3.0
[4]	EKF2_GPS_CTRL    	7.0		0.0
[5]	EKF2_EV_QMIN    	0.0		1.0

Would you like to continue with the parameter differences listed above? (y/n)
y

[INFO] Setting params...
[INFO] PX4 parameter configuration successful!
Select an option:
[1] Quit
[2] Reset ALL PX4 params to default
[3] experimental_do_not_use
[4] EKF2_helpers
[5] battery_helpers
[6] ci_helpers
[7] voxl2_io_helpers
[8] other_helpers
[9] platforms
[10] radio_helpers
[11] voa_helpers
[12] calibration files
1
voxl2:/$ 

Then reboot the Starling and PX4 should be configured to use the VIO data.

@LucaVertiq Great work!! I'm going to be OOO for a while but as we discussed before I'm happy to review it when I'm back and integrate into the SDK.

@Jetson-Nano If you are using a stereo-pair that are being published together on the same pipe like on a Seeker, M500, or Sentinel, then you should just need to set up the following config files:

/etc/modalai/voxl-qvio-server.conf:

    "cam_name":     "stereo", (or stereo_front for sentinel)
    "lens_cal_path":        "/data/modalai/opencv_stereo_intrinsics.yml", (or stereo_front_intrinsics for Sentinel)
    "cam_extrinsics_name":  "stereo_l", (or stereo_front_l for sentinel)
    "use_second_cam_if_stereo":     false,

To use the right camera, you can swap "use_second_cam_is_stereo" to true, and qvio will pull in instrinsic lens params from the right camera which is also part of the same intrinsics file. Note that we only have camera locations in the extrinsics files for the left camera, but you can add to /etc/modalai/extrinsics.conf an entry for the right camera if you wish.

@rogerli vibration and lead to the lenses rotating. I often use a dab of hot-glue to hold the lens in place after focusing. On the new Starling 2 and Starling 2 Max drones we now use a clamping ring around the entire camera and lens assembly to lock them in place.

@wilkinsaf voxl-vision-hub's configuration file can be modified to subscribe to multiple sources of relevant data. By default it it set up to consume point cloud data from voxl-dfs-server (stereo cameras on Sentinel and M500), TOF data from PMD TOF sensors (Seeker, Starling, and Starling 2), as well as VL53L1CX mini rangefinders. Data from all concurrent sources are merged together and passed to the autopilot as an obstacle_distance message over mavlink.

Multiple of the VL53L1CX rangefinders can be set up using the open-source voxl-rangefinder-server https://docs.modalai.com/rangefinders/

However, you can integrate any sensor you like by publishing one of the above 3 MPA types that voxl-vision-hub can consume.

@Kris I found and fixed a bug in voxl-camera-server that seems to be the culprit. I flashed my seeker with SDK1.2.0 and sideloaded the following voxl-camera-server with the bugfix to re-enable ingestion of TOF data into voxl-vision-hub

http://voxl-packages.modalai.com/dists/apq8096/dev/binary-arm64/voxl-camera-server_1.9.7_202407111658.ipk

Note VOA simply does not work in the default PX4 acceleration-based position mode so MPC_POS_MODE needs to be set to 0 (velocity control) as done by the voa_enable px4 param helper file here:

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-px4-params/-/blob/dev/params/v1.14/voa_helpers/voa_enable_indoor.params?ref_type=heads

@Jetson-Nano

In SDK 1.2.0 we did start pushing the default config of voxl-mapper a big harder with smaller voxl sizes, longer ray lengths, and double the UI update rate to better complement the newer TOF sensors on the Starling 2. Part of this update was also to lock some of the heavier parts of the mapper processing to cores 4,5,6 to make sure VIO wasn't affected too much. However, if you are still seeing a difference in VIO performance between mapper being on and off then there may still be some optimization to be done.

Just to confirm you are seeing worse VIO performance when mapper is enabled, correct?

If so, please try increasing the voxl_size parameter in /etc/modalai/voxl-mapper.conf from the default 0.1 to 0.25, and set the cpu into performance mode on bootup by changing the normal_cpu_mode field to "performance" in /etc/modalai/voxl-cpu-monitor.conf. The map will be very course and ugly with such a big voxel size but I'm just curious if the power cpu use helps resolve the issue.

For reference, here is the commit history for voxl-mapper showing the recent tweaks for SDK 1.2 and newer that turn up some of the knobs.

Hi @Jetson-Nano

The I2C port on J19 is connected to the Sensors DSP and therefore can't be accessed from the main applications processor.

Here is a new documentation page describing how to hook up and use these sensors.

https://docs.modalai.com/rangefinders/