@Sem-Andeweg

I don't work for ModalAI but can try to help.

You need to adjust the IMU0, and IMU1 to body RPY at minimum, which stands for Roll, Pitch, Yaw.

If you look at the Seeker example at https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/master/misc_files/usr/share/modalai/extrinsic_configs/seeker_v1_voxlcam.conf#L64

you'll see the VOXL is rotated 90 degrees "up" and yawed 90 degrees right. From there the tracking camera is then rotated "down" 45 degrees to account for the VOXL CAM being mounted on the front of the Seeker and its orientation relative to the VOXL inside the camera housing.

Hope this helps a bit, definitely watch the video at https://docs.modalai.com/configure-extrinsics/

Is this a unit direct from ModalAI? If it is, it should be tuned or you could load default parameters from https://gitlab.com/voxl-public/flight-core-px4/px4-parameters/-/tree/master/platforms for your version of PX4 and platform.

Then from there you load a helper through QGroundControl parameters page: https://gitlab.com/voxl-public/flight-core-px4/px4-parameters/-/tree/master/helpers

That should get you close to a decent tune.

If this is a custom drone, you need to configure your extrinsics, calibrate your camera, and validate QVIO first before attempting to fly which it looks like you are attempting to do. The VOXL portal qVIO overlay will show you points selected as well as distance traveled which you can validate against manually moving the drone around prior to any attempts to fly. The camera needs to have areas of contrasting features in view -- so reflective floors or some repetitive patterns can be a challenge for qVIO.

https://docs.modalai.com/configure-extrinsics/

https://docs.modalai.com/calibrate-cameras/

Late to the party on this -- but I have been able to build ipk's off of an M1 recently.

I followed: https://docs.modalai.com/install-voxl-docker/

And then also downloaded the image from the developer portal as laid out: https://docs.modalai.com/voxl-cross/

To load:
docker load --input voxl-cross_V1.8.tar

To run:
voxl-docker -i voxl-cross:V1.8

Then the docs in all of the Modal repo's work well.

I do have buildkit etc enabled as laid out here: https://medium.com/nerd-for-tech/developing-on-apple-m1-silicon-with-virtual-environments-4f5f0765fd2f but you shouldn't need any of that. Let me know if that helps you. Worked just fine for me -- no special rosetta2 terminal or Docker version.

I'll see if I can contribute something to the docs when I have a moment.

https://docs.modalai.com/setup-ros-on-voxl-2/

May help you with setting up ROS on VOXL2.

https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-to-ros may also help you bridge the Modal Pipe Architecture into ROS.

Extrinsics and QVIO server configuration will determine camera orientation and which IMU to use respectively.

https://docs.modalai.com/configure-extrinsics/

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/config_file.h#L51

You will need to measure from the IMU (Parent) to child (camera) for translation in meters and also the roll, pitch, and yaw in degrees. I recommend watching the video on the configure extrinsics page.

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-portal/-/blob/master/src/video_manager.cpp#L77

Is an example of allocating a buffer and copying the frame into it. There are some other good examples in that file as well for processing frames and converting them to various formats within the callback. Hope that helps!

@rohitpillai

VOXL2 supports many camera configurations including front and rear stereo, a hires camera, and a tracking camera.

The drone development kits for VOXL2 such as the Sentinel (https://www.modalai.com/products/sentinel-development-drone?variant=40446488805427) and VOXL2 Flight Deck include the tracking camera at 45 degrees. You could adapt this to point straight down.

You could also put any camera on a gimbal or statically using the MIPI connectors for these "standard" cameras ModalAI provides (https://docs.modalai.com/voxl2-camera-configs/) or even interface over USB. https://www.modalai.com/pages/voxl-cameras

https://forum.modalai.com/topic/1328/camera-servers-docker-build-royale-331-spectre-4-7 may have some answers for you on voxl-camera-server which is something ModalAI is working on WRT the build.

In case anyone else comes on this thread --

https://gitlab.com/voxl-public/voxl-sdk/core-libs/libmodal-pipe/-/blob/master/library/include/modal_pipe_interfaces.h#L848 defines the VIO messages within MPA.

float T_imu_wrt_vio[3]; contains the translation in x, y, z and is what is printed in the overlay within VOXL Portal.

• Positive X = Forward
• Positive Y = Left
• Positive Z = Increase in Altitude

If you then look in the Odometry messages used by PX4 or run voxl-inspect-pose these coordinates will be in FRD/NED frame of reference which is:
Positive X = Forward
Positive Y = Right
Positive Z = Decrease Altitude
(I think I've got this right -- hopefully someone from Modal can confirm)

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-vision-px4/-/blob/master/src/vio_manager.c#L350-393 walks through the publication of the odometry MAVLINK message to PX4 and rotation into appropriate frame for processing.

========
TLDR; Most of my issues for this platform were solved by disabling the QVIO server, keeping voxl-vision-px4 running, and dialing in PID for PX4 without any external inputs from vision or GPS. Once tuned, you can focus on calibrating cameras and etc/modalai/extrinsics.conf to ensure consistency in VIO inputs.

Shoutout to @wilkinsaf for tips and tricks on tuning.

I don't work for Modal but,
Questions 1,2,3:

I recommend checking out https://docs.modalai.com/voxl-tflite-server/#voxl-2 and following along with deploying the TF Lite server on your VOXL2.

The source is available here https://gitlab.com/voxl-public/voxl-sdk/services/voxl-tflite-server and includes many built-in models.

Questions 4:
Yes for both cameras, but likely will need some adaptation particularly for thermal.

Zach,

Thanks for the quick response!
I can run both on the dev branch, or roll back to staging.

I thought that voxl-vision-hub would talk to the voxl-mavlink-server which handles all communication with PX4.

From there voxl-mavlink-server forwards between visionhub, px4, and QGC once it connections.

I'll try running the new config tool again to see if that cleans anything up.

Anything else to look for for the bus error?

We were running a VOXL-1 based system off of staging.

Aftern an opkg update and opkg upgrade it looks like the MAVLINK server is broken. We're investigating now, just wanted to get this out there.

voxl:~$ voxl-mavlink-server
loading our own config file
=================================================================
Parameters as loaded from config file:
primary_static_gcs_ip:        192.168.8.10
secondary_static_gcs_ip:      NULL
autopilot_uart_backup_bus:    12
autopilot_uart_bus:           5
autopilot_uart_baudrate:      921600
udp_mtu:                      0
en_external_ap_timesync:      1
=================================================================
Successfully opened primary mavparser on bus 5
Starting primary thread for bus 5
Successfully opened backup mavparser on bus 12
Starting backup thread for bus 12
got first packet on bus 5, stopping the other bus: 12
Adding primary GCS IP address from conf file to list: 192.168.8.10
Added new UDP connection to 192.168.8.10
Not connecting to secondary gcs ip: NULL
Added new UDP connection to 10.147.20.136
CONNECTED to GCS at 10.147.20.136
Init complete, entering main loop
Detected Autopilot Mavlink SYSID 1
thread for bus 12 trying to close
thread for bus 12 Successfully closed
Bus error

If I rollback the mavlink server to previous versions it seems happy but I can't connect to voxl-vision-hub through the mavlink server from something like QGC.

Will be the VOXL2 IMU's.

https://docs.modalai.com/configure-extrinsics/#voxl2 recommends using imu_apps for configuration.

Use cam name = "stereo" and there should be an option for using either the left or right stereo camera.

As Chad pointed out you will need to also configure extrinsics if they aren't already for the stereo camera.
https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/config_file.h#L155

I have a PR to fix an issue with using a stereo camera here with the QVIO Server: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/merge_requests/10 that you will need as well I believe.

Extrinsics and QVIO server configuration will determine camera orientation and which IMU to use respectively.

https://docs.modalai.com/configure-extrinsics/

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/config_file.h#L51

You will need to measure from the IMU (Parent) to child (camera) for translation in meters and also the roll, pitch, and yaw in degrees. I recommend watching the video on the configure extrinsics page.

Yes, you can. You may need to evaluate the performance of both at the same time and make tweaks.

You can change the QVIO server to use a stereo camera if you want in the front, but these are much narrower FOV.

https://docs.modalai.com/configure-extrinsics/ would need to be adjusted to accommodate any camera location changes as well. This is not something supported out of the box with the current VOXL software and would require some work on your part to make it fit your platform and camera configuration.

Resurrecting this topic. We have two drones with identical hardware builds.
Only camera's plugged in are tracking cameras to J4.
Both are running the latest system image.

Drone 1's camera server never crashes while flying Indoors/outdoors.
Drone 2's camera server sometimes crashes and seems to be triggered by exposure changes when outside in bright environments. It is running a slightly newer version of voxl-camera-server.

The failing drone has this in the logs from logcat | grep "camera":

voxl:~$ logcat | grep camera


01-26 20:06:14.004  5732  5732 V HAL     : loaded HAL id=camera path=/usr/lib/hw/camera.msm8996.so hmi=(nil) handle=0x64450
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> get_num_of_cameras: 2767: dev_info[id=0,name='video2']
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> sort_camera_info: 2611: Camera id: 0 facing: 1, type: 1 is_yuv: 0
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> sort_camera_info: 2617: Number of cameras 1 sorted 1
01-26 20:06:14.005  5732  5732 I QCamera : <MCI><INFO> get_num_of_cameras: 2785: num_cameras=1
01-26 20:06:14.006  5732  5732 I QCamera : <HAL><INFO> QCamera2Factory: 142: 1 camera devices detected!
01-26 20:06:14.107  5732  5732 E QCamera : <HAL><ERROR> initStaticMetadata: 9846: static int qcamera::QCamera3HardwareInterface::initStaticMetadata(uint32_t)@9846: from gCamCapability QCAMERA3_EXP_TIME_RANGE= (19425, 1020461640)
01-26 20:06:14.107  5732  5732 E QCamera : <HAL><ERROR> initStaticMetadata: 9877: static int qcamera::QCamera3HardwareInterface::initStaticMetadata(uint32_t)@9877: cameraId 0 raw_dim[0].w 640 raw_dim[0].h 480
01-26 20:06:14.107  5732  5732 I QCamera : <HAL><INFO> getCamInfo: 10272: camera 0 resource cost is 100
01-26 20:06:14.107  5732  5732 I QCamera : <HAL><INFO> cameraDeviceOpen: 414: Open camera id 0 API version 768
01-26 20:06:14.117  5732  5732 I QCamera : <HAL><INFO> openCamera: 851: [KPI Perf]: E PROFILE_OPEN_CAMERA camera id 0
01-26 20:06:14.197  5732  5732 I QCamera : <HAL><INFO> openCamera: 861: [KPI Perf]: X PROFILE_OPEN_CAMERA camera id 0, rc: 0
01-26 20:06:14.197  5732  5732 E QCamera : <HAL><ERROR> initParameters: 8374: int qcamera::QCamera3HardwareInterface::initParameters()@8374 HAL params are initialized


01-26 20:07:55.283  5732  5789 E QCamera : <MCI><ERROR> mm_camera_event_notify: 192: Camera Event DAEMON DIED received
01-26 20:07:55.284  5732  5788 E QCamera : <HAL><ERROR> camEvtHandle: 803: Fatal, camera daemon died

journalctl on the drone with crashed voxl-camera-server shows:

Jan 01 00:00:52 apq8096 systemd[1]: Started voxl-camera-server.
Jan 26 20:01:17 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:23 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:24 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:35 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:37 apq8096 bash[2966]: ERROR:   Recieved "Buffer" error from camera: tracking
Jan 26 20:01:42 apq8096 bash[2966]: ERROR:   Recieved Fatal error from camera: tracking
Jan 26 20:01:42 apq8096 bash[2966]: ERROR:   Sending request 7188, ErrorCode: -ENODEV
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Thread: tracking request thread recieved ESTOP
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Thread: tracking result thread recieved ESTOP
Jan 26 20:01:42 apq8096 bash[2966]: ------ voxl-camera-server INFO: Camera server is now stopping
Jan 26 20:01:42 apq8096 bash[2966]: WARNING: Deleting buffers: 24 of 32 still in use
Jan 26 20:01:42 apq8096 bash[2966]: Segmentation fault:
Jan 26 20:01:42 apq8096 bash[2966]: Fault thread: voxl-camera-ser(tid: 2966)
Jan 26 20:01:42 apq8096 bash[2966]: Fault address: 0x8
Jan 26 20:01:42 apq8096 bash[2966]: Address not mapped.
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Main process exited, code=killed, status=11/SEGV[[0m
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Unit entered failed state.[[0m
Jan 26 20:01:43 apq8096 systemd[1]: [[1;39mvoxl-camera-server.service: Failed with result 'signal'.[[0m

They have not been crashed... and the failure happens regularly when in bright environments. I matched up software versions as well and the crash still happens, so it has to be some kind of hardware issue and the below can be disregarded.

Hardware problems again with expansion or something weird in this specific camera/cable?

Drone 1 package versions (a bit older/stable repo):

libvoxl-cci-direct - 0.1.5
libvoxl-cutils - 0.1.1
voxl-camera-calibration - 0.2.3
voxl-camera-server - 1.3.5
voxl-cpu-monitor - 0.3.0
voxl-docker-support - 1.2.4
voxl-gphoto2-server - 0.0.10
voxl-jpeg-turbo - 2.1.3-4
voxl-libgphoto2 - 0.0.4
voxl-libuvc - 1.0.7
voxl-logger - 0.3.4
voxl-mavlink - 0.1.0
voxl-mavlink-server - 0.2.0
voxl-modem - 0.16.1
voxl-mongoose - 7.7.0-1
voxl-mpa-to-ros - 0.3.6
voxl-mpa-tools - 0.7.6
voxl-opencv - 4.5.5-1
voxl-portal - 0.4.2
voxl-qvio-server - 0.9.0
voxl-streamer - 0.4.1
voxl-suite - 0.9.4
voxl-system-image - 4.0-r0
voxl-tag-detector - 0.0.4
voxl-tflite-server - 0.3.1
voxl-utils - 1.2.2
voxl-uvc-server - 0.1.3
voxl-vision-px4 - 1.4.0
voxl-vpn - 0.0.6

Drone 2 package versions (a bit newer, dev repos):

libvoxl-cci-direct - 0.1.5
libvoxl-cutils - 0.1.1
voxl-camera-calibration - 0.3.0
voxl-camera-server - 1.3.7
voxl-cpu-monitor - 0.4.1
voxl-docker-support - 1.2.5
voxl-gphoto2-server - 0.0.10
voxl-jpeg-turbo - 2.1.3-4
voxl-lepton-server - 1.0.1
voxl-libgphoto2 - 0.0.4
voxl-libuvc - 1.0.7
voxl-logger - 0.3.4
voxl-mavlink - 0.1.0
voxl-mavlink-server - 0.2.0
voxl-modem - 0.16.2
voxl-mongoose - 7.7.0-1
voxl-mpa-to-ros - 0.3.6
voxl-mpa-tools - 0.8.4
voxl-opencv - 4.5.5-1
voxl-portal - 0.4.3
voxl-qvio-server - 0.9.0
voxl-streamer - 0.4.3
voxl-suite - 1.0.0
voxl-system-image - 4.0-r0
voxl-tag-detector - 0.0.4
voxl-tflite-server - 0.3.1
voxl-utils - 1.2.3
voxl-uvc-server - 0.1.3
voxl-vision-px4 - 1.4.1
voxl-vpn - 0.0.6

https://github.com/PX4/PX4-Autopilot/issues/20950

@Sem-Andeweg Just cross-posting here too for anyone that comes back to it. I'm interested in these issues as well.

Sem, (I don't work for modal!)

A few ideas..

Have you tried disabling the QVIO service and adjusting the EKF2_AID_MASK values to ignore the vision inputs across the modes you've tested with PX4?

See how the platform behaves and that should either point to VIO or rule it out. In stabilized mode and without GPS and VIO inputs the platform should be relatively stable.