@Rowan-Dempster ,

We have not tried this recently, but it should work. Here are some tips:

• Use IMX412 camera (M0161 or similar) because it has great image quality and the fastest readout speed of all of our cameras (IMX214 is not recommended for this, it is an old and "slow" camera sensor)
  • faster readout = less rolling shutter skew
• use the latest camera drivers, which max out the camera operating speed in all modes : https://storage.googleapis.com/modalai_public/temp/imx412_test_bins/20250919/imx412_fpv_eis_20250919_drivers.zip
• The readout times are documented here for all modes : https://docs.modalai.com/camera-video/low-latency-video-streaming/#imx412-operating-modes
• for example 1996x1520 (2x2 binned) mode has about 5.5ms readout time, which is pretty short
• QVIO (mvVISLAM.h) has a parameter "readout time", which suggests that it supports rolling shutter. I have not tried it myself, but i heard that it does work.

mvVISLAM_Initialize(...float32_t readoutTime ..)
@param readoutTime
      Frame readout time (seconds). n times row readout time. Set to 0 for 
      global shutter camera.  Frame readout time should be (close to) but 
      smaller than the rolling shutter camera frame period.

Here is where this param is currently set to 0 in voxl-qvio-server : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/main.cpp?ref_type=heads#L371

• in order to correctly use the readout time, you have to ensure that the camera pipeline indeed selects the correct camera mode (for which there is the corresponding readout time) : https://docs.modalai.com/camera-video/low-latency-video-streaming/#how-to-confirm-which-camera-resolution-was-selected-by-the-pipeline
  • also, readout time is printed out by voxl-camera-server when you run it in -d mode (readout time in nanoseconds here):

VERBOSE: Received metadata for frame 86 from camera imx412
VERBOSE: 	Timestamp: 69237313613
VERBOSE: 	Gain: 1575
VERBOSE: 	Exposure: 22769384
VERBOSE: 	Readout Time: 16328284

• keep the exposure low to avoid motion blur (IMX412 has quite a high analog gain, up to 22x and 16x digital gain). If you want to prioritize gain vs exposure, would need to tweak the auto exposure params in camera server (when you get to that point, i can help you)..
• it would be interesting to compare performance against QVIO with AR0144 - that would probably require collecting images from AR0144 and IMX412 (side by side) + IMU data and running QVIO offline with each camera.

Good luck if you try it! let me know if you have any other questions. Please keep in mind that QVIO is based on a closed-source library from Qualcomm and our support of QVIO is limited.

Alex

@mark , thank you for confirming. The error is the same that i am observing. It should not matter whether it is 8 or 12 bit image (in your case it is 8 bit).

In my case the issue happens on M0154 board. I don't think M0054 vs M0154 makes any difference.

Just to clarify what is actually happening..

The issue starts with some CRC errors (interference between multiple cameras, it seems). In my case, the rate of CRC errors depends on the position of the ucoax cables. (you can also watch these CRC errors and move the coax cables for the second tracking camera).

If the error occurs at a critical point (such as beginning of the frame), the camera pipeline reports a critical problem and should produce an error in the camera pipeline.

The correct behavior should be that the problematic camera stops streaming and in latest SDK, it will actually be restarted and stream will recover. However, the camera is never stopped, instead it reports a duplicate image from the other tracking camera.

We are investigating the root cause, as it is somewhere low level in the camera stack (not in voxl-camera-server).

Alex

@clange Yes, definitely. All the support to build a Debian package for installation is in https://github.com/ArduPilot/ardupilot/tree/master/libraries/AP_HAL_QURT/packaging

@mark , i am testing this.. unable to reproduce it with the 8-bit drivers, but i can reproduce the behavior with 12-bit AR0144 drivers.

can you please test again while having dmesg -w running in another terminal (with 8 bit standard AR0144 drivers) and see if you see something like the following when the camera feed starts being duplicated:

thanks!

[ 1765.339349] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1765.939747] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.034093] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.067988] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.068475] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.071001] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.072165] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.100680] CAM_ERR: CAM-ISP: cam_ife_csid_irq: 4903 CSID:3 ERROR_CRC
[ 1766.234702] CAM_INFO: CAM-ISP: cam_ife_csid_halt_csi2: 1931 CSID: 3 cnt: 1 Halt csi2 rx
[ 1766.234714] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 0 CSIPHY index: 0
[ 1766.234719] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 103 PHY base addr=        pK-error offset=0x8b0 size=11
[ 1766.234725] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR0 = 0xc4
[ 1766.234731] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR1 = 0x1
[ 1766.234737] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR2 = 0x0
[ 1766.234742] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR3 = 0x0
[ 1766.234748] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR4 = 0x0
[ 1766.234753] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR5 = 0x0
[ 1766.234758] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR6 = 0x7
[ 1766.234764] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR7 = 0x0
[ 1766.234769] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR8 = 0x60
[ 1766.234775] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR9 = 0x50
[ 1766.234780] CAM_INFO: CAM-CSIPHY: cam_csiphy_status_dmp: 117 CSIPHY0_IRQ_STATUS_ADDR10 = 0xc4
[ 1766.234784] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 1 CSIPHY index: 0
[ 1766.234788] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 2 CSIPHY index: 0
[ 1766.234791] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 3 CSIPHY index: 0
[ 1766.234795] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 4 CSIPHY index: 0
[ 1766.234798] CAM_INFO: CAM-CSIPHY: cam_csiphy_subdev_handle_message: 22 subdev index : 5 CSIPHY index: 0
[ 1766.234813] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4616 idx 3 err 5 phy 0  lane type:0 ln num:1 ln cfg:0x2 cnt 1
[ 1766.234816] cam_csid_evt_bottom_half_handler: 1 callbacks suppressed
[ 1766.234818] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI0: cc0
[ 1766.234822] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI1: 0
[ 1766.234824] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI2: 0
[ 1766.234827] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RDI3: 0
[ 1766.234830] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status TOP: 0
[ 1766.234833] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status RX: 80011
[ 1766.234836] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status IPP: 0
[ 1766.234839] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status PPP: 0
[ 1766.234842] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI0: 0
[ 1766.234844] CAM_ERR: CAM-ISP: cam_csid_evt_bottom_half_handler: 4621 status UDI1: 0

Hi @mark , thanks again for confirming the results.

I will be testing this today and will get back to you.

Alex

@Ashish-Kumar This is very abnormal, I'd recommend submitting an RMA: https://www.modalai.com/pages/rma

@mark ,

Thank you for running the tests!

Regarding installing resistors without knowing what the function is - not recommended . I will look into what this resistor is. Are you saying that after installing this resistor, the issue is not reproducible?

I was going to ask you to do one more test, if you can reproduce the issue. If the issue is indeed at the very low level (the same image is returned for both cameras into voxl-camera-server), then both instances of Auto Exposure algorithm would react to one camera's image changing. So the test would be..

• reproduce the issue to get the same image appear in both camera streams, it seems like tracking_front is the one that is being duplicated
• note the exposure and gain values, reported in the stats below the images
• quickly cover up the front camera (without affecting down-facing camera) and see if the expsure / gain of tracking_down also changes at all.
  • please note that if the image is indeed duplicated at a very low level, the Auto Exposure algorithm's output for the tracking_down will not actually affect the image (since the stream is from tracking_front), but you should still see some changes while AE is converging after a sudden change. If there is no duplication of the image at the camera server, the AE behavior will not change for tracking_down camera when you cover up tracking_front. I hope that makes sense.

Basically i am trying to figure out if this is a camera server issue or somewhere downstream.

Alex

@mark, thank you for the details. It should not matter whether M0054 or M0154 voxl is used.

Can you please do a test and disable both hires cameras and see if the issue is still reproducible? If the issue is gone, then enable one hires camera and test again.

Also, does setting cpu to perf mode help with this? (Voxl-set-cpu-mode perf)

I have never seen this happen with out stable camera configs, such as C28, so this is very odd!

Alex

Hello @Teon,

This is more of a general question, not specifically related to VOXL2. However, here is some information that you might find helpful.

Base station

• Raspberry Pi
• ZED-F9P receiver with dual band helical antenna, connected to RPI via USB
• configure ZED-F9P to be a in base station mode (will require data collection for localizing the base station precisely). Check Ublox docs
• configure ZED-F9P to output RTCM3 corrections over usb to RPI
• run NTRIP caster software (you will need to find one..) which will connect to USB port and serve the RTCM3 data to the clients via NTRIP protocol
  -- this would require an NTRIP client on the client side
• (alternatively) publish the RTCM data via Mavlink directly to the drone (MAVSDK).
• would need a client that would connect to Ublox via USB and send out mavlink messages to your drone
  -- https://mavsdk.mavlink.io/main/en/cpp/api_reference/classmavsdk_1_1_rtk.html
  -- https://github.com/mavlink/MAVSDK/issues/2352

Rover (VOXL2)

• wifi connection to the base station server
• based, on the following docs (https://docs.px4.io/main/en/advanced/rtk_gps) the PX4 gps driver will automatically forward RTCM packets to the attached GPS receiver. The Px4 GPS driver will need to receive the Mavlink GPS_RTCM_DATA packet.
• if you use the NTRIP caster in the step above, you could use a NTRIP client and have the client publish the mavlink GPS_RTCM_DATA message locally on VOXL2 .

I will ask around to see if we have a better documentation for this..

Alex

@mark , we have seen this issue when using 10- or 12-bit drivers for the AR0144 tracking cameras. These drivers are shipped with the SDK but are not stable (for this exact reason).

Can you please confirm which ar0144 sensormodules you are using - you can check in /usr/lib/camera.

Alex