Recording RoyaleRecordingFile format from the ToF

[Rowan Dempster]

Hi ModalAI,

We at Cleo have been corresponding with PMD (the ToF supplier) about how to tune the ToF for optimal tradeoff between false positives (floating artifacts) and false negatives (not seeing dark surfaces). One of the main action items from our discussion (which I had ChatGPT summarize below for your reference) is the need to record what PMD calls the "RoyaleRecordingFile" format from the ToF. This will unlock the ability to tune processing parameters on a dataset of pre-recorded files, which makes large scale tuning feasible. Has ModalAI looked into producing the "RoyaleRecordingFile" format from the ToF using the VOXL2?

Thank you,

Rowan

Summary of Email Correspondence on Recording RRF Files on VOXL2

The core discussion revolves around whether it is feasible to record RRF (Royale Recording Format) files on the VOXL2 platform, which is used by Cleo Robotics in conjunction with a 3D Time-of-Flight (ToF) sensor.

---

1. Motivation: Reducing False Positives and False Negatives

Cleo Robotics has observed conflicting effects from tuning software parameters:

• Increasing exposure time and adjusting detection thresholds sometimes improves detection of dark surfaces but introduces false positives (e.g., floating points or noise on edges).
• Conversely, stricter thresholds reduce false positives but lead to false negatives, especially with low-reflectivity (dark) objects.

This trade-off has motivated the team to find an optimal set of processing parameters that minimizes both FP and FN. To do this effectively, they are seeking a way to record RRF files, which would enable offline experimentation and replay with different tuning values.

---

2. VOXL2 Limitations and Inquiry to PMD

Rowan Dempster reached out to PMD Technologies to:

• Ask whether RRF files can be recorded directly from the 3D ToF sensor on VOXL2.
• Understand the value of producing RRF files and whether recorded data could be “played back” with alternative parameters for comparative analysis.
• Explore potential development board solutions for this purpose.

---

3. PMD’s Feedback and ModalAI Software Potential

Tim Werner from PMD explained that:

• The VOXL2 platform does not natively support RRF recording.
• However, the underlying software stack used by ModalAI includes this capability through CAMX, suggesting it is technically feasible.
• Rowan was encouraged to contact ModalAI to request access or support for enabling this feature.
• PMD expressed willingness to assist ModalAI if needed.

---

4. Cleo’s Proposed Workaround and Implementation Details

Rowan noted:

• They are considering using a separate development board (if available) that supports RRF recording for tuning.
• Once optimal parameters are found, the same could be transferred to the VOXL2 by modifying the 3D ToF driver (e.g., setSpectreParams in VOXL SDK).
• This avoids the repeated burden of recompiling and redeploying firmware for each test case.

Rowan also provided a Google Drive link and GitLab reference to their current tuning approach for PMD’s review.

---

5. PMD’s Supplemental Suggestions

In addition to the ModalAI pathway, PMD suggested:

• Cleo could share their current parameter values.
• PMD might then provide tuning suggestions to improve detection performance, although this would be less precise than replaying real RRF data.

---

Conclusion

• Recording RRF files on VOXL2 is not currently supported, but may be possible with cooperation from ModalAI.
• Cleo’s goal is to optimize ToF sensor parameters to reduce both false positives and false negatives, particularly for dark surfaces.
• Next steps involve contacting ModalAI to explore unlocking RRF functionality, with potential support from PMD.
• In parallel, Cleo can continue parameter tuning directly on VOXL2 through driver-level code adjustments, albeit with more effort.

If helpful, we’d be happy to provide a concise technical summary of our current parameter usage or collaborate on getting the RRF feature exposed.

[Rowan Dempster]

Hey guys any words of advice here?

[Alex Kushleyev]

@Rowan-Dempster ,

Here is where the raw data from the TOF sensor comes into voxl-camera-server : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/hal3_camera_mgr.cpp?ref_type=heads#L1242

The data is in the MIPI12 packed format. The TOF library expects RAW16 format, so the data is converted from MIPI12 to RAW16 and fed into the TOF processing library. It is possible that dumping the RAW data is what RRF recording would be doing, so you could add that functionality to camera server to write the raw data to file. I could also help with publishing the RAW12 data as is to a mpa channel, so you can log it outside of the camera server (for example using voxl-record-raw-image utility. You could clarify with PMD if that would work (in other words, you would need to know the log format of the RRF recording)

We have not tried enabling the raw data logging. However, if you know what function needs to be called within PMD library (maybe you just enable logging, and it does everything for you), you should be able to add it somewhere in the TOF init code, maybe here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/tof_interface.cpp?ref_type=heads#L982

You can find the API header files for Royale / Spectre in /usr/include/royale either on VOXL2 directly on in /opt/sysroots/qrb5165_1/usr/include/royale the voxl-cross:V4.4 docker.

Hopefully that helps!

Alex