Posts made by Rowan Dempster

@Alex-Kushleyev Here are the diffs:

• libmodal-pipe: https://gitlab.com/rowan.dempster/libmodal-pipe/-/merge_requests/1/diffs
• voxl-mpa-tools: https://gitlab.com/rowan.dempster/voxl-mpa-tools/-/merge_requests/1/diffs

Thanks for your help!

@Alex-Kushleyev Hi Alex, I compiled libmodal-pipe with ION client-only support in 32 bit builds. I then changed the voxl-mpa-tools toolchain to 32 bit and set the 32 bit toolchain to actually build the tool executables and removed the ones that didn't compile. After deploying those two modified packages to the VOXL2 and running the same repub command that I ran before, I am getting a runtime error while parsing the incoming ION buffers on the client side

voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test                                                                                                                                                 
Subscribed to ION input pipe: top_misp_norm_ion                                                                                                                                                                       
invalid metadata, magic number=85, expected 1229934146                                                                                                                                                                
invalid metadata, magic number=85, expected 1229934146                                                                                                                                                                
invalid metadata, magic number=85, expected 1229934146

Tracking it down, it looks like the call to pipe_client_bytes_in_pipe in the _ion_buf_cb is what's printing the error. If I remove that line (so not returning if pipe_client_bytes_in_pipe returns 0) it prints some garbage (incorrect) metadata and then segfaults:

voxl2:/$ voxl-image-repub top_misp_norm_ion -o top_misp_norm_ion_test
Subscribed to ION input pipe: top_misp_norm_ion
CH0 Got input ion frame size: 10330x-2843, format: STEREO_NV12, stride: 50834, size: 396 bytes

Segmentation fault:
Fault thread: voxl-image-repu(tid: 7900)
Fault address: 0x4
Address not mapped.
Segmentation fault

Note that I didn't recompile voxl-camera-server on-top of the modified libmodal-pipe, don't think that would change anything.

Let me know if you want me to send you my diffs of libmodal-pipe (w.r.t. master) and voxl-mpa-tools (w.r.t. add-new-image-tools).

P.S. I also tried voxl-inspect-ion-stream top_misp_norm_ion (that tool is also 32 bit now) and it's printing garbage too

|         Pipe Name |  fd  |  bytes  | wide |  hgt |exp(ms)| gain | frame id |latency(ms)|  fps |  mbps  |t
| top_misp_norm_ion |    6 |     455 | 3237 |-29847 | 52.43 |-24576 |1448040524 |1955547.7  | 30.0 |    0.6

@Alex-Kushleyev Sounds good, I proceeded with resolving the basic typing issues related to 32 bit monotonic time. Next issue is a linker error that looks a bit more tricky:

-- Build files have been written to: /home/root/build32
[ 10%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o
[ 40%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o
[ 50%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o
[ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o
[ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o
[ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o
[ 90%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o
[100%] Linking CXX shared library libmodal_pipe.so
/opt/sysroots/qrb5165_1/usr/lib/libgbm.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:228: library/libmodal_pipe.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

@Alex-Kushleyev Hi Alex I'm running into compilation errors when building libmodal-pipe after turning the EN_ION_BUF flag ON (builds fine with the flag OFF)

Found voxl-cross version: 4.4
-- ---------------------------------------------------------
-- Using voxl-cross 32-bit toolchain for QRB5165
-- C Compiler     : /usr/bin/arm-linux-gnueabi-gcc-7
-- C++ Compiler   : /usr/bin/arm-linux-gnueabi-g++-7
-- Sysroot        : /opt/sysroots/qrb5165_1
-- C flags        : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4
-- CXX flags      : -isystem=/usr/arm-linux-gnueabi/include -isystem=/usr/include -idirafter /usr/include -fno-stack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv4
-- EXE Link Flags : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 -L/usr/lib
-- SO Link Flags  : -L/opt/sysroots/qrb5165_1/usr/lib32 -L/opt/sysroots/qrb5165_1/lib -L/opt/sysroots/qrb5165_1/usr/lib -L/opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi -L/opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7 /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libgcc_eh.a /opt/sysroots/qrb5165_1/usr/lib/gcc-cross/arm-linux-gnueabi/7/libssp_nonshared.a /opt/sysroots/qrb5165_1/usr/arm-linux-gnueabi/lib/libc_nonshared.a -L/usr/lib
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/arm-linux-gnueabi-gcc-7 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/arm-linux-gnueabi-g++-7 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- INFO: building with ion buf support
-- INFO: building with mavlink support
INFO: Skipping Building Python Bindings
-- INFO: Skipping examples and tools
-- Configuring done (1.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/root/build32
[ 10%] Building C object library/CMakeFiles/modal_pipe.dir/src/start_stop.c.o
[ 30%] Building C object library/CMakeFiles/modal_pipe.dir/src/client.c.o
[ 30%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers.cpp.o
[ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/interfaces.c.o
[ 50%] Building C object library/CMakeFiles/modal_pipe.dir/src/sink.c.o
[ 60%] Building C object library/CMakeFiles/modal_pipe.dir/src/common.c.o
[ 70%] Building C object library/CMakeFiles/modal_pipe.dir/src/misc.c.o
[ 80%] Building C object library/CMakeFiles/modal_pipe.dir/src/server.c.o
[ 90%] Building CXX object library/CMakeFiles/modal_pipe.dir/src/buffers/gbm.cpp.o
/home/root/library/src/server.c: In function 'pipe_server_write_ion_buffer':
/home/root/library/src/server.c:1757:102: error: format '%ld' expects argument of type 'long int', but argument 5 has type 'int64_t {aka long long int}' [-Werror=format=]
         printf("server preparing to send ion buffer id %d (fd %d) to clients, n_clients: %d, time: %ld\n",
                                                                                                    ~~^
                                                                                                    %lld
                ion_buf->buffer_id, fd, c[ch].n_clients, _time_monotonic_ns());
                                                         ~~~~~~~~~~~~~~~~~~~~                          
/home/root/library/src/client.c: In function '_stop_helper_and_remove_pipe':
/home/root/library/src/client.c:1353:56: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         if(en_debug) printf("ch: %d, shutdown socket %ld ns\n", ch, _time_monotonic_ns());
                                                      ~~^            ~~~~~~~~~~~~~~~~~~~~
                                                      %lld
/home/root/library/src/client.c: In function 'pipe_client_close':
/home/root/library/src/client.c:1475:43: error: format '%ld' expects argument of type 'long int', but argument 3 has type 'int64_t {aka long long int}' [-Werror=format=]
         printf("ch: %d, shutdown socket %ld ms\n", ch, _time_monotonic_ns());
                                         ~~^            ~~~~~~~~~~~~~~~~~~~~
                                         %lld
cc1: all warnings being treated as errors
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:93: library/CMakeFiles/modal_pipe.dir/src/client.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cc1: all warnings being treated as errors
make[2]: *** [library/CMakeFiles/modal_pipe.dir/build.make:149: library/CMakeFiles/modal_pipe.dir/src/server.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:106: library/CMakeFiles/modal_pipe.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Seems like I'm going down an uncharted path here. I'm okay with spending my time charting it out since getting QVIO to work with ION pipes will save us critical CPU util. Does ModalAI have plans to switch the official SDK QVIO server release (for qrb5165) to using ION pipes (I saw ModalAI switched to using the MISP pipes for QVIO a few months ago, taking a CPU hit)?

@Alex-Kushleyev Actually had some time today to do the profiling and explore the ION pipes.

I see the same improvements that you quoted:

On 2.2.17
Normal pipes: 45%
MISP pipes: 66%
MISP pipes + 4x tracking pipes: 160%

On perf improvements
Normal pipes: 45%
MISP pipes: 52%
MISP pipes + 4x tracking pipes: 71%

On perf improvements + ION pipes
MISP pipes: 48%
MISP pipes + 4x tracking pipes: 52%

I started looking into using the ION pipe in the QVIO server code but saw this in the voxl-mpa-tools build script:

	qrb5165)
		check_docker "4.3"
		mkdir -p build32
		cd build32
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} \
				-DBUILD_TOOLS=OFF \
				-DEN_ION_BUF=OFF ../
		make -j$(nproc)
		cd ../
		mkdir -p build64
		cd build64
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_64} \
				-DBUILD_TOOLS=ON \
				-DEN_ION_BUF=ON ../
		make -j$(nproc)
		cd ../
		;;

	qrb5165-2)
		check_docker "4.3"
		mkdir -p build
		cd build
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_2_64} \
				-DBUILD_TOOLS=ON \
				-DEN_ION_BUF=OFF ../
		make -j$(nproc)
		cd ../
		;;

Since we're using the qrb5165 (not the qrb5165-2) we have to use the 32bit toolchain to build QVIO (right?):

	qrb5165)
		check_docker "4.4"
		mkdir -p build
		cd build
		cmake -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_QRB5165_1_32} ../
		make -j$(nproc)
		cd ../
		;;

Does this mean that on the qrb5165 (using the 32 bit toolchain), the QVIO server cannot be a client to the ION pipes?

Thank you,
Rowan

@Alex-Kushleyev Hi Alex, just acknowledging your messages, thanks for looking into this and providing us with a path forward. I will have time over the next 2-3 days to test the perf-optimizations branch and familiarize myself the ION based clients. I will update you then.

Hi Modal,

Just following up here, I am going to put our migration to OpenVINS onto the backlog for the time being. Please let me know when the SDK is ready for QVIO -> OpenVINS migration, and when there is bandwidth to assist us with the migration. You can reach me at rowan.dempster@cleorobotics.com.

@Alex-Kushleyev Hi Alex, just following up on any update regarding the shared ION buffers or other methods to work around the CPU hit taken by having many clients to misp based image pipes. Happy to try out any methods/suggestions with our specific clients!

Rowan

@Alex-Kushleyev Glad to hear you are optimistic about this! And thank you, as always, for the tips to get us started. I will hopefully dive into experimenting with this in the next week or so. Do you have a suggestion for which of the hires camera pipes we should use?

Hi Modal team,

Has anyone tried to run QVIO on one of the MPA pipes produced by a hires camera (either IMX214 or 412 I suppose)? Wondering if that's feasible, or if the rolling shutter on those sensors makes feature tracking quite impossible under dynamic movements. Happy to try it out myself if the answer is "unsure"!

Thanks,
Rowan (Cleo Robotics)

Tagging people who might have the most context here: @Cliff-Wong @Clifford-Wong @zauberflote1, any comments / studies on comparative performance, as well tuning routines to get the best performance out of a running OpenVINS on a new airframe?

Perusing posts about OpenVINS from the past year: https://forum.modalai.com/search?matchWords=all&in=titles&showAs=posts&replies=&repliesFilter=atleast&timeFilter=newer&timeRange=&sortBy=topic.lastposttime&sortDirection=desc&term=openvins I found some information about drifting and how to address it. But I'm not sure if that's the most up to date advice, it looks like the module is still under active development. Is the module in a place where it is ready for general customer use? The starling2 drones that ship with SDK1.6 use OpenVINs for localization, not QVIO, correct?

Thank you for any input!

Hi Modal,

Cleo is in the evaluation phase of switching from QVIO to OpenVINS (specifically version https://gitlab.com/voxl-public/voxl-sdk/services/voxl-open-vins-server/-/tags/v0.5.7). As part of this process, I would like to gather the latest information from the VOXL-OpenVINS developers regarding what to expect in terms of relative performance moving from QVIO to OpenVINS. Additionally, are there any platform/airframe specific tuning routines (besides of course cam<->imu extrinsics and cam intrinsics)? We have a very different airframe than the starling https://cleorobotics.com/specifications/

Some general areas that come to mind that we didn't touch on QVIO but might be relevant to OpenVINS:

• IMU static intrinsics (calibrated once)
• IMU dynamic calibration (ie temperature based)
• IMU noise tuning
• IMU<->Camera time synchronization
• Synchronization of 3 camera frames (we have 3 non-overlapping tracking cameras)
• The parameters in the different https://gitlab.com/voxl-public/voxl-sdk/services/voxl-open-vins-server/-/tree/master/misc_files/usr/share/modalai/voxl-open-vins/VoxlConfig?ref_type=heads files, which I noticed were not the same.

I'm sure I'm forgetting other areas that could affect OpenVINS performance, I'm by no means a CV or filter expert Overall, we want our migration to be as seamless as possible and result in the best performance of OpenVINS as possible on our airframe with our sensor suite. Your assistance in this is very much appreciated!

@Alex-Kushleyev Hi Alex, I appreciate you looking into our use case for us (as always!). Please let me know if I can help by providing more details regarding the structure of our clients that are consuming the camera pipes. If there is a detail Cleo can't share publicly on the forum I can always email you as well or hop on a quick call to elaborate.

Understood about the misp pipes, that expensive read access would explain both point #1 and the strange part of point #2 in my original post.

We at Cleo will be monitoring this closely, since CPU usage regressions is pretty much the only gating item for us upgrading our robotic perception stack to SDK1.6. The misp_norm pipes are a great benefit to that perception stack and we'd of course love to take advantage of them as soon as possible. Thus we are definitely open to trying out zero-copy client approaches for keeping CPU usage down, or any other optimizations you could share examples of for us to try out on our use case.

Hi Modal,

As we at Cleo update voxl-camera-server to SDK 1.6 (from SDK 1.0 with lots of backported changes), I've done some preliminary CPU profiling and have some questions on how to keep voxl-camera-server CPU usage down in SDK 1.6. Two things stand out to me:

1. Using the new tracking_misp_norm pipe for our tracking cameras uses ~25% more of a CPU core than using the tracking pipe. 25% is total change across our 3 tracking cameras. So the usage goes from ~85% of a core to ~110% of a core (exact numbers change based on core allocation). To illustrate this you can run voxl-inspect-cam top bottom back hires_small_color tof_depth, record the CPU usage of voxl-camera-server, and then run voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm hires_small_color tof_depth and record the CPU usage again and compare. Is there a way to prevent this from happening, or can you explain why the tracking_misp_norm pipes use more CPU?

2. Adding additional clients to the camera pipe topics causes voxl-camera-server to use more CPU. To illustrate this I only run voxl-camera-server (no other services), then inspect our baseline pipes (e.g. voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm hires_small_color tof_depth) and then in a new terminal(s) I open new clients again using voxl-inspect-cam. For example, when I open 4 new terminals (on top of the baseline terminal) and run voxl-inspect-cam top_misp_norm bottom_misp_norm back_misp_norm in each, then the voxl-camera-server CPU usage increases ~70% (from ~110% of a core, to ~180% of a core). This behavior on its own doesn't quite make sense to me, why would additional clients change the CPU load of the server, and by that much? The images should only be computed once, not per client? Furthermore, what's extra strange is that which tracking pipe is used also matters. If the baseline terminal is instead running voxl-inspect-cam top bottom back hires_small_color tof_depth and the 4 new client terminals are running voxl-inspect-cam top bottom back, then the CPU usage increase only by ~7% (from ~85% to ~92%). The different behavior between type of tracking pipe doesn't make sense to me, could you explain it?

Overall, at Cleo we are looking to use the tracking_misp_norm pipes going forward, and be able to have multiple clients consuming those pipes with having to worry about increasing CPU usage of the voxl-camera-server. If you could comment on whether these asks are possible, or explain why not (either at this time, or never), that would help us a great deal in our process of taking advantage of Modal's great work in robotic perception!

Hi Modal,

When upping the IMU_GYRO_RATEMAX (for lower end-to-end worse case latency in the gyro movement -> actuation loop), I'm seeing the DSP load jump (measured by px4-listener cpuload). After digging into it, it looks like the main culprit is not the processing modules, but rather the FIFO read (transfer((uint8_t *)&buffer, (uint8_t *)&buffer, transfer_size) in the ICM42688P driver) and the UART write (uart_write(&data, sizeof(DataPacket)) != sizeof(DataPacket)) to our STM board (for actuation).

About 40% of the DSP is being used for the FIFO transfer and the UART write with IMU_GYRO_RATEMAX set to 1kHz (1000Hz).

Any recommendations on how to minimize the DSP being choked by these IO operations?

I can only think of marginal gains like decreasing how many bytes the FIFO is configured to contain, or optimizing our DataPacket size down to like 12 bytes (currently 17 bytes). But maybe there is some more fundamental strategy that I'm missing? Or is IO just generally a bad time for the DSP?

Thanks for any advice!

Rowan (Cleo Robotics)

Hey guys any words of advice here?

Hi ModalAI,

We at Cleo have been corresponding with PMD (the ToF supplier) about how to tune the ToF for optimal tradeoff between false positives (floating artifacts) and false negatives (not seeing dark surfaces). One of the main action items from our discussion (which I had ChatGPT summarize below for your reference) is the need to record what PMD calls the "RoyaleRecordingFile" format from the ToF. This will unlock the ability to tune processing parameters on a dataset of pre-recorded files, which makes large scale tuning feasible. Has ModalAI looked into producing the "RoyaleRecordingFile" format from the ToF using the VOXL2?

Thank you,

Rowan

Summary of Email Correspondence on Recording RRF Files on VOXL2

The core discussion revolves around whether it is feasible to record RRF (Royale Recording Format) files on the VOXL2 platform, which is used by Cleo Robotics in conjunction with a 3D Time-of-Flight (ToF) sensor.

1. Motivation: Reducing False Positives and False Negatives

Cleo Robotics has observed conflicting effects from tuning software parameters:

• Increasing exposure time and adjusting detection thresholds sometimes improves detection of dark surfaces but introduces false positives (e.g., floating points or noise on edges).
• Conversely, stricter thresholds reduce false positives but lead to false negatives, especially with low-reflectivity (dark) objects.

This trade-off has motivated the team to find an optimal set of processing parameters that minimizes both FP and FN. To do this effectively, they are seeking a way to record RRF files, which would enable offline experimentation and replay with different tuning values.

2. VOXL2 Limitations and Inquiry to PMD

Rowan Dempster reached out to PMD Technologies to:

• Ask whether RRF files can be recorded directly from the 3D ToF sensor on VOXL2.
• Understand the value of producing RRF files and whether recorded data could be “played back” with alternative parameters for comparative analysis.
• Explore potential development board solutions for this purpose.

3. PMD’s Feedback and ModalAI Software Potential

Tim Werner from PMD explained that:

• The VOXL2 platform does not natively support RRF recording.
• However, the underlying software stack used by ModalAI includes this capability through CAMX, suggesting it is technically feasible.
• Rowan was encouraged to contact ModalAI to request access or support for enabling this feature.
• PMD expressed willingness to assist ModalAI if needed.

4. Cleo’s Proposed Workaround and Implementation Details

Rowan noted:

• They are considering using a separate development board (if available) that supports RRF recording for tuning.
• Once optimal parameters are found, the same could be transferred to the VOXL2 by modifying the 3D ToF driver (e.g., setSpectreParams in VOXL SDK).
• This avoids the repeated burden of recompiling and redeploying firmware for each test case.

Rowan also provided a Google Drive link and GitLab reference to their current tuning approach for PMD’s review.

5. PMD’s Supplemental Suggestions

In addition to the ModalAI pathway, PMD suggested:

• Cleo could share their current parameter values.
• PMD might then provide tuning suggestions to improve detection performance, although this would be less precise than replaying real RRF data.

Conclusion

• Recording RRF files on VOXL2 is not currently supported, but may be possible with cooperation from ModalAI.
• Cleo’s goal is to optimize ToF sensor parameters to reduce both false positives and false negatives, particularly for dark surfaces.
• Next steps involve contacting ModalAI to explore unlocking RRF functionality, with potential support from PMD.
• In parallel, Cleo can continue parameter tuning directly on VOXL2 through driver-level code adjustments, albeit with more effort.

If helpful, we’d be happy to provide a concise technical summary of our current parameter usage or collaborate on getting the RRF feature exposed.

@Alex-Kushleyev Gotcha, thanks for the info Alex and your help resolving this issue!

@Alex-Kushleyev

So, please try the code in the branch (with frame id fix) with different roi sources and let me know how it behaves. It should not matter whether the camera is IMX214 or IMX412, but i have not tried with IMX214. If this branch has the issue with IMX214 based on your testing, i can try it.

Is there any reason to use the preview source? I would prefer to leave that stream disabled since we do not need it for anything except (in the past) the ROI pipe (but now that ROI works on the large video source), and we are already using the large video pipe for recording videos.

Also, if you have not already done so on your branch, please lock voxl-camera-server to use only upper 4 cores, as it significantly improves camera server performance when the load is high (the low level ISP interrupts happen on lower cores only, so this way camera server is not interfering). The reason why i mention this is that you are using an old branch of camera server, so just double checking.

Yes we have the voxl-camera-server locked to 4-7:

[Unit]
Description=voxl-camera-server
SourcePath=/usr/bin/voxl-camera-server
After=voxl-wait-for-fs.service
Requires=voxl-wait-for-fs.service

[Service]
User=root
Type=simple
PIDFile=/run/voxl-camera-server.pid
ExecStart=/usr/bin/taskset -c 4-7 /usr/bin/voxl-camera-server

@Alex-Kushleyev Looking at that branch you sent, I might see why you are having better luck than us. We at Cleo only have the code up to and including add roi transition filter to control command; re-enable publishing of preview...

In green are the commits we have in our codebase, in red are the commits we do not have.

I note that in the commits we do not have you added the option to change the ROI pipe source:

    enum ROI_SOURCE_STREAM {
        ROI_SOURCE_PREVIEW,
        ROI_SOURCE_VIDEO_SMALL,
        ROI_SOURCE_VIDEO_LARGE
    };

and default it to

        roi_source_stream    = ROI_SOURCE_VIDEO_LARGE; //ROI_SOURCE_PREVIEW; //ROI_SOURCE_VIDEO_SMALL // ROI_SOURCE_VIDEO_LARGE

Perhaps in the code you tried yesterday the ROI source is the large encoded stream, whereas with my version of the code the ROI source is the preview stream. I will try to port over those five most recent commits from your work on ROI that we do not have in our code base and hopefully get the ROI pipe to work well using the large encoded stream, and that will serve as a solution (or rather no longer have the need) to get the preview stream working alongside the large encoded stream. Thank you for your help!!