Topics created by Rowan Dempster

@Rowan-Dempster , you should use a monochrome stream (_grey), since QVIO needs a RAW8 image.

If you are not using MISP on hires cameras, that is fine, you can start off using the output of the ISP.

You should calibrate the camera using whatever resolution you decide to try. This is to avoid any confusion, since if you using ISP pipeline, the camera pipeline may select a higher resolution and downscale + crop. So whenever you are changing resolutions, it is always good to do a quick camera calibration to confirm the camera parameters.

When using MISP, we have more control over which camera mode is selected, because MISP gets the RAW data, not processed by the ISP, so we know the exact dimensions of the image sent from camera.

Alex

Tagging people who might have the most context here: @Cliff-Wong @Clifford-Wong @zauberflote1, any comments / studies on comparative performance, as well tuning routines to get the best performance out of a running OpenVINS on a new airframe?

Perusing posts about OpenVINS from the past year: https://forum.modalai.com/search?matchWords=all&in=titles&showAs=posts&replies=&repliesFilter=atleast&timeFilter=newer&timeRange=&sortBy=topic.lastposttime&sortDirection=desc&term=openvins I found some information about drifting and how to address it. But I'm not sure if that's the most up to date advice, it looks like the module is still under active development. Is the module in a place where it is ready for general customer use? The starling2 drones that ship with SDK1.6 use OpenVINs for localization, not QVIO, correct?

Thank you for any input!

Hi @Rowan-Dempster ,

Please take a look at this example (you can build and run it too) : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/add-new-image-tools/tools/voxl-image-repub.cpp

This app can accept a regular image (RAW8 or YUV) and either re-publish it unchanged or crop and publish the result. Sometimes this is useful for quickly cropping an image that is fed into a different application that expects a smaller image or different aspect ratio.

The app shows how to subscribe and handle ion buffer streams.

Usage:

voxl2:/$ voxl-image-repub ERROR: Pipe name not specified Re-publish cropped camera frames (RAW8 or YUV) Options are: -x, --crop-offset-x crop offset in horizontal dimension -y, --crop-offset-y crop offset in vertical dimension -w, --crop-size-x crop size in horizontal dimension (width) -h, --crop-size-y crop size in vertical dimension (height) -o, --output-name output pipe name -u, --usage print this help message The cropped image will be centered if the crop offsets are not provided. typical usage: /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 /# voxl-image-repub tracking --crop-size-x 256 --crop-size-y 256 --crop-offset-x 128 --crop-offset-y 128

example re-publishing ion buffer image as regular image (which you can view in voxl-portal ) :

voxl-image-repub tracking_front_misp_norm_ion -o test

(you can see which ion pipes are available by running voxl-list-pipes | grep _ion)

Please note that without the previous fix that i posted above, the client process that receives and uncached ION buffer will incur extra CPU load while accessing this buffer. For example, the same voxl-image-repub client uses 1.7% cpu while republishing the normalized image (cached ion buffer), while using 7.3% cpu republishing an image from an uncached ION buffer. (cpu usage % using one of the smaller cores).

Please try and let me know if you have any questions.

I know this cached / uncached buffering may be a bit confusing, but i will document this a bit more to help explain it a little better.

Alex

@Alex-Kushleyev The IMU is being configured with an 8K ODR so by increasing IMU_GYRO_RATEMAX you are reading the FIFO more often but reading less samples each time. So you are mainly increasing the overhead of context switching. Can you characterize how much the load increases just by increasing IMU_GYRO_RATEMAX and not doing any of the extra UART IO? Unfortunately we don't have a lot of control over the low level implementation of the IO drivers that are in the Qualcomm code. And there is no DMA that could help lower IO overhead. One idea would be to lower the ODR to 1K so that you are only reading one sample from the FIFO at each interrupt.

@Rowan-Dempster ,

Here is where the raw data from the TOF sensor comes into voxl-camera-server : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/hal3_camera_mgr.cpp?ref_type=heads#L1242

The data is in the MIPI12 packed format. The TOF library expects RAW16 format, so the data is converted from MIPI12 to RAW16 and fed into the TOF processing library. It is possible that dumping the RAW data is what RRF recording would be doing, so you could add that functionality to camera server to write the raw data to file. I could also help with publishing the RAW12 data as is to a mpa channel, so you can log it outside of the camera server (for example using voxl-record-raw-image utility. You could clarify with PMD if that would work (in other words, you would need to know the log format of the RRF recording)

We have not tried enabling the raw data logging. However, if you know what function needs to be called within PMD library (maybe you just enable logging, and it does everything for you), you should be able to add it somewhere in the TOF init code, maybe here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/dev/src/tof_interface.cpp?ref_type=heads#L982

You can find the API header files for Royale / Spectre in /usr/include/royale either on VOXL2 directly on in /opt/sysroots/qrb5165_1/usr/include/royale the voxl-cross:V4.4 docker.

Hopefully that helps!

Alex

@Alex-Kushleyev Gotcha, thanks for the info Alex and your help resolving this issue!

@Rowan-Dempster No problem! Kind of primitive but effective 🙂

(top is Royale 4, bottom is 5)

@Rowan-Dempster ,

I myself tried to do this some time ago and ran into issues, which i could not resolve. I am sure also someone else at Modal also tried it before me.

If I remember correctly, my issue was that it could not get camera module in the second instance of the camera server, on this line, and i did not pursue this further (did not spend too much time on it though).

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/hal3_helpers.cpp?ref_type=heads#L79

So in your case you are able to get the camera module?

To help debug, you should run logcat while you are trying this and see what messages might show up there.

And, finally, just wanted to mention that we don't know if what you are doing is possible, but if you want to keep pursuing this direction, I can help if i know the answer to your questions, but may not be able to spend time working on it right now.

EDIT: in order to see if separate instances of camera server are possible, you could try to see if you can run multiple instances of gstreamer plugin that uses qmmfsrc. I found a doc that should be helpful in testing this, take a look : https://thundercomm.s3-ap-northeast-1.amazonaws.com/shop/doc/1596593567074634/Thundercomm EB5_Multimedia SDK User Guide_V1.1_ie89e.pdf

Specifically, the following command should work, although i did not test it:

gst-launch-1.0 -e qtiqmmfsrc name=camsrc ! video/x-raw\(memory:GBM\),format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! omxh264enc control-rate=max-bitrate target-bitrate=6000000 interval-intraframes=29 periodicity-idr=1 ! queue ! filesink location="/data/vid.h264"

Also, i dont see how you sent the camera id in this example, but it should definitely be possible 🙂

See if you can run two independent instances of gstreamer using different cameras. I think it should work, but not sure.

Alex

@Rowan-Dempster I think it is safest to just not use data that reads 0,0,0 as the sensor cannot measure that close anyways

About 95C is when the temperature control loop will kick in and start reducing the maximum core frequencies (gradually). You can monitor the cpu usage and current core frequencies using voxl-inspect-cpu.

Here are the maximum core frequencies for all cores:

cpu0 1804.8 cpu1 1804.8 cpu2 1804.8 cpu3 1804.8 cpu4 2419.2 cpu5 2419.2 cpu6 2419.2 cpu7 2841.6

If you set the cpu governor mode to perf (voxl-set-cpu-mode perf), it will pin all the cores to max frequency and they will stay there unless they are being throttled due to temperature, which you can check using voxl-inspect-cpu.

@Rowan-Dempster Mainline PX4 now works on VOXL 2. It hasn't received that much testing yet but you are welcome to try it out.

@Rowan-Dempster No, it isn't

@Rowan-Dempster They've been released and are a part of the SDK but we just don't install them by default to save space.

The .debs are onboard already and should install with an apt install

In that context "blackisted" just means don't install by default.

@thomas I assume we don't want to be dealing with 4k raw arrays on the CPU for performance reasons. So yeah as early as possible in the image proc pipeline, and how early we can push it is what I'm asking for advice on. We'll also reach out to @Alex-Kushleyev thanks!!

@Rowan-Dempster, i looked through the changelog of our project (https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-eigen3/) and it does not look that there are any changes done to eigen3 itself, i think it is just a package we inherited from VOXL1, which did not use apt.

If you are willing to try it, remove voxl-eigen3 package (which may temporarily break something - note which packages require it). We could potentially update whatever gets broken to be happy with either voxl-eigen3 or eigen3 installed using apt.