Hi Alex,
To be greedy, ideally as many of those as possible. Ones that appeal most to us would be a
Then for format,
JPG probably makes the most sense but ofc, the more options the better haha.
Thanks for the detailed response Alex.
I think this addresses my most important concern, which was stretching of the misp pipes, I didn't realize that it was just cropping which is good news.
One more thing: Seems enabling Snapshots isn't compatible alongside MISP . Is that a fundamental limitation, or more of a how the stack is built thing? And is there currently any recommended way to still save a still image with MISP enabled? Or if not, a future implementation plan?
What I’m aiming for long term is high-res recording, lower-res streaming, and the option to grab a still in one configuration. With MISP ofc.
Best,
James
Working with EIS a bit more and was hoping for some clarification/advice on a couple things
First, want to clarify calibration for EIS. In the doc seems like it’s fine to calibrate at the full 4040x3040 (via disabling eis and setting misp_width and misp_height temporarily) or can alternatively calibrate at half-res at 2020x1520. Could I also hypothetically calibrate at 1010x760? We have our own calibration routine and the smaller the faster it can run.
I’m assuming as long as the aspect ratio is the same it should be fine? Furthermore, if I were to use the alternate resolution of 1996x1520 I should obviously do a calibration at that resolution? Or, keeping aspect ratio perhaps at 998x760?
Basically, just want to confirm that as long as a calibrate at some scale of whatever I set the preview frame to should be fine? And once I'm done calibrating I'm ok to then set misp_width and misp_height to my desired output resolution? This leads to my next clarification
I also wanted to clarify that EIS runs on whatever I set the preview frame to correct? As in that frame/resolution is what the algorithm runs on? misp_width and misp_height are just what gets outputted (as in its not like the algorithm is also getting run at that resolution)?
In that case what do you recommend eventually setting the output misp_width and misp_height to? Can they be anything? (Obviously something small enough to stream for example) I noticed in one example you chose 1280x720 but notably that is not the same aspect ratio as 4040x3040 or 1996x1520. Will stretching or warping occur? Should I pick something that is the same aspect ratio? Or does that not matter?
Any guidance much appreciated, thanks!
Super super low priority but I noticed that voxl-send-command doesn't output a return code. Even on an error can look like a success since echo will return exit status 0. This is particularly frustrating when an error does actually occur.
Here is a fix below:
if [ -p "/run/mpa/${1}/control" ]; then
_CONTROL_FILE="/run/mpa/${1}/control"
shift
echo "$@" >> "$_CONTROL_FILE"
echo "Successfully sent command to ${_CONTROL_FILE}"
exit 0
elif [ -p "${1}/control" ]; then
_CONTROL_FILE="${1}/control"
shift
echo "$@" >> "$_CONTROL_FILE"
echo "Successfully sent command to ${_CONTROL_FILE}"
exit 0
else
echo "Unable to find valid pipe that accepts control commands at: ${1}"
exit 1
fi
Would love to see an update here!
Also side note: Let me know if this is the proper channel so submit code issues like these.
Thanks Alex, that did the trick I was able to get voxl-portal successfully up and running afterwards. The zoom slider works here too!
However, I still can't seem to get EIS to work. Unlike your demo, there is no side-by-side picture present when i inspect hires_misp_color in voxl-portal. I realized there was an "eis_view" parameter so tried both "pip" and "side-by-side" as listed in the tutorial but with no luck. It's still just the single picture.
Furthermore, nothing looks like its being stabilized. I am missing an easy gotcha somewhere in my config file? Here's mine its basically just the one from the tutorial.
/**
* voxl-camera-server Configuration File
*
* Each camera has configurations for up to 4 HAL3 streams:
* - `preview` stream for raw unprocessed images from CV cameras
* - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
* - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
* - `snapshot` ISP-processed JPG snapshots that get saved to disk
*
* on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
*
* This file is generated from default values by voxl-configure-cameras.
* Do not expect arbitrary resolutions to work, the ISP and video compression
* pipelines only support very specific resolutions.
*
* The default video compression mode is cqp or Constant Quantization Parameter
*
*
*
*/
{
"version": 0.1,
"cameras": [{
"type": "imx412",
"name": "hires",
"enabled": true,
"camera_id": 0,
"fps": 30,
"en_preview": true,
"en_raw_preview": true,
"en_misp": true,
"preview_width": 4040,
"preview_height": 3040,
"misp_width": 1280,
"misp_height": 720,
"misp_awb": "auto",
"misp_zoom": 1.5,
"en_eis": true,
"eis_mode": "horizon-level",
"eis_view": "side-by-side",
"eis_follow_rate": 0.01,
"en_large_video": false,
"en_small_video": false,
"en_snapshot": false,
"ae_mode": "lme_msv",
"en_rotate": false,
"misp_venc_enable": true,
"misp_venc_mode": "h264",
"misp_venc_br_ctrl": "cbr",
"misp_venc_Qfixed": 30,
"misp_venc_Qmin": 15,
"misp_venc_Qmax": 51,
"misp_venc_nPframes": 29,
"misp_venc_mbps": 3,
"ae_desired_msv": 100,
"exposure_min_us": 25,
"exposure_max_us": 15000,
"gain_min": 100,
"gain_max": 32000,
"exposure_soft_min_us": 5000,
"snapshot_jpeg_quality": 75,
"small_video_width": 1024,
"small_video_height": 768,
"small_venc_mode": "h264",
"small_venc_br_ctrl": "cqp",
"small_venc_Qfixed": 30,
"small_venc_Qmin": 15,
"small_venc_Qmax": 40,
"small_venc_nPframes": 9,
"small_venc_mbps": 2,
"small_venc_osd": false,
"large_video_width": 4056,
"large_video_height": 3040,
"large_venc_mode": "h264",
"large_venc_br_ctrl": "cqp",
"large_venc_Qfixed": 40,
"large_venc_Qmin": 15,
"large_venc_Qmax": 50,
"large_venc_nPframes": 29,
"large_venc_mbps": 40,
"large_venc_osd": false,
"en_snapshot_width": 4056,
"en_snapshot_height": 3040,
"exif_focal_length": 3.0999999046325684,
"exif_focal_length_in_35mm_format": 17,
"exif_fnumber": 1.2400000095367432,
"misp_venc_osd": false,
"misp_gamma": 1.6,
"ae_filter_alpha": 0.600000023841858,
"ae_ignore_fraction": 0.20000000298023224,
"ae_slope": 0.05000000074505806,
"ae_exposure_period": 1,
"ae_gain_period": 1
}],
"fsync_en": false,
"fsync_gpio": 109
}
If not, whats the best method to debug this? Again I'm on SDK_1.5.0 on a Starling 2 with the only upgrades being voxl-portal and voxl-mongoose (and all the changes/requirements from the tutorial (as far as I can tell)).
Thanks!
Hi Alex, Thanks for the help
I was able to get the hires_misp_color stream working and mess around with the set_misp_zoom command. Very cool!
I was missing the "Use latest camera drivers" step. I will say that step could probably be flushed out a bit more. I only knew what to do with the downloaded driver files due to previous experience.
Unfortunately, I was unable to get the voxl-portal from the eis-integration branch running. Ran into a
starling2 (D0014):/data$ voxl-portal
voxl-portal: symbol lookup error: voxl-portal: undefined symbol: mg_log_level
error. Almost certainly a dependency mismatch. Maybe I messed up a build step?
Everything else I'm using is Stock voxl-sdk 1.5.0 (including the voxl-camera-server!) configured for the starling 2 . What were you using for the rest of your setup?
Let me know if there is any more info you need from me.
Best
James
Hi!
Was recently following along with https://docs.modalai.com/camera-video/electronic-image-stabilization/ and noticed it seemed a little out of date/potentially inaccurate.
I attempted to try this out by flashing voxl2 SDK 1.5.0 onto a starling 2 dev drone with a IMX412 M0161 (Coax).
I made sure to set
imu0_fifo_poll_rate_hz to 250
imu0_sample_rate_hz to 1000
in voxl-imu-server.conf
and
maxRAWSizes=20
in camxoverridesettings.txt
First oddity I found was the resolution which is listed as 4040x3040 but as far as I can tell that resolution isn't supported by the IMX412 M0161. With the Full-Frame preview being 4056x3040.
Here is the corrected voxl-camera-server.conf I used
/**
* voxl-camera-server Configuration File
*
* Each camera has configurations for up to 4 HAL3 streams:
* - `preview` stream for raw unprocessed images from CV cameras
* - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
* - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
* - `snapshot` ISP-processed JPG snapshots that get saved to disk
*
* on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
*
* This file is generated from default values by voxl-configure-cameras.
* Do not expect arbitrary resolutions to work, the ISP and video compression
* pipelines only support very specific resolutions.
*
* The default video compression mode is cqp or Constant Quantization Parameter
*
*
*
*/
{
"version": 0.1,
"cameras": [{
"type": "imx412",
"name": "hires",
"enabled": true,
"camera_id": 0,
"fps": 30,
"en_preview": true,
"en_raw_preview": true,
"en_misp": true,
"preview_width": 4056,
"preview_height": 3040,
"misp_width": 1280,
"misp_height": 720,
"misp_awb": "auto",
"misp_zoom": 1.5,
"en_eis": true,
"eis_mode": "horizon-level",
"eis_view": "pip",
"eis_follow_rate": 0.01,
"en_large_video": false,
"en_small_video": false,
"en_snapshot": false,
"ae_mode": "lme_msv",
"en_rotate": false,
"misp_venc_enable": true,
"misp_venc_mode": "h264",
"misp_venc_br_ctrl": "cbr",
"misp_venc_Qfixed": 30,
"misp_venc_Qmin": 15,
"misp_venc_Qmax": 51,
"misp_venc_nPframes": 29,
"misp_venc_mbps": 3,
"ae_desired_msv": 100,
"exposure_min_us": 25,
"exposure_max_us": 15000,
"gain_min": 100,
"gain_max": 32000,
"exposure_soft_min_us": 5000,
"snapshot_jpeg_quality": 75,
"small_video_width": 1024,
"small_video_height": 768,
"small_venc_mode": "h264",
"small_venc_br_ctrl": "cqp",
"small_venc_Qfixed": 30,
"small_venc_Qmin": 15,
"small_venc_Qmax": 40,
"small_venc_nPframes": 9,
"small_venc_mbps": 2,
"small_venc_osd": false,
"large_video_width": 4056,
"large_video_height": 3040,
"large_venc_mode": "h264",
"large_venc_br_ctrl": "cqp",
"large_venc_Qfixed": 40,
"large_venc_Qmin": 15,
"large_venc_Qmax": 50,
"large_venc_nPframes": 29,
"large_venc_mbps": 40,
"large_venc_osd": false,
"en_snapshot_width": 4056,
"en_snapshot_height": 3040,
"exif_focal_length": 3.0999999046325684,
"exif_focal_length_in_35mm_format": 17,
"exif_fnumber": 1.2400000095367432,
"misp_venc_osd": false,
"misp_gamma": 1.6,
"ae_filter_alpha": 0.600000023841858,
"ae_ignore_fraction": 0.20000000298023224,
"ae_slope": 0.05000000074505806,
"ae_exposure_period": 1,
"ae_gain_period": 1
}],
"fsync_en": false,
"fsync_gpio": 109
}
Unfortunately this is the image I got out of the portal from hires_misp_color. I must be missing a step here. Furthermore I didn't see anything in the portal about a advanced control panel for misp control commands.
I didn't do an intrinsic cal yet but I wouldn't think it would produce this without that.
Assumably, the extrinsics are also already set since this is a Starling 2 with defaults present.
Any help getting this up and running would be super appreciated. A small demo or updated setup guide would be extremely helpful. If there is anything else you need from me in terms of my setup let me know!
Thanks! James
Side Question:
Am curious if the zoom level of the final image can be changed dynamically. Poking into the voxl-camera-server's code for SDK 1.5.0 i noticed there is a
set_misp_zoom
Cl function available. Assuming I get the camera pipe up and running correctly, am I correct in thinking this could "zoom in" or "zoom out" the image while it is running?
Thanks!
Currently I'm trying to get the GPS working on the Starling2 along with a custom airframe. After a bunch of experimentation I'm seeing that having specifically the hires camera (M0161) plugged in and running is causing a noticeable degradation in GPS quality. (Higher EPH, takes longer to acquire satellites/fix, etc.) Assumably, this is due to EMI.
Is this something anyone else has noticed with the hires? I’m hypothesizing it is because it is using proportionally more power and transferring more data (compared to the other cameras) that this is occurring/is more noticeable. Besides moving the GPS module further away, are there any other techniques or suggestions Modal has for reducing EMI noise? Namely,
Shielding
MIPI Clock rates
Any advice is much appreciated, thanks!
@Alex-Kushleyev Sorry for the delay got sidetracked with some other things but wanted to close this ticket out.
Focus did indeed seem to be the issue. However, seems like most of our M0161 modules simply did not come with any epoxy, or at the very least not enough to prevent the lens from spinning accidentally. Something to watch out for.
Also, I found that at least on Voxl-SDK-1.4.0 for me it seems
com.qti.tuned.imx412_fpv.bin
is indeed what is being loaded as default. By renaming com.qti.tuned.cmk_imx577.bin to force the camera server to load it I found the picture at least aesthetically more pleasing. But that is obviously subjective.
Anyways, yeah focus seemed to be the major issue.