Status of Image Stabilization and Potentially Zoom?

jameskuesel

Hi!

Was recently following along with https://docs.modalai.com/camera-video/electronic-image-stabilization/ and noticed it seemed a little out of date/potentially inaccurate.

I attempted to try this out by flashing voxl2 SDK 1.5.0 onto a starling 2 dev drone with a IMX412 M0161 (Coax).

I made sure to set
imu0_fifo_poll_rate_hz to 250
imu0_sample_rate_hz to 1000
in voxl-imu-server.conf

and
maxRAWSizes=20
in camxoverridesettings.txt

First oddity I found was the resolution which is listed as 4040x3040 but as far as I can tell that resolution isn't supported by the IMX412 M0161. With the Full-Frame preview being 4056x3040.

Here is the corrected voxl-camera-server.conf I used

/**
 * voxl-camera-server Configuration File
 *
 * Each camera has configurations for up to 4 HAL3 streams:
 *    - `preview` stream for raw unprocessed images from CV cameras
 *    - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
 *    - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
 *    - `snapshot` ISP-processed JPG snapshots that get saved to disk
 *
 * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
 *
 * This file is generated from default values by voxl-configure-cameras.
 * Do not expect arbitrary resolutions to work, the ISP and video compression
 * pipelines only support very specific resolutions.
 *
 * The default video compression mode is cqp or Constant Quantization Parameter
 *
 *
 *
 */
{
	"version":	0.1,
	"cameras":	[{
			"type":	"imx412",
			"name":	"hires",
			"enabled":	true,
			"camera_id":	0,
			"fps":	30,
			"en_preview":	true,
			"en_raw_preview":	true,
			"en_misp":	true,
			"preview_width":	4056,
			"preview_height":	3040,
			"misp_width":	1280,
			"misp_height":	720,
			"misp_awb":	"auto",
			"misp_zoom":	1.5,
			"en_eis":	true,
			"eis_mode":	"horizon-level",
			"eis_view":	"pip",
			"eis_follow_rate":	0.01,
			"en_large_video":	false,
			"en_small_video":	false,
			"en_snapshot":	false,
			"ae_mode":	"lme_msv",
			"en_rotate":	false,
			"misp_venc_enable":	true,
			"misp_venc_mode":	"h264",
			"misp_venc_br_ctrl":	"cbr",
			"misp_venc_Qfixed":	30,
			"misp_venc_Qmin":	15,
			"misp_venc_Qmax":	51,
			"misp_venc_nPframes":	29,
			"misp_venc_mbps":	3,
			"ae_desired_msv":	100,
			"exposure_min_us":	25,
			"exposure_max_us":	15000,
			"gain_min":	100,
			"gain_max":	32000,
			"exposure_soft_min_us":	5000,
			"snapshot_jpeg_quality":	75,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h264",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"small_venc_osd":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h264",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	40,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	40,
			"large_venc_osd":	false,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"misp_venc_osd":	false,
			"misp_gamma":	1.6,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}],
	"fsync_en":	false,
	"fsync_gpio":	109
}

Unfortunately this is the image I got out of the portal from hires_misp_color. I must be missing a step here. Furthermore I didn't see anything in the portal about a advanced control panel for misp control commands.

I didn't do an intrinsic cal yet but I wouldn't think it would produce this without that.
Assumably, the extrinsics are also already set since this is a Starling 2 with defaults present.

Any help getting this up and running would be super appreciated. A small demo or updated setup guide would be extremely helpful. If there is anything else you need from me in terms of my setup let me know!

Thanks! James

Side Question:

Am curious if the zoom level of the final image can be changed dynamically. Poking into the voxl-camera-server's code for SDK 1.5.0 i noticed there is a

set_misp_zoom

Cl function available. Assuming I get the camera pipe up and running correctly, am I correct in thinking this could "zoom in" or "zoom out" the image while it is running?

Thanks!