ModalAI Forum

@alex-kushleyev Tagging you for visibility

This likely originated a while ago, but noticed recently we had two different version of IMX412 M12 lenses within our IMX412 stockpile.

One, which I assume is newer as it looks like what is posted online has a larger fisheye effect than the other version we have, which is much flatter. Not as concerned with the differing image output right now, but just wanted to seek clarification on why we might have two different lenses. Was this a known change? Which ones should we expect to receive moving forward? Or is the two different types expected?

From what I understand, all of our lenses we've received from starling kits.

On the Left is the (newer?) more fisheye lens and on the right is the flatter looking one.

Any help would be much appreciated, thanks!

@Alex-Kushleyev

Aha yes that was my original understanding. Yes knew you'd need a zoom level for it to actually look good/work effectively. Would probably have a lower bound of 1.2-1.3 likely from what I experimented with. TBD on dimensions.

Sorry, was confused, sounds good!

@Alex-Kushleyev

Awesome, sounds good Alex. That does indeed sound like it would meet our needs.

If EIS is enabled, How much smaller would the image be than 4040x3040? What is the max output resolution with EIS enabled?

Hi @Alex-Kushleyev,

Thanks for the reply.

The 4 AR0144 cameras would be used independently rather than as stereo pairs, so I don't believe strict synchronization is required for my use case. My main goal is to add a 4th AR0144 for an independent tracking/perception task.

As long as the 4th camera can be timestamped accurately and integrated alongside the other tracking cameras, I think that should be sufficient. The resistor swap seems a little tricky, so I might post again if I need help with that.

Thanks!

Will an AR0144 work on J8? Would probably need a Micro-coax Adapter? Not sure if its one output maps to Lower or Upper.

But couldn't find anything about if J8 would support AR0144. At the very least i was able to configure the camera-server with a AR0144 on both J8 Upper and Lower and it succeeded (although cannot test since i cannot connect the actual sensor).

Is this something that has been tested in the past?

For some context, am trying to see if we can run 4 AR0144s, 1 imx412, and 1 pmd-tof-liow2. Potentially some other sorting of these would work. (Such as putting the imx412 on J8 and another AR0144 on J6. Totally okay if it isn't an officially supported config haha can internally test.

current config with the starling breakout board looks like, looking to add one more AR0144

J6_LOWER_SENSOR="ar0144-fsin-combo"
J6_LOWER_NAME="arone"
J6_LOWER_ROTATE="true"

J6_LOWER_COMBO1_SENSOR="ar0144-fsin-combo"
J6_LOWER_COMBO1_NAME="artwo"
J6_LOWER_COMBO1_ROTATE="false"

J6_UPPER_SENSOR="imx412-fpv"
J6_UPPER_NAME="hires"
J6_UPPER_ROTATE="false"

J7_LOWER_SENSOR="ar0144-fsin"
J7_LOWER_NAME="arthree"
J7_LOWER_ROTATE="true"

J7_UPPER_SENSOR="pmd-tof-liow2"
J7_UPPER_NAME="tof"
J7_UPPER_ROTATE="false"

J8_UPPER_SENSOR=""
J8_UPPER_NAME=""
J8_UPPER_ROTATE="false"

Any help much appreciated, Thanks!

Best,
James

@Alex-Kushleyev

Hi Alex, apologies again!

Yes independent zoom is what we would be looking for. Awesome to hear you’ve been working on it. Yeah the three main use cases for us would be recording, streaming, and calibration (Makes it easier for an assembly process to not have to switch camera configs). So here is the minimum channel use case we were looking for

We currently use 
J6_UPPER_SENSOR: imx412-fpv

Channel 1 (Potential Recording Pipe 1 (Full Frame)):
Dimensions: 4040x3040
FPS: 30fps
Zoom: OFF/1x
EIS: OFF
EIS MODE: N/A

Channel 2: (Potential Recording Pipe 2 (Variable Zoom):
Dimensions: 4040x3040
FPS: 30fps
Zoom: Variable (1x-whatever, would be sync with streaming pipe. We would implement this logic)
EIS: ON
EIS MODE: Horizon (or full-follow)

Notes: Would likely leave it to our users whether to record full frame or be synced with streaming pipe. So would only ever need one of the recording pipes likely never both at once.

Channel 3: (Streaming Pipe):
Dimensions: 1280x800 (16:10 matches our tablet aspect ratio, not sure if there is something better to use here.)
FPS: 30fps
Zoom: Variable (1x-whatever)
EIS: On
EIS MODE: Horizon (or full-follow)

Channel 3 (Calibration Pipe):
Dimensions: 1010x760 (we’ve been calibrating at quarter resolution with our own internal calibration routine)
FPS: 30fps
Zoom: Off/1x (zero zoom full picture)
EIS: OFF
EIS MODE: N/A
Notes: Really only need the grayscale/normalized from this.

Anyways, happy to give it a try if you have available. Zero rush, currently juggling a few projects.

Best,
James

@Alex-Kushleyev

Hi Alex,

Sorry has been awhile, keep having to step away and re-visit this.

First, seems like snapshot seems to be fixed in one of your latest dev releases, tried that out and seems to work so that’s awesome. Able to capture a full frame non-MISP image from the 412. That being said, although I tested out voxl-record-raw-image on the hires_misp_color stream, and was able to eventually convert it to a jpg, its not as usable if not in a standard image format by default.

In addition I had a few more clarifying questions.

1.

Seems like there’s no way to have the large hires pipe on at the same time as the MISP pipe? Meaning it’s not possible to get a recording of the full frame, no eis, at the same time the MISP pipe is being used? The reason why this might be useful is because although I could have a high-resolution MISP pipe for recording it would have the same zoom level as the other misp pipes as it seems that is shared parameter. I assume this is a system limitation and the voxl2 might not be able to handle having multiple 4k pipes and encoding them. **Just want to verify this.**


2.

I noticed there is a imx412 and imx412-misp driver. What is the difference between these two? MISP seems to work fine using the regular imx412 driver (and that’s what I’ve been using since https://docs.modalai.com/camera-video/electronic-image-stabilization/ didn’t make mention of the other one).

Is it just the same thing, different name or are there gains to be made switching to the dedicated MISP driver? What are those?

Let me know, Thanks!

P.S love to see the continued work being done by you all here on the camera server very cool!

@jameskuesel

Hi Alex,

To be greedy, ideally as many of those as possible. Ones that appeal most to us would be a

1. A frame from high-res recording (with eis)
2. A full frame, full res (no eis)

Then for format,
JPG probably makes the most sense but ofc, the more options the better haha.

@Alex-Kushleyev

Thanks for the detailed response Alex.

I think this addresses my most important concern, which was stretching of the misp pipes, I didn't realize that it was just cropping which is good news.

One more thing: Seems enabling Snapshots isn't compatible alongside MISP . Is that a fundamental limitation, or more of a how the stack is built thing? And is there currently any recommended way to still save a still image with MISP enabled? Or if not, a future implementation plan?

What I’m aiming for long term is high-res recording, lower-res streaming, and the option to grab a still in one configuration. With MISP ofc.

Best,
James

@Alex-Kushleyev

Working with EIS a bit more and was hoping for some clarification/advice on a couple things

First, want to clarify calibration for EIS. In the doc seems like it’s fine to calibrate at the full 4040x3040 (via disabling eis and setting misp_width and misp_height temporarily) or can alternatively calibrate at half-res at 2020x1520. Could I also hypothetically calibrate at 1010x760? We have our own calibration routine and the smaller the faster it can run.

I’m assuming as long as the aspect ratio is the same it should be fine?

 Furthermore, if I were to use the alternate resolution of 1996x1520 I should obviously do a calibration at that resolution? Or, keeping aspect ratio perhaps at 998x760?

Basically, just want to confirm that as long as a calibrate at some scale of whatever I set the preview frame to should be fine? And once I'm done calibrating I'm ok to then set misp_width and misp_height to my desired output resolution? This leads to my next clarification

I also wanted to clarify that EIS runs on whatever I set the preview frame to correct? As in that frame/resolution is what the algorithm runs on? misp_width and misp_height are just what gets outputted (as in its not like the algorithm is also getting run at that resolution)?

In that case what do you recommend eventually setting the output misp_width and misp_height to? Can they be anything? (Obviously something small enough to stream for example) I noticed in one example you chose 1280x720 but notably that is not the same aspect ratio as 4040x3040 or 1996x1520. Will stretching or warping occur? Should I pick something that is the same aspect ratio? Or does that not matter?

Any guidance much appreciated, thanks!

@Eric-Katzfey

Sorry my bad! Done!

Super super low priority but I noticed that voxl-send-command doesn't output a return code. Even on an error can look like a success since echo will return exit status 0. This is particularly frustrating when an error does actually occur.

Here is a fix below:

if [ -p "/run/mpa/${1}/control" ]; then
    _CONTROL_FILE="/run/mpa/${1}/control"
    shift
    echo "$@" >> "$_CONTROL_FILE"
    echo "Successfully sent command to ${_CONTROL_FILE}"
    exit 0

elif [ -p "${1}/control" ]; then
    _CONTROL_FILE="${1}/control"
    shift
    echo "$@" >> "$_CONTROL_FILE"
    echo "Successfully sent command to ${_CONTROL_FILE}"
    exit 0
else
    echo "Unable to find valid pipe that accepts control commands at: ${1}"
    exit 1
fi

Would love to see an update here!

Also side note: Let me know if this is the proper channel so submit code issues like these.

@Alex-Kushleyev

Thanks Alex, that did the trick I was able to get voxl-portal successfully up and running afterwards. The zoom slider works here too!

However, I still can't seem to get EIS to work. Unlike your demo, there is no side-by-side picture present when i inspect hires_misp_color in voxl-portal. I realized there was an "eis_view" parameter so tried both "pip" and "side-by-side" as listed in the tutorial but with no luck. It's still just the single picture.

Furthermore, nothing looks like its being stabilized. I am missing an easy gotcha somewhere in my config file? Here's mine its basically just the one from the tutorial.

/**
 * voxl-camera-server Configuration File
 *
 * Each camera has configurations for up to 4 HAL3 streams:
 *    - `preview` stream for raw unprocessed images from CV cameras
 *    - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
 *    - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
 *    - `snapshot` ISP-processed JPG snapshots that get saved to disk
 *
 * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
 *
 * This file is generated from default values by voxl-configure-cameras.
 * Do not expect arbitrary resolutions to work, the ISP and video compression
 * pipelines only support very specific resolutions.
 *
 * The default video compression mode is cqp or Constant Quantization Parameter
 *
 *
 *
 */
{
	"version":	0.1,
	"cameras":	[{
			"type":	"imx412",
			"name":	"hires",
			"enabled":	true,
			"camera_id":	0,
			"fps":	30,
			"en_preview":	true,
			"en_raw_preview":	true,
			"en_misp":	true,
			"preview_width":	4040,
			"preview_height":	3040,
			"misp_width":	1280,
			"misp_height":	720,
			"misp_awb":	"auto",
			"misp_zoom":	1.5,
			"en_eis":	true,
			"eis_mode":	"horizon-level",
			"eis_view":	"side-by-side",
			"eis_follow_rate":	0.01,
			"en_large_video":	false,
			"en_small_video":	false,
			"en_snapshot":	false,
			"ae_mode":	"lme_msv",
			"en_rotate":	false,
			"misp_venc_enable":	true,
			"misp_venc_mode":	"h264",
			"misp_venc_br_ctrl":	"cbr",
			"misp_venc_Qfixed":	30,
			"misp_venc_Qmin":	15,
			"misp_venc_Qmax":	51,
			"misp_venc_nPframes":	29,
			"misp_venc_mbps":	3,
			"ae_desired_msv":	100,
			"exposure_min_us":	25,
			"exposure_max_us":	15000,
			"gain_min":	100,
			"gain_max":	32000,
			"exposure_soft_min_us":	5000,
			"snapshot_jpeg_quality":	75,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h264",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"small_venc_osd":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h264",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	40,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	40,
			"large_venc_osd":	false,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"misp_venc_osd":	false,
			"misp_gamma":	1.6,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}],
	"fsync_en":	false,
	"fsync_gpio":	109
}

If not, whats the best method to debug this? Again I'm on SDK_1.5.0 on a Starling 2 with the only upgrades being voxl-portal and voxl-mongoose (and all the changes/requirements from the tutorial (as far as I can tell)).

Thanks!

@Alex-Kushleyev

Hi Alex, Thanks for the help

I was able to get the hires_misp_color stream working and mess around with the set_misp_zoom command. Very cool!

I was missing the "Use latest camera drivers" step. I will say that step could probably be flushed out a bit more. I only knew what to do with the downloaded driver files due to previous experience.

Unfortunately, I was unable to get the voxl-portal from the eis-integration branch running. Ran into a

starling2 (D0014):/data$ voxl-portal
voxl-portal: symbol lookup error: voxl-portal: undefined symbol: mg_log_level

error. Almost certainly a dependency mismatch. Maybe I messed up a build step?

Everything else I'm using is Stock voxl-sdk 1.5.0 (including the voxl-camera-server!) configured for the starling 2 . What were you using for the rest of your setup?

Let me know if there is any more info you need from me.

Best
James

Hi!

Was recently following along with https://docs.modalai.com/camera-video/electronic-image-stabilization/ and noticed it seemed a little out of date/potentially inaccurate.

I attempted to try this out by flashing voxl2 SDK 1.5.0 onto a starling 2 dev drone with a IMX412 M0161 (Coax).

I made sure to set
imu0_fifo_poll_rate_hz to 250
imu0_sample_rate_hz to 1000
in voxl-imu-server.conf

and
maxRAWSizes=20
in camxoverridesettings.txt

First oddity I found was the resolution which is listed as 4040x3040 but as far as I can tell that resolution isn't supported by the IMX412 M0161. With the Full-Frame preview being 4056x3040.

Here is the corrected voxl-camera-server.conf I used

/**
 * voxl-camera-server Configuration File
 *
 * Each camera has configurations for up to 4 HAL3 streams:
 *    - `preview` stream for raw unprocessed images from CV cameras
 *    - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
 *    - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
 *    - `snapshot` ISP-processed JPG snapshots that get saved to disk
 *
 * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
 *
 * This file is generated from default values by voxl-configure-cameras.
 * Do not expect arbitrary resolutions to work, the ISP and video compression
 * pipelines only support very specific resolutions.
 *
 * The default video compression mode is cqp or Constant Quantization Parameter
 *
 *
 *
 */
{
	"version":	0.1,
	"cameras":	[{
			"type":	"imx412",
			"name":	"hires",
			"enabled":	true,
			"camera_id":	0,
			"fps":	30,
			"en_preview":	true,
			"en_raw_preview":	true,
			"en_misp":	true,
			"preview_width":	4056,
			"preview_height":	3040,
			"misp_width":	1280,
			"misp_height":	720,
			"misp_awb":	"auto",
			"misp_zoom":	1.5,
			"en_eis":	true,
			"eis_mode":	"horizon-level",
			"eis_view":	"pip",
			"eis_follow_rate":	0.01,
			"en_large_video":	false,
			"en_small_video":	false,
			"en_snapshot":	false,
			"ae_mode":	"lme_msv",
			"en_rotate":	false,
			"misp_venc_enable":	true,
			"misp_venc_mode":	"h264",
			"misp_venc_br_ctrl":	"cbr",
			"misp_venc_Qfixed":	30,
			"misp_venc_Qmin":	15,
			"misp_venc_Qmax":	51,
			"misp_venc_nPframes":	29,
			"misp_venc_mbps":	3,
			"ae_desired_msv":	100,
			"exposure_min_us":	25,
			"exposure_max_us":	15000,
			"gain_min":	100,
			"gain_max":	32000,
			"exposure_soft_min_us":	5000,
			"snapshot_jpeg_quality":	75,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h264",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"small_venc_osd":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h264",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	40,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	40,
			"large_venc_osd":	false,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"misp_venc_osd":	false,
			"misp_gamma":	1.6,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}],
	"fsync_en":	false,
	"fsync_gpio":	109
}

Unfortunately this is the image I got out of the portal from hires_misp_color. I must be missing a step here. Furthermore I didn't see anything in the portal about a advanced control panel for misp control commands.

I didn't do an intrinsic cal yet but I wouldn't think it would produce this without that.
Assumably, the extrinsics are also already set since this is a Starling 2 with defaults present.

Any help getting this up and running would be super appreciated. A small demo or updated setup guide would be extremely helpful. If there is anything else you need from me in terms of my setup let me know!

Thanks! James

Side Question:

Am curious if the zoom level of the final image can be changed dynamically. Poking into the voxl-camera-server's code for SDK 1.5.0 i noticed there is a

set_misp_zoom

Cl function available. Assuming I get the camera pipe up and running correctly, am I correct in thinking this could "zoom in" or "zoom out" the image while it is running?

Thanks!

Currently I'm trying to get the GPS working on the Starling2 along with a custom airframe. After a bunch of experimentation I'm seeing that having specifically the hires camera (M0161) plugged in and running is causing a noticeable degradation in GPS quality. (Higher EPH, takes longer to acquire satellites/fix, etc.) Assumably, this is due to EMI.

Is this something anyone else has noticed with the hires? I’m hypothesizing it is because it is using proportionally more power and transferring more data (compared to the other cameras) that this is occurring/is more noticeable. 

Besides moving the GPS module further away, are there any other techniques or suggestions Modal has for reducing EMI noise? Namely,

1. Shielding

   • Any materials in specific for wrapping cables or blocking EMI, copper, EMI tape, others?
   • Cables to shield, the gps wire? The Hires Coax, the M0161 itself, the camera board, barrier below the gps module, everything?

2. MIPI Clock rates 

   • Bit of a long shot (I’m not really familiar with any of this) but I read online that sometimes changing the clock rate can help with EMI. Not sure if this an option, would help, or even possible. Wondering if I could apply this to the Hires. Again, a long shot.
   • https://en.wikipedia.org/wiki/Camera_Serial_Interface

Any advice is much appreciated, thanks!

@Alex-Kushleyev Sorry for the delay got sidetracked with some other things but wanted to close this ticket out.

Focus did indeed seem to be the issue. However, seems like most of our M0161 modules simply did not come with any epoxy, or at the very least not enough to prevent the lens from spinning accidentally. Something to watch out for.

Also, I found that at least on Voxl-SDK-1.4.0 for me it seems

com.qti.tuned.imx412_fpv.bin

is indeed what is being loaded as default. By renaming com.qti.tuned.cmk_imx577.bin to force the camera server to load it I found the picture at least aesthetically more pleasing. But that is obviously subjective.

Anyways, yeah focus seemed to be the major issue.