@Alex-Kushleyev

Hi Alex, Thanks for the help

I was able to get the hires_misp_color stream working and mess around with the set_misp_zoom command. Very cool!

I was missing the "Use latest camera drivers" step. I will say that step could probably be flushed out a bit more. I only knew what to do with the downloaded driver files due to previous experience.

Unfortunately, I was unable to get the voxl-portal from the eis-integration branch running. Ran into a

starling2 (D0014):/data$ voxl-portal
voxl-portal: symbol lookup error: voxl-portal: undefined symbol: mg_log_level

error. Almost certainly a dependency mismatch. Maybe I messed up a build step?

Everything else I'm using is Stock voxl-sdk 1.5.0 (including the voxl-camera-server!) configured for the starling 2 . What were you using for the rest of your setup?

Let me know if there is any more info you need from me.

Best
James

Hi!

Was recently following along with https://docs.modalai.com/camera-video/electronic-image-stabilization/ and noticed it seemed a little out of date/potentially inaccurate.

I attempted to try this out by flashing voxl2 SDK 1.5.0 onto a starling 2 dev drone with a IMX412 M0161 (Coax).

I made sure to set
imu0_fifo_poll_rate_hz to 250
imu0_sample_rate_hz to 1000
in voxl-imu-server.conf

and
maxRAWSizes=20
in camxoverridesettings.txt

First oddity I found was the resolution which is listed as 4040x3040 but as far as I can tell that resolution isn't supported by the IMX412 M0161. With the Full-Frame preview being 4056x3040.

Here is the corrected voxl-camera-server.conf I used

/**
 * voxl-camera-server Configuration File
 *
 * Each camera has configurations for up to 4 HAL3 streams:
 *    - `preview` stream for raw unprocessed images from CV cameras
 *    - `small_video` 720p (ish) h264/h265 compressed for fpv video streaming
 *    - `large_video` 4k (ish) h264/h265 for onboard video recording to disk
 *    - `snapshot` ISP-processed JPG snapshots that get saved to disk
 *
 * on QRB5165 platforms (VOXL2 and VOXL2 mini) you can only have 3 of the 4 enabled
 *
 * This file is generated from default values by voxl-configure-cameras.
 * Do not expect arbitrary resolutions to work, the ISP and video compression
 * pipelines only support very specific resolutions.
 *
 * The default video compression mode is cqp or Constant Quantization Parameter
 *
 *
 *
 */
{
	"version":	0.1,
	"cameras":	[{
			"type":	"imx412",
			"name":	"hires",
			"enabled":	true,
			"camera_id":	0,
			"fps":	30,
			"en_preview":	true,
			"en_raw_preview":	true,
			"en_misp":	true,
			"preview_width":	4056,
			"preview_height":	3040,
			"misp_width":	1280,
			"misp_height":	720,
			"misp_awb":	"auto",
			"misp_zoom":	1.5,
			"en_eis":	true,
			"eis_mode":	"horizon-level",
			"eis_view":	"pip",
			"eis_follow_rate":	0.01,
			"en_large_video":	false,
			"en_small_video":	false,
			"en_snapshot":	false,
			"ae_mode":	"lme_msv",
			"en_rotate":	false,
			"misp_venc_enable":	true,
			"misp_venc_mode":	"h264",
			"misp_venc_br_ctrl":	"cbr",
			"misp_venc_Qfixed":	30,
			"misp_venc_Qmin":	15,
			"misp_venc_Qmax":	51,
			"misp_venc_nPframes":	29,
			"misp_venc_mbps":	3,
			"ae_desired_msv":	100,
			"exposure_min_us":	25,
			"exposure_max_us":	15000,
			"gain_min":	100,
			"gain_max":	32000,
			"exposure_soft_min_us":	5000,
			"snapshot_jpeg_quality":	75,
			"small_video_width":	1024,
			"small_video_height":	768,
			"small_venc_mode":	"h264",
			"small_venc_br_ctrl":	"cqp",
			"small_venc_Qfixed":	30,
			"small_venc_Qmin":	15,
			"small_venc_Qmax":	40,
			"small_venc_nPframes":	9,
			"small_venc_mbps":	2,
			"small_venc_osd":	false,
			"large_video_width":	4056,
			"large_video_height":	3040,
			"large_venc_mode":	"h264",
			"large_venc_br_ctrl":	"cqp",
			"large_venc_Qfixed":	40,
			"large_venc_Qmin":	15,
			"large_venc_Qmax":	50,
			"large_venc_nPframes":	29,
			"large_venc_mbps":	40,
			"large_venc_osd":	false,
			"en_snapshot_width":	4056,
			"en_snapshot_height":	3040,
			"exif_focal_length":	3.0999999046325684,
			"exif_focal_length_in_35mm_format":	17,
			"exif_fnumber":	1.2400000095367432,
			"misp_venc_osd":	false,
			"misp_gamma":	1.6,
			"ae_filter_alpha":	0.600000023841858,
			"ae_ignore_fraction":	0.20000000298023224,
			"ae_slope":	0.05000000074505806,
			"ae_exposure_period":	1,
			"ae_gain_period":	1
		}],
	"fsync_en":	false,
	"fsync_gpio":	109
}

Unfortunately this is the image I got out of the portal from hires_misp_color. I must be missing a step here. Furthermore I didn't see anything in the portal about a advanced control panel for misp control commands.

I didn't do an intrinsic cal yet but I wouldn't think it would produce this without that.
Assumably, the extrinsics are also already set since this is a Starling 2 with defaults present.

Any help getting this up and running would be super appreciated. A small demo or updated setup guide would be extremely helpful. If there is anything else you need from me in terms of my setup let me know!

Thanks! James

Side Question:

Am curious if the zoom level of the final image can be changed dynamically. Poking into the voxl-camera-server's code for SDK 1.5.0 i noticed there is a

set_misp_zoom

Cl function available. Assuming I get the camera pipe up and running correctly, am I correct in thinking this could "zoom in" or "zoom out" the image while it is running?

Thanks!

Currently I'm trying to get the GPS working on the Starling2 along with a custom airframe. After a bunch of experimentation I'm seeing that having specifically the hires camera (M0161) plugged in and running is causing a noticeable degradation in GPS quality. (Higher EPH, takes longer to acquire satellites/fix, etc.) Assumably, this is due to EMI.

Is this something anyone else has noticed with the hires? I’m hypothesizing it is because it is using proportionally more power and transferring more data (compared to the other cameras) that this is occurring/is more noticeable. 

Besides moving the GPS module further away, are there any other techniques or suggestions Modal has for reducing EMI noise? Namely,

1. Shielding

   • Any materials in specific for wrapping cables or blocking EMI, copper, EMI tape, others?
   • Cables to shield, the gps wire? The Hires Coax, the M0161 itself, the camera board, barrier below the gps module, everything?

2. MIPI Clock rates

   • Bit of a long shot (I’m not really familiar with any of this) but I read online that sometimes changing the clock rate can help with EMI. Not sure if this an option, would help, or even possible. Wondering if I could apply this to the Hires. Again, a long shot.
   • https://en.wikipedia.org/wiki/Camera_Serial_Interface

Any advice is much appreciated, thanks!

@Alex-Kushleyev Sorry for the delay got sidetracked with some other things but wanted to close this ticket out.

Focus did indeed seem to be the issue. However, seems like most of our M0161 modules simply did not come with any epoxy, or at the very least not enough to prevent the lens from spinning accidentally. Something to watch out for.

Also, I found that at least on Voxl-SDK-1.4.0 for me it seems

com.qti.tuned.imx412_fpv.bin

is indeed what is being loaded as default. By renaming com.qti.tuned.cmk_imx577.bin to force the camera server to load it I found the picture at least aesthetically more pleasing. But that is obviously subjective.

Anyways, yeah focus seemed to be the major issue.

@Alex-Kushleyev

Hi Alex,

To get the M0161 (COAX) all I did was run the install script, and then selected all the options relating to the starling2 since that is what I was collecting these on.

I've been recording the preview frame (en_preview: true) modified to be at full res (4056x3040), stream via voxl-logger.

voxl-logger --cam hires_color.

I did a similar process for the M0107 (RIBBON), except selecting the first kernel option. Then I modified the custom_camera_config.txt in /data/modalai and just set

J6_LOWER_SENSOR="imx412"
J6_LOWER_NAME="hires"
J6_LOWER_ROTATE="false"

I think you are correct that our Starling M0161 lens is a little bit out of focus. However, we tried with another M0161 module we had around that looked more "in focus" but still had the sharpness issues. The resulting images between the M0107 and M0161 look different enough that I feel it is something more than just the focus but I am unsure.

M0161 but DIFFERENT module (more focused?)

Anyways, yes let me know when you get some sample images and would also love to experiment with the logging of the raw images. Also, how did you set up both modules at the same time? I was trying to figure that out but had no luck.

Hi Alex, Thanks for the reply.

Unfortunately, I didn’t see any improvement with any of the changes you suggested. That being said I was a little confused on some of the setup. 
To reiterate, this was done on a

Starling2 with IMX412 M0161 attached via a coax cable.

After a stock flash of Voxl-SDK-1.4.0, running 

logcat | grep -i tuning

would return com.qti.tuned.imx412_fpv.bin as being loaded. However, your response seemed to imply that com.qti.tuned.cmk_imx577.bin would be loaded after a stock flash. After removing com.qti.tuned.cmk_imx577.bin there seemed to be no change. Logcat still implied com.qti.tuned.imx412_fpv.bin was being loaded. However, I did also try removing com.qti.tuned.imx412_fpv.bin which then indeed resulted in com.qti.tuned.default.bin being loaded. I saw a difference in how bright the image was but not how sharp it looked.

I did another stock flash to reset everything and proceed with running your linking command. That looked same as stock. But as previously mentioned, if com.qti.tuned.imx412_fpv.bin was already being loaded, then linking com.qti.tuned.imx412_fpv.bin would probably look the same, which it did.

I also attempted to force the system to use com.qti.tuned.cmk_imx577.bin by replacing the contents of com.qti.tuned.imx412_fpv.bin with it. This provided an even brighter image, but still no noticeable improvement in sharpness.

I’ve included the pictures here and what driver I THINK was being used. To me it seems that for the Starling2 kernel that maybe com.qti.tuned.cmk_imx577.bin isn’t being loaded by default.

The downsampled images are hard to compare so I included a drive link again.

com.qti.tuned.imx412_fpv.bin (Stock??) VVV

com.qti.tuned.default.bin VVV

com.qti.tuned.cmk_imx577.bin (forced it to use (or at least I think I did))

VVV

Moving on, decided to do some more experimentation which was even more puzzling. We happen to have an

IMX412 M0107 that attaches via a ribbon

module that we attached on J6Lower (with a different breakout board). To achieve this had to do a re-flash and use Kernel #1 (not the Starling2 Kernel as it seems like that one is incompatible). This surprisingly produced great results, much brighter, and much much better sharpness. The difference is night and day compared to the coax M0161. This time, it does seem that com.qti.tuned.cmk_imx577.bin was the one being loaded as you implied might happen after a stock flash.

A little confusing on what exactly is going on here. Obviously they must be using different drivers as this test and our Starling2 test use different ports. (J6Lower vs J6Upper). Furthermore, these experiments required using different kernels.

Should there be this big a difference based on using a coax vs the ribbon (M0161 vs M0107) or is it because of the different kernels? Do you have any sample images from the Starling2 Hires (M0161) with coax so we know what we should be expecting?

What we saw with the ribbon (M0107) is what we are aiming for but with the coax (M0161).

Thanks again for the help, any insight would be much appreciated.

Drive Link (better for comparison in sharpness)
Drive Pictures

Recently I went about upgrading from the

M0024 IMX214

to the

M0161 IMX412
Lens Mount: m12
Lens Part No: 27629F-16MAS-CM.

However, initial tests left us somewhat puzzled. For the imx412 I used the latest VOXL2 platform release with SDK 1.4.0. I saw a significant degradation in quality of the image with the IMX412. For the IMX412 I used the preview frame at the full resolution (4056x3040). For the IMX214 I also used the preview frame but at (4208x3120) Posted below are some samples. I recorded them via voxl-logger

I believe the imx412 has a wider field of view so perhaps some degradation in quality it to be expected as it might have a slightly lower angular resolution? As perhaps fewer pixels are available per degree of FOV? That being said I wouldn't think this much worse. Similarly, the exposure looked much worse. A good target here is the donut box.

Here are two downsampled images since the forum wouldn't let me post the full images. For the full images straight from voxl-logger I included a link to them in google drive folder (granted google might also downsample them but you'll still be able to see the difference. Any help/input on the quality difference would be much appreciated. Not sure if this is to be expected? Maybe something is wrong with the driver?

IMX214

IMX412

[Google Drive link] (https://drive.google.com/drive/folders/1rZV6LM5qSqLjSTxz58Eq4tsZ0AO3cMxz?usp=sharing)

I noticed you can only run the TOF at a rate of 5fps or 15 fps in voxl-camera-server. Is there a specific reason for this. Why not 10fps, for example? Just curious. For my specific application 10 is really all I need.

Hi!

What frame rates does the VOXL2 support with OV9782 cameras? Does it depend on the configuration? Max it seemed I could bump it up to was 30 fps but maybe that was only in my specific configuration (I'll note not one of the officially supported ones!).

Which frame rates are supported and is there a max frame rate? Is there a list I can view somewhere I could view of sensors and supported frame rates? I didn't see fps on the spec sheets.

Also, I am assuming this is how this works but does voxl-qvio-server run at the rate the camera it is using is set to? I know in previous posts it has been mentioned that bumping the fps doesn't make a huge impact. That being said, I did run an AR0144 camera at 60 fps and do I think I saw a bit of an improvement compared to 30. However, the results are a little anecdotal. Observed slightly better quality, num features, and less dropouts in difficult enviroments.

My plan is to run the OV9782 at 60 fps if supported and makes a difference.

Thanks!

@Eric-Katzfey a TOF ranging sensor - VL53L1CB by ST