Boson 640 MIPI M0153 16-bit

mkriesel

Re: Boson 640 MIPI M0153: 16-bit Pre-AGC

Hello, I saw this post and I'm in a similar boat where I'm looking at using a boson over mipi. Is there any update that has been done or is planned with supporting 16 bit mipi and possibly 60 hz update (less important but would still be nice)?

Alex Kushleyev

@mkriesel , we were able to get the 14 bit data from Boson after performing a firmware update on our old Hadron unit.

Enabling this will require updated voxl-camera-server, a new Boson sensormodule file as well as a change in the Boson configuration.

We were not able to use the FLIR GUI to switch Boson to 14bit MIPI, but using their python API, we were able to do it over USB (while connected to a linux PC).

While outputting 14 bit data, which is RAW output from the sensor, you cannot simultaneously get the post AGC 8 bit data, so you would need to do your own processing on the 14 bit data.

Our plan is to enable the RAW data publishing first along with perhaps a simple post-processed version of the data. The data will be published in uint16 format, so that you won't have to do the 14->16 bit unpacking.

Here are the Boson versions that worked / did not work to enable 14 bit MIPI data.

Hadron (640x512)
Before update: Software Version 3 0 26416 (not able to switch to 14 bit mode via gui/ python api)
After update: Software Version: 4 1 27660 (able to switch to 14 bit mode via python api and get data on voxl2)

Small Boson (320x256):
Software Version: 4 0 27141 (able to switch to 14 bit mode via python api and get data on voxl2)

Update is coming probably next week.

Regarding the 60Hz, actually we made some improvements in camera server a while ago and 60Hz is working now. You can just update your Boson FPS using the GUI and try it. You should update the FPS in voxl-camera-server.conf to 60 as well, so that if you are using the encoded output, the encoder knows what FPS to expect (remember, the camera server cannot actually change the FPS of the Boson).

Finally, we were previously using the ISP to process the Boson data, which made the image look a bit worse. Please make sure to use MISP when testing boson and disable the small / large encoded streams.

Here is a sample config:

{
			"type":	"boson",
			"name":	"boson640",
			"enabled":	true,
			"camera_id":	2,
			"fps":	30,
			"en_preview":	true,
			"en_raw_preview":	true,
			"en_misp":	true,
			"preview_width":	640,
			"preview_height":	512,
			"en_snapshot":	false,
			"ae_mode":	"off",
			"en_rotate":	false,			
			"misp_width":	640,
			"misp_height":	512,
			"misp_venc_enable":	true,
			"misp_venc_mode":	"h264",
			"misp_venc_br_ctrl":	"cqp",
			"misp_venc_Qfixed":	30,
			"misp_venc_Qmin":	15,
			"misp_venc_Qmax":	50,
			"misp_venc_nPframes":	29,
			"misp_venc_mbps":	2,
			"misp_venc_osd":	false
		}

In order to use MISP for boson, you need to have the camera server which includes the following commits: -- see here : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commits/master?ref_type=heads

By the way, there is now also a Boson color stream (heat map) which is generated from the original 8 bit post AGC image. You can see an example in this forum thread : https://forum.modalai.com/topic/4757/connecting-2-boson-cameras-and-2-mipi-cameras-4-cameras-total (where there are two bosons, both running at 60fps, by the way)

Alex

mkriesel

@Alex-Kushleyev I only need the RAW output I plan to do all the post processing myself so this is exactly what I need, Thanks!

Alex Kushleyev

Hello @mkriesel ,

Please see below instructions for test 14 bit Boson data stream on VOXL2:

voxl-camera-server : use branch https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/tree/add-boson-14bit-support
latest Boson drivers (8 bit unchanged, added 14 bit drivers) : link
python scripts to read Boson config, set to 8 bit, set to 14 bit :

Boson 14 bit release notes

14 bit data is published at RAW16 format (unpacked from 14 bit).
The boson_bayer image stream can be viewed with voxl-portal, but will appear dark grey because the data is not scaled to full 16 bit range. voxl-portal will just convert 16->8 bit with a bit shift.
8-bit sensormodule has to be used to receive 8 bit data : com.qti.sensormodule.boson_X.bin (replace X with camera slot id)
14-bit sensormodule has to be used to receive 14 bit data : com.qti.sensormodule.boson_14bit_X.bin (replace X with camera slot id)

Configuring Boson to 14 bit:

use SDK4.0_Boson_Plus_SDK
connect Boson to Linux PC via USB
run python3 boson_read.py to check the current settings
run python3 boson_set_14bit.py to set to 14 bit output (check FPS setting in the script)
run python3 boson_set_8bit.py to set to 14 bit output (check FPS setting in the script)
using incorrect sensormodule (which does not match the bit resolution actual Boson output),
will result in the following message in camera server and no data will be received from Boson:

ERROR:   Received "Buffer" error from camera: boson

30 or 60 FPS works with 8 and 14 bit. The fps setting in camera server config does not change FPS, but you should make the value in the voxl camera server config file consistent with the actual setting (otherwise video encoder bandwidth may be incorrect, if you use that)
using Boson configured for 14 bit with version of camera server which does not support 14 bit will result in VOXL2 board crash due to insufficient buffer size allocated for the incoming frames (buffer overflow)
use the camera server config shared above (with misp enabled for Boson camera)

voxl-camera-server will detect the frame size and output either:

camera boson640 detected frame size: 640x512 327680 bytes (819200 alloc).. raw bpp: 8

or

camera boson640 detected frame size: 640x512 573440 bytes (819200 alloc).. raw bpp: 14

Please give it a try. We will merge this to the dev branch of camera server soon.

Alex

Alex Kushleyev

Also, a quick update.. after some additional testing, we are actually able to communicate with Boson via the CCI connection directly from VOXL2. Hopefully we will be able to enable (partial) Boson configuration from VOXL2 directly, avoiding the use of the USB connection for simple configuration change.

Here is a sample script that can be used to test triggering FFC. The script uses cci-direct library which allows to communicate to cameras while they are running (in camera server).

Before running this script, make sure the camera server is running and it is configured to enable Boson camera (and take note of the camera id (sw id, not slot)), which is needed by cci-direct and the script. This is also abit of a hack since we are writing part of the payload as the the address in addressed write (as it is common in i2c transactions), but Boson's i2c interface does not support addressing.

#!/bin/bash

if [ "$#" -eq 0 ]; then
	echo "Error: No arguments provided."
	echo "Usage: $0 <boson camera id>"
	exit 1
fi

cam_id=$1

#command to trigger FFC
#(extra two 00's at the end to make it a multiple of 3, since we are sending 3 bytes at a time in this test)

data="8E A1 00 0C 00 00 00 FC 00 05 00 07 FF FF FF FF 00 00"

IFS=' ' read -r -a words <<< "$data"

for (( i=0; i<${#words[@]}; i+=3 )); do
	echo "${words[i]} ${words[i+1]} ${words[i+2]}"
	voxl-cci-direct -c ${cam_id} -w 0x${words[i]}${words[i+1]} 0x${words[i+2]}
done

mkriesel

@Alex-Kushleyev Cool, follow up then the Boson has an external sync pin, does the mipi board support this still?

Alex Kushleyev

@mkriesel , our M0153 / M0201 adapter does have an option to connect Boson to the common sync line that is used for synchronizing AR0144 tracking cameras (GPIO 109). This line is typically driven at 30Hz, but the frequency is adjustable.

There is a resistor pad on M0153 / M0201 adapter which is DNI by default, so a 0-ohm resistor would need to be installed to enable this functionality. However, we have not verified that it works.

It is on our to-do list to verify the Boson sync. Once we confirm that it works, I can provide instructions for installing the resistor if you wanted to enable that functionality. Boson's configuration will also need to be updated to enable the sync input, but it's not a big change.

Let me get back to you next week..

Alex

Alex Kushleyev

@mkriesel ,

I did a basic test of configuring Boson for external sync using bosonSetExtSyncMode(FLR_BOSON_EXT_SYNC_MODE_E.FLR_BOSON_EXT_SYNC_SLAVE_MODE) in the FLIR python API.

If the sync signal is absent, then Boson just stops sending frames (as expected). Once I installed the missing resistor on the ModalAI Boson adapter to patch the common 30FPS camera sync signal from VOXL2 to Boson, I started receiving frames from Boson at 30FPS. Adjusting the sync signal from 30 to 60FPS resulted in some FPS instability (as well as some minor flickering artifacts) in Boson frames due to the fact that the sync signal coming out of voxl2 has some jitter. Boson timing at 60FPS is very tight, so sync signal coming in too early can cut off the previous frame, so i believe that was happening.

I also noticed that when the sync signal was at 30FPS, the FFC would take twice as long, which is interesting. I wonder if Boson indeed expects 60FPS sync signal and perhaps something may not work properly otherwise. will need to check Boson documentation..

Also keep in mind that the tracking cameras (AR0144) cannot operate at 60FPS in sync'ed mode (different story). So, if you are using sync'ed AR0144 tracking cameras and would like to use 60FPS sync'ed Boson, this may not be possible with the same sync line.

I will need to experiment with this some more. Can you please elaborate on the use of Boson sync in your application (and whether you need VOXL2 to generate the sync signal or you have your own sync signal)?

Alex

mkriesel

@Alex-Kushleyev

Thanks for looking into this. At the moment there aren't any plans using the sync pin but we are trying to see if this is something worth looking into.

Back to testing, I recently got the boson board in and am trying to test it but have run into an issue with trying to get the camera-server installed.

I'm getting
voxl-camera-server depends on libmodal-pipe (>= 2.14.0); however:
Version of libmodal-pipe on system is 2.13.2.

Then for libmodal-pipe 2.14.0
dpkg: regarding libmodal-pipe_2.14.0_arm64.deb containing libmodal-pipe:
libmodal-pipe conflicts with voxl-cpu-monitor (<< 0.7.0)
voxl-cpu-monitor (version 0.6.0) is present and installed.

Bu voxl-cpu-monitor 0.7.0
dpkg: regarding voxl-cpu-monitor_0.7.0_arm64.deb containing voxl-cpu-monitor:
voxl-cpu-monitor conflicts with libmodal-pipe (<< 2.14.0)
libmodal-pipe (version 2.13.2) is present and installed.

So I'm stuck in a bit of a circle with the libmodal-pipe if you could help out

Alex Kushleyev

@mkriesel , oh yeah, i now that dpkg stuck issue..

use dpkg --force-all -i <package.deb> , you can install voxl-cpu-monitor and libmodal-pipe together or separately.

Alex

mkriesel

@Alex-Kushleyev
Alright, I've tried a few things and I can't see to get things to work.

voxl2-mini:~$ voxl-camera-server
Setting MISP AWB to Auto
MISP channels enabled in defaults : 0
MISP channels enabled in config file: 0

configuration for 1 cameras:

cam #0
name: boson
type: boson-fpv
bayer_type: 0
enabled: 1
camera_id: 0
camera_id_second: -1
fps: 30
en_rotate: 0
en_rotate2: 0

en_preview:          1
en_raw_preview:      1
preview_width:       640
preview_height:      512

en_misp:             1
misp_width:          640
misp_height:         512

en_small_video:      0
small_video_width:   640
small_video_height:  512

en_large_video:      0
large_video_width:   -1
large_video_height:  -1

en_snapshot:         0
snap_width:          -1
snap_height:         -1
exif_focal_length:   0.000000
exif_focal_len_35mm_format:0
exif_fnumber:        0.000000

ae_mode:             lme_msv
msv_exposure_min_us: 20
msv_exposure_max_us: 33000
gain_min           : 54
gain_max           : 8000
standby_enabled:     0
decimator:           1
independent_exposure:0

fsync_en: 0
fsync_gpio: 109

thread is locked to cores: 4 5 6 7
connected to mavlink pipe
Starting Camera: boson (id #0)
Opened GBM fd
gbm_create_device(156): Info: backend name is: msm_drm
Created GBM device
MISP Initializing for camera boson
Detected 1 platform(s)
Detected 1 GPU device(s)
Estimated imu dt = 0.000977s
Got unsupported format in getUVStartFromFmt, returning nullptr
WARNING: OMX SetTargetBitrate: H265 CBR requires bps >= 3.0Mbit (2000000 bps provided). Using FPS hack. scale = 1.500000

------ voxl-camera-server: Started 1 of 1 cameras

------ voxl-camera-server: Camera server is now running
ERROR: Received "Result" error from camera: boson
ERROR: Received "Buffer" error from camera: boson
ERROR: Received "Result" error from camera: boson
ERROR: Received "Buffer" error from camera: boson
ERROR: Received "Result" error from camera: boson
ERROR: Received "Buffer" error from camera: boson
ERROR: Received "Result" error from camera: boson
ERROR: Received "Buffer" error from camera: boson
ERROR: Received "Result" error from camera: boson
ERROR: Received "Buffer" error from camera: boson
ERROR: Received "Request" error from camera: boson

I copied the drivers into /usr/lib/camera/ and both the 8 bit and 14 bit drivers you provided gave the same error running various combinations of the set and read python scripts you sent.

(base) ➜ boson python3 boson_set_14bit.py
C serial FSLP load
Serial port open
Software Version: 4 0 27141
camera SN: 322022
dvoSetOutputFormat:
0
dvoSetType:
0
dvoSetOutputIr16Format:
0
roicSetFrameSkip:
0
Apply Custom Settings:
0
Save Settings To Flash:
0
Reboot
0
Closing com port and freeing serial port instance.
Serial port close
(base) ➜ boson python3 boson_read.py
C serial FSLP load
Serial port open
Software Version: 4 0 27141
camera SN: 322022
dvoGetType:
0
5
dvoGetOutputFormat:
0
3
dvoGetOutputIr16Format:
0
0
dvoGetLCDConfig:
0
0
ClockInfo:
0
60.0
27.0
16

Let me know if it seemed like I did something wrong, I tried both the J6 and J7 ports on the voxl 2 mini with the corresponding numbered driver.

Alex Kushleyev

@mkriesel

Please make sure that you only have one sensormodule file for boson for the given configuration in /usr/lib/camera/.

One thing to check is to see if the Boson is in active mipi state by default. You can add this to your boson_read.py file :

print('dvoGetMipiStartState: ')
result, mipi_start_state = myCam.dvoGetMipiStartState()
print(result)
print(mipi_start_state)

print('dvoGetMipiState: ')
result, mipi_state = myCam.dvoGetMipiState()
print(result)
print(mipi_state)

My output is:

dvoGetMipiStartState: 
FLR_RESULT.R_SUCCESS
FLR_DVO_MIPI_STATE_E.FLR_DVO_MIPI_STATE_ACTIVE
dvoGetMipiState: 
FLR_RESULT.R_SUCCESS
FLR_DVO_MIPI_STATE_E.FLR_DVO_MIPI_STATE_ACTIVE

FLR_DVO_MIPI_STATE_ACTIVE is 2, FLR_DVO_MIPI_STATE_OFF is 0

If your sensor does not show active start state, then it may need to be set using the following:

myCam.dvoSetMipiStartState(FLR_DVO_MIPI_STATE_E.FLR_DVO_MIPI_STATE_ACTIVE)

If this still does not work, it is possible that my script does not do the full setup for mipi output, so you may want to try using the FLIR GUI for the initial setup (for 8 bit post AGC and use the default 8 bit sensormodule): https://docs.modalai.com/M0153/#boson-software-setup

Alex

Alex Kushleyev

@mkriesel , I have updated the boson configuration and read scripts (boson_read.py, boson_set_8bit.py, etc) to include configuring additional settings like mipi start state, dvo output interface, etc. Please download them again (from links earlier in this thread) and try again. Also, just to confirm that the original Boson setup instruction using the FLIR GUI should work for 8 bit format.

Just one note, even though I set myCam.dvoSetOutputFormat(FLR_DVO_OUTPUT_FORMAT_E.FLR_DVO_IR16), where FLR_DVO_IR16 = 3, when i read back the configuration, i get the default format (2) : FLR_DVO_OUTPUT_FORMAT_E.FLR_DVO_DEFAULT_FORMAT = 2 . It seems that in your output the ouput format is set to 3 - i am not sure why, there could be some minor quirks in the boson firmware.

In any case, please try the GUI and updated scripts and let me know how it goes!

Alex

mkriesel

@Alex-Kushleyev It seems to be working now, I'll do more testing but this looks correct

| Pipe Name | bytes | wide | hgt |exp(ms)| gain | frame id |latency(ms)| fps | mbps | format
| boson | 327680 | 640 | 512 | 0.00 | 799 | 163 | 21.6 | 60.0 | 157.3 | RAW8
| boson_bayer | 655360 | 640 | 512 | 0.00 | 0 | 163 | 20.0 | 60.0 | 314.6 | RAW16
| boson_color | 983040 | 640 | 512 | 0.00 | 799 | 163 | 22.7 | 60.0 | 471.9 | RGB
| boson_misp_encoded | 8826 | 640 | 512 | 0.00 | 799 | 163 | 25.5 | 60.0 | 4.1 | H265 (P)

Thanks for the help! Also, what version of the bosonSDK are you using for your python scripts? My version (3.0) doesn't have the dvoSetMipiStartState and a few other functions and I can't find a higher version on their website?

Alex Kushleyev

@mkriesel , I am glad that you got it working! You should also be able to view the boson output in voxl-portal -- for the 14 bit case, there is a very simple re-scaling done from 14 to 8 bit, just for testing (WIP).

I am using SDK4.0_Boson_Plus_SDK. You should probably contact Flir to get access to it, since we can't distribute it.

Alex

mkriesel

@Alex-Kushleyev

I'm trying to pull the images off and they seem to be corrupted somewhat. It almost seems like I'm getting the 16 bit image but in an 8 bit format. I'm subscribing to the boson_bayer and the image I'm getting seems to be the correct size and length but when I save its not correct. The RAW 8 stream I get on the voxl-portal looks the same, which I would expect for testing.

Heres what I'm getting for reference
https://drive.google.com/file/d/1fG5XEPM-b1NbC6H9R4jWSvZTDxm-a6Y_/view?usp=sharing

And then code I'm using to save it

static void _frame_cb(__attribute__((unused)) int ch, camera_image_metadata_t meta, char* frame, void* context)
{

    CameraInterface* interface = (CameraInterface*)context;

    if (interface->GetState() != ST_RUNNING) {
        return;
    }

    if (interface->ShouldSkip()) {
        return;
    }

    if (meta.format == IMAGE_FORMAT_NV12) {

    }
    else if (meta.format == IMAGE_FORMAT_RAW8) {
     
    }
    else if (meta.format == IMAGE_FORMAT_RAW10) {

    }
    else if (meta.format == IMAGE_FORMAT_RAW16) {
        std::cout << "16 bit image " << meta.size_bytes << " stride " << meta.stride << std::endl;
        int input_height = meta.height;
        int input_width  = meta.width;

        size_t frame_size = input_width * input_height * 2;

        char outname[100];

        std::string image_folder_name = "/data/pgm_frames";

        // uint16_t* frame_data = reinterpret_cast<uint16_t*>(frame);

        sprintf(outname, "%s/frame_%06lu.pgm", image_folder_name.c_str(), meta.timestamp_ns);
        printf("Output filename: %s\n", outname);

        // out_image.write(reinterpret_cast<char*>(pixels), IMAGE_PIXELS * sizeof(uint16_t));
        std::ofstream out_image(outname, std::ios::binary | std::ios::out);

        out_image << "P5\n640 512 65535\n";
        out_image.write(frame, frame_size);

    }
    else {
        // Not implemented
        std::cout << "other" << std::endl;
    }
}

Any thoughts?