Sounds good. Please keep in mind that if you relocate the gps receiver, you should update the offsets for gps sensor position in px4 (prior to flight)
My test will be putting a 4-6in mast either where the gps receiver is now (in the back) or right in the middle, on top of the “roll cage”.
Alex
Hi @ROBERT-JUDD,
We are actively testing to find a solution (including myself, I just got my hands on a Starling 2 Max a few days ago and was able to confirm similar SNR to what you have experienced - my drone is without 5G modem). My colleague is testing one with a 5G modem. We acknowledge that there is an issue and we are working on a short-term workaround and a longer term solution.
We understand that this can be concerning and if you are not able to wait for a solution, please reach out to https://www.modalai.com/pages/rma and we can work on a return / refund.
If you are able to work with us on this issues, we are trying out different options including different gps units, disabling different components to find the source of the interference, and also re-locating the position of the gps unit to help reduce the effect of the noise. My gut feeling is that elevating the GPS receiver on a lightweight but rigid mast (4-6in) above the drone and include some shielding right below the receiver should significantly improve the performance (as a short-term workaround). I will test this ASAP and provide the results and pictures). Please give me a few more days.
Alex
@paul-ammer , yes thank you. The hires_front camera on Starling 2 Max is the IMX412 camera, specifically M0161 module (with IR cut filter) : https://www.modalai.com/products/msu-m0161?variant=48860038365488
Can you also please let me know if you are looking to record a higher resolution video (what resolution) while streaming a smaller resolution via wifi? This is important because it will determine whether we can use a lower resolution mode directly on camera, which reduced delay, as opposed to getting a very large 4K image and then down sampling it just for the smaller image. I will explain what I mean shortly in my detailed write up (sorry, taking a bit longer than expected).
Alex
@Taka , as you are aware that this configuration is not supported, so you would be testing it at your own risk (in terms of drone flying normally after all the changes and potential issues with motor overheating). Electrically, Starling 2 is compatible with 3S. The entry point of power into Starling 2 is the mini ESC, which is rated for up to 4S Lipo.
https://docs.modalai.com/voxl-mini-esc-datasheet/
Now, regarding the motor.. you would need to do a ESC motor calibration using the following procedure : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-esc/-/blob/master/voxl-esc-tools/calibration.md?ref_type=heads (mainly update the battery voltage, test and update rpm limits, calibrate the feed forward (pwm_vs_rpm) curve, probably lower (a little bit) start-up power, confirm start up is smooth). You may also want to slightly lower the kp term (proportional term in the rpm controller, to account for increase in battery voltage, so that the controller does not become more aggressive). Maybe lower that from 100 to 66-75 or so.
These should be the ESC params that are loaded on your ESC : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-esc/-/blob/master/voxl-esc-params/D0014_starling_1504_3000kv_ms_props/D0014_starling_1504_3000kv_ms_props.xml?ref_type=heads . You can double check this using voxl-esc-verify-params.py script and then modify these params with your changes. The esc tools are already installed on voxl2 in /usr/share/modalai/voxl-esc-tools.
The main issue you may see is if you apply full power to the motor continuously, it may overheat due to the fact that it was designed for 2S. However, you can artificially cap your maximum rpm (in esc params) to prevent the motor from going to higher rpms (and update your px4 params for max rpm accordingly).
Depending on what you are trying to achieve, you will probably have to put some SW protection and prevent the motor from overheating.
If you are OK with waiting for a few days, I can give the motor a spin at 3S and just confirm nothing BAD will happen.
Alex
Are you asking about a specific camera? (which one)
In general, you should not specify max exposure (in camera server config) which exceeds 1/fps. This can be often overlooked, so we recently added a check that enforces maximum exposure to be slightly less than 1/fps : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/commit/aa3ce587f6be769e1b07edf57f66e1eec40d4644
However, in some special cases, there may be other limitations. One case that I know of is the synced AR0144 at 30fps has a maximum exposure of less than half of 1/fps, otherwise it will skip frames, so our defaults for AR0144 are set to have 12ms maximum exposure. Also, you don't want to set the exposure too high for tracking cameras, since you will get too much motion blur.
Alex
@paul-ammer , i am going to write up a document with some tips for getting low latency streaming working. Let me get back to you tomorrow.
I will also answer original questions from Dan.
Can you please clarify which camera you are using?
Alex
@AidanGallagher , you are welcome.
Just one note, if you are going to try 4040x3040 maximum fps, please choose something slightly under 60, as 60 is sometimes causing issues due to very little time between end of frame and start of frame, but even 55 is going to be stable. When I use this resolution, i test at 50 or 55FPS. I will look into 60FPS at full resolution a bit later, but may need to cut down on the vertical size a bit to give some time between EOF and SOF.
Alex
@dan64 ,
Can you please specify what resolution and FPS you need for your low latency stream? Also, are you going to simultaneously record a higher resolution stream no VOXL2 (not sent via WIFI).
which encoding are you using h264 or h265 and approximate desired bit rate? Please note for h265 you will need about half the bandwidth of h264, so h265 is strongly recommended if it is supported on the receiving end.
Also, what camera are you using?
We can provide guidance for achieving minimum latency with your specifications.
Alex
Hi @AidanGallagher , can you please clarify what resolution and fps you are trying to achieve using the "fpv" version of the IMX412 drivers? Please note this version of drivers is still "experimental" but I can help you get the results that you need.
When using MISP (Modalai ISP) pipeline, the 4056 resolution currently does not work (long story). But we have a workaround for using 4040x3040, which is just 16 pixel crop from the full frame. If that would work for you, you should pick up the latest drivers for MISP here :
You will simply need to replace the sensormodule file for the slot number that you have for your imx412 (in /usr/lib/camera/). and might as well update the .so that's in the zip to overwrite the existing one in the same location)
The following RAW resolutions are supported by these drivers:
- 4056x3040 @ 30 16.5ms readout time
- 4040x3040 @ 30, **60** 16.5ms readout time
- 3840x2160 @ 30, 60, 80 11.8ms readout time
- 1996x1520 @ 30, 60, 120 5.5ms readout time
- 1936x1080 @ 30, 60, 90, 120, 240 4.0ms readout time
- 1996x480 @ 30, 480 1.8ms readout time
- 1996x240 @ 30, 800 0.9 - 1.0 ms readout time
*** Please note 4040x3040 60FPS is not stable, please use slightly lower FPS like 55 or even 57-58 should be OK if needed.
1996x1520 is the 2x2 binned resolution of full frame with a small horizontal crop to make it compatible with MISP.
The readout times are noted as they are important if you are concerned with overall latency.
These drivers are configured to use IMX412 at its maximum throughput of 2.1Gbps per MIPI lane (4 lanes), which ensures the lowest latency and maximum possible FPS.
When you use this driver, please set the following value in /vendor/etc/camera/camxoverridesettings.txt - this is a workaround for camera pipeline not showing all possible RAW resolutions / fps.
maxRAWSizes=20
Also, we recently (I believe SDK 1.4.1) disabled debug prints from camera pipeline to logcat, so if you wanted to confirm that correct resolution is selected (see below), enable the following (in the same file), if the param is set to 0 (need to set to 1):
systemLogEnable=1
After confirming the resolution, you can set the param back to 0 to reduce cpu usage.
You should not use 4056x4030, but instead use 4040x3040. if you use 1936x1080, then you probably want to specify misp_width as 1920, not 1936. The reason for 1936 is the same as for 4056 -- MISP does not work with 1920, but we just crop 1936 down to 1920. There are no performance issues associated with this.
Please note the 480 and 800fps modes are experimental, may cause camera server to crash, depending on what you are doing (especially 800fps). also if you try to stream these high fps to voxl-portal, you will have really large frame drop, but you will see some frames. (maybe up to 120fps, since frames are small). For high fps modes, set cpu to perf using voxl-set-cpu-mode perf.
Finally, for best performance you should use the latest camera server from our nightly builds, if you do not, please specify which version you are using and I can check if there are any relevant issues related to what you are trying to do.
In camera server config, you should specify one of the above resolutions as your preview_width and preview_height and misp_width and misp_height can actually be pretty much any resolution you want. Based on the misp resolution that you request, the source image will be down-scaled and cropped to match your dimensions and maintain the proper aspect ratio (no stretching, etc).
Finally, if you are experimenting with different modes and FPS, and the image you get looks completely wrong (zigzags, etc), you should verify that the camera pipeline indeed selects correct resolution by running the following before you start camera server:
this will show something like this:
logcat | grep -i selected
03-19 20:26:09.694 1894 1894 I CHIUSECASE: [CONFIG ] chxsensorselectmode.cpp:635 FindBestSensorMode() Selected Usecase: 7, SelectedMode W=4040, H=3040, FPS:30, NumBatchedFrames:0, modeIndex:0
03-19 20:26:09.694 1894 1894 I CHIUSECASE: [CONFIG ] chxpipeline.cpp:371 CreateDescriptor() Pipeline[Raw] Pipeline pointer 0x55ae2d6620 Selected sensor Mode W=4040, H=3040
03-19 20:26:09.694 1894 1894 I CamX : [CONFIG][CORE ] camxsession.cpp:5907 SetRealtimePipeline() Session 0x55ae411ea0 Pipeline[Raw] Selected Sensor Mode W=4040 H=3040
if the selected resolution is not what you reqested as preview (this may happen if you select 1996x240 resolution but choose FPS equal or below 480, the pipeline will choose 1996x480 resolution (because it is closer in FPS...), which may cause the board to crash (because buffers are allocated for 1996x240 resolution). So if you use use high FPS modes, either use the exact FPS that i listed or use FPS above the the higher resolution modes. I hope that makes sense.. This is an issue we are going to work out soon. If you want to avoid crashing the board (which is temporary, just power cycle to recover). you should start the camera server but do not have any clients for the hires camera stream, which would not actually start the camera streaming, but you will see the resolution selected and can stop the camera server before it would cause a crash.
In the near future we will update these experimental binaries and make them the default, as they are backwards compatible with existing drivers that are shipped with the SDK.
Alex
@Yoshimaru-Tanaka , lets focus on the case when only IMX412 is enabled.
After you start the camera server and run voxl-inspect-cam -a (and you see no frames), does camera server abort?
Can you check what is the output of dmesg, perhaps you can paste the output after the camera starts streaming, which would look similar to this (so you can see if there are any errors after this print):
[ 1455.171322] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 997 CAM_ACQUIRE_DEV Success, sensor_id:0x577,sensor_slave_addr:0x34
[ 1455.685718] msm_vidc: high: 00000001: ....e: Opening video instance: ffffffdc7a86c000, 0
[ 1455.897358] CAM_ERR: CAM-ISP: cam_ife_hw_mgr_print_acquire_info: 710 Successfully acquire single IFE[6 -1] with [0 pix] [0 pd] [1 rdi] ports for ctx:4
[ 1455.902246] CAM_INFO: CAM-CSIPHY: cam_csiphy_core_cfg: 1137 START_DEV: CSIPHY_IDX: 0, Device_slot: 0, Datarate: 2500000000, Settletime: 2200000000
[ 1455.910641] CAM_INFO: CAM-ISP: cam_vfe_bus_ver3_init_hw: 3659 Overriding clock gating at bus input
[ 1455.910644] CAM_INFO: CAM-ISP: cam_vfe_top_ver3_init_hw: 246 Disable clock gating at IFE top
[ 1455.910795] CAM_ERR: CAM-ISP: cam_ife_mgr_start_hw: 4510 ->Config HW, 000000008601d43d
[ 1455.911552] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 1089 CAM_START_DEV Success, sensor_id:0x577,sensor_slave_addr:0x20
Also, is your drone set up with 3 tracking cameras, configuration C27 and the cameras are plugged into correct locations based on the diagram? https://docs.modalai.com/voxl2-coax-camera-bundles/
Alex
@voxl-phoenix , I think you also mentioned that you tried swapping the cables between tracking and imx412 camera, right? If so, it seems something got damaged in the IMX412 camera during connecting / disconnecting. For disconnecting the cables, we suggest using a special tool, please see https://docs.modalai.com/micro-coax-user-guide/#-proper-cable-and-adapter-handling- .
Also, it is hard to say for sure where exactly the issue is - M0173 board or IMX412 camera. the only good way find the cause is to replace components one at a time.
Just to elaborate on the symptoms that you are seeing : voxl-camera-server -l command essentially probes all the cameras (based on the sensormodule files located in /usr/lib/camera). Probing involves turning the camera on via the reset GPIO and attempting to talk to the camera via I2c interface. If the camera responds, the probe is successful, otherwise failed. In your case, the probe of IMX412 failed, which means either the camera could not be turned on for various reasons or the i2c communication was not working.
If you have another tracking camera AR0144, you could connect it instead of the IMX412 and see if the probe succeeds. You would need to add /usr/share/modalai/chi-cdk/ar0144/com.qti.sensormodule.ar0144_1.bin file to /usr/lib/camera (which is the AR0144 sensormodule for the camera slot 1, where the IMX412 is currently connected). Then if you run voxl-camera-server -l and detect two AR0144 cameras (3 cameras total) then i2c communication and power should be working).
Alex
Can you please stop voxl-camera-server service and run the camera server in foreground to make sure that all cameras are started:
systemctl stop voxl-camera server
voxl-camera-server
If there is a problem, you should see it in the terminal output. I am surprised that voxl-inspect-cam -a does not produce any output, which means none of the cameras are alive, so you should see something in the output from voxl-camera-server process.
Perhaps your board is not correctly configured for the cameras, so you may want to run voxl-configure-cameras and select the correct option for your camera setup. Then re-start voxl-camera-server.
Alex
@voxl-phoenix , the connections look correct. Can you please clarify whether this setup was working (hires camera was working) and then suddenly stopped working? Or what changed between the time when it was working and now.
Alex
Ohh, sorry for the confusion. I misread your original post and thought that you have 4 cameras connected, like C26 configuration.
Can you please disconnect your M0173 from VOXL2 and take a picture showing the connectors on M0173 and where each camera is connected? just to double check.
Alex
@voxl-phoenix , can you please confirm something - did you mention that this exact hardware setup was previously working with 4 cameras and now two cameras are no longer detected?
Regarding camera calibration, it seems the issue is that the calibrator is attempting to calibrate extrinsics (which is only done for a stereo camera setup), but you are only calibrating one camera. Let me look into that.
Alex
Can you please list
/usr/lib/camera/*sensormodule*
@voxl-phoenix please make sure the cameras are connected as shown in C26 configuration https://docs.modalai.com/voxl2-coax-camera-bundles/
@voxl-phoenix , if you use VOXL2 with the M0173 camera front-end, when you install the VOXL2 SDK, you need to select the option 2, which enables M0173:
PLEASE SELECT WHICH KERNEL TO FLASH
The reason for this is the M0173 has different mapping of various GPIO that controls camera functions (including on/off). The camera GPIO control is done from the Kernel, so that is why a different kernel needs to be loaded if you use M0173.
If you don't want to re-install the whole SDK, you can update the kernel manually:
Test the new kernel without overwriting (will not persist after reboot):
adb reboot bootloader
fastboot boot m0054-1-var00.1-kernel.img
Then you can adb into the board and do a quick test to see if all cameras are detected : voxl-camera-server -l will print at the top all detected cameras (in your case should be 4 cameras).
To permanently flash the new kernel:
adb reboot bootloader
fastboot flash boot_a m0054-1-var00.1-kernel.img
fastboot flash boot_b m0054-1-var00.1-kernel.img
fastboot reboot
(in order to flash the standard kernel, use m0054-1-var00.0-kernel.img)
In order to check which kernel variant you are currently running:
voxl2:/$ voxl-version | grep mach
mach.var: 1.0.0
# this is the default kernel, not for use with M0173
or
voxl2:/$ voxl-version | grep mach
mach.var: 1.0.1
# this is the kernel for use with M0173
If you are using M0154 (not M0054), It uses the same kernel as M0054, and will be detected as such.
Hopefully this solves your issue.
@Aaron-Porter , the voxl-logger does not control what format the images are coming from voxl-camera-server. Currently, voxl-camera-server can output images in RAW8, YUV format and JPEG only in the case of a snapshot (not continuous). So, to answer your first question, the camera server does not support outputting PNG or JPEG in continuous streams.
voxl-logger already supports saving images in png or jpeg format, depending on the type of image that is being logged (RAW8 mono or YUV color, etc). You could tweak the behavior to your needs or add a flag to voxl-logger to force a logging format : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-logger/-/blob/master/tools/voxl-logger.cpp?ref_type=heads#L1455 .
Alex
Hi @ROBERT-JUDD , my colleague is setting up the exact test with Starling 2 Max with the 5G modem to see if there is any possible interference. We will get back to you soon. Meanwhile, please let us know if swapping of the GPS module produces any positive results. Thank you!
Alex