@dlee ,

Yes the new TOF sensor (IRS2975C) is more powerful that the previous generation. What I mean by that is that it can emit more IR power but also heats up more. Emitting more power allows the sensor detect objects at larger distances or objects that are not as reflective.

In current operating mode, the auto exposure control is enabled inside the sensor itself, which modulates the emitted IR power based on the returns that the sensor is getting. That is to say, the power draw will vary depending on what is in the view of the sensor. If there are obstacles nearby, the output power should be low, otherwise it can be high. At full power, the module can consume close to 0.8-0.9W

So the first solution, if design allows, is to add a heat spreader to dissipate the heat, which you already started experimenting with. The sensor has a large exposed copper pad in the back for heat sinking purposes for this exact reason. Just be careful not to short this pad to anything, use non-conducting (but heat transfering) adhesive pad between the sensor and heat spreader.

In terms of a software solution to the issue, we can query the temperature of the emitter. We can also control the maximum emitted power used by the auto exposure algorithm. That is to say, still leave the auto exposure running in the sensor, but limit the maximum power that it is allowed to use.

We are planning to add some software protection that limits the maximum output power as a function of the emitter temperature. This will require some implementation and testing.

Meanwhile, please consider using a heat spreader, which will be the best solution if you want to make use of the full sensor's operating range and not have our software limit the output power in order to prevent overheating.

Hello @Kashish-Garg-0

we have a curve that is "motor voltage vs rpm", meaning that for a desired RPM, it tells the ESC what average motor voltage should be applied. The average motor voltage is defined as battery_voltage * motor_pmw_duty_cycle. The battery voltage in this curve is in millivolts. Since you are typically controlling the desired RPM, as a user you do not need to worry about what "throttle" or voltage to apply - the ESC does this automatically in order to achieve the desired RPM. this calibration curve is used as a feed-forward term in the RPM controller. The ESC does support an "open loop" type of control where you specify the power from 0 to 100%, which is similar to a standard ESC, but PX4 does not use that ESC control mode.

By the way, you can test the ESC directly (not using PX4) using our voxl-esc tools (https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-esc/-/tree/master/voxl-esc-tools) which works directly on VOXL2 or a standalone linux PC (or mac). voxl-esc-spin.py has a --power argument where you specify the power from 0 to 100, which translates directly to the average duty cycle applied to the motor.

Here is the calibration for the Starling V2 motor / propeller that we use:
https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-esc/-/blob/master/voxl-esc-params/mavic_mini_2/mavic_mini_2.xml?ref_type=heads#L63

Also, you can take a look at this post to see how to interpret those parameters a0, a1, a2 : https://forum.modalai.com/topic/2522/esc-calibration/2

We also have some dyno tests for this motor / propeller : https://gitlab.com/voxl-public/flight-core-px4/dyno_data/-/blob/master/data/mavic_mini2_timing_test/mavic_mini2_modal_esc_pusher_7.4V_timing0.csv . We are not sure how accurate that is, but it can be used as a starting point. @James-Strawson can you please confirm that is the correct dyno data for the Starling V2 motors?

Alex

@reber34 , perhaps this approach can work for you:

• record a video encoded at high bit rate (using voxl-camera-server and voxl-record-video . Please note that the output of voxl-record-video will not be in a standard container (such as mp4, etc), but you can fix it with ffpeg : ffmpeg -r 30 -i voxl-record-video.h264 -codec copy videofile.mp4
• re-encode the video offline with desired codecs / bit rates / resolutions
• install gst-rtsp-launch which uses gstreamer to set up an RTSP stream https://github.com/sfalexrog/gst-rtsp-launch/
  • you will first need to figure out what gstreamer pipeline to use on voxl2 that will load your video and parse the h264/h265 frames (can use null sink for testing) and then use that pipeline with gst-rtsp-launch which will take the encoded frames and serve them over rtsp stream.
• gstreamer may be more flexible for tuning the encoding parameters of h264/h265 (compared to voxl-camera-server) and you can also use it in real time later (using voxl-streamer, which uses gstreamer under the hood)

Another alternative is to use voxl-record-raw-image to save raw YUVs coming from voxl-camera-server and then use voxl-replay and voxl-streamer - the latter will accept YUVs from the MPA pipe and encode them using the bit rate that you want. Note that depending on the image resolution, YUV images will take a lot more space than encoded video, but maybe that is also OK since VOXL2 has lots of storage.

Alex

Hello Rolando,

It will not be possible to fully use four ov7251 cameras at the same time due to i2c address conflict. the downward camera can coexist with the stereo pair because

• one of the stereo cameras is actually on a different i2c bus
• ov7251 camera has two i2c addresses, so with some software configuration it works. However, having 3 ov7251 cameras on i2c bus cannot be done without a conflict.

In a condition of i2c conflict between cameras, what will happen is..

• system will be able to detect each camera at boot time because probing happens while other cameras are powered off
• you should be able to fully use the conflicting cameras one at a time (not concurrently)
• if you use the conflicting cameras concurrently, commands sent to one camera will go out to both cameras, for example exposure and gain control. nothing will physically break, but this is currently not a supported use case.
• there will also be a short glitch after you started the first conflicting camera and when you are starting the second conflicting camera, since the first one will also get re-initialized with the commands intended for the second camera

in your case, the ov7251 plugged into J2 was not detected because the camera configuration xml file is not configured to query this camera on J2 connector. If you wanted to enable it, you need to back up and edit the following file : /system/etc/camera/camera_config.xml and add an entry that looks like below (then reboot). The slave address 0xC0 is one of the two possible slave addresses for this camera. The other one is 0xE0 and is already used by the camera plugged into J4 (downward facing).

So are you using one of the stereo cameras + stereo flex + extension cable? if you are using the stereo flex with one camera, make sure to use the left port on the flex because the right port is connected to the secondary i2c bus, accessible only in stereo configuration. Also, multiple stereo pairs are not supported on VOXL and stereo can only be used in J3.

With the below config addition, you should be able to detect all 4 ov7251 cameras and successfully use the following configurations without confilicts:

• J2 + J4 (hires slot + tracking slot)
• J3 + J4 (stereo slot + tracking slot)
• if you want to use J2 + J3 (hires + tracking slot), change the I2C id to 0xE0, but then it's the same thing as current stereo + tracking.

Now, in this configuration you should also be able to run all 4 cameras together but depending on the i2c ID of the camera in slot 0, it will conflict either with left stereo or tracking camera. I would not mess with stereo, so you could set i2c id of slot 0 to the same as in slot 2 (hires and tracking). then there is is still an issue of exposure/gain controls between the two cameras fighting with each other (both cameras will receive both exposure/gain updates).

In order to prevent two cameras from fighting exposure/gain settings the only way to do it is to prevent one of the camera drivers from sending out exposure/gain updates at all. Then both cameras will have the same exposure/gain computed based on one camera's image. This would require a little bit of work on the camera driver side but it's possible (though not supported now).

Anyway, this is just a long explanation of what you could try and what you will see.

<!-- ov7251 tracking camera in camera slot 0 -->
  <CameraModuleConfig>
    <CameraId>0</CameraId>
    <SensorName>ov7251</SensorName>
    <SensorSlaveAddress>0xC0</SensorSlaveAddress>
    <ChromatixName>ov7251_chromatix</ChromatixName>
    <ModesSupported>1</ModesSupported>
    <Position>FRONT</Position>
    <MountAngle>360</MountAngle>
    <CSIInfo>
      <CSIDCore>0</CSIDCore>
      <LaneMask>0x1F</LaneMask>
      <LaneAssign>0x4320</LaneAssign>
      <ComboMode>0</ComboMode>
    </CSIInfo>
    <LensInfo>
      <FocalLength>4.73</FocalLength>
      <FNumber>2.2</FNumber>
      <TotalFocusDistance>1.9</TotalFocusDistance>
      <HorizontalViewAngle>64.1</HorizontalViewAngle>
      <VerticalViewAngle>51.6</VerticalViewAngle>
      <MinFocusDistance>0.1</MinFocusDistance>
    </LensInfo>
  </CameraModuleConfig>

Hello @Gicu-Panaghiu,

I am going to assume you are using VOXL1, since you did not specify..

We do have RAW8 and RAW10 support for OV7251. The selection of the format has to be done in several places.

First, you have to select the correct camera driver, specifically..

ls /usr/lib/libmmcamera_ov7251*.so
/usr/lib/libmmcamera_ov7251.so
/usr/lib/libmmcamera_ov7251_8bit.so
/usr/lib/libmmcamera_ov7251_hflip_8bit.so
/usr/lib/libmmcamera_ov7251_rot180_8bit.so
/usr/lib/libmmcamera_ov7251_vflip_8bit.so

there are 5 options and one of them is _8bit.so which means it will natively ouptput 8bit data (all others output 10 bit data).

the driver name, such as ov7251_8bit has to be the sensor name <SensorName>ov7251_8bit</SensorName> in /system/etc/camera/camera_config.xml.

You can check camera_config.xml for what sensor library is used for your OV7251.

When you run voxl-configure-cameras script, it will actually copy one of the default camera_config.xml that are set up for a particular use case, and I believe it will indeed select the 8bit one - this was done to save cpu cycles needed to convert 10bit to 8bit, since majority of the time only 8bit pixels are used.

Now, you mentioned that HAL_PIXEL_FORMAT_RAW10 is passed to the stream config and unfortunately this does not have any effect on what the driver outputs. If the low level driver (e.g. libmmcamera_ov7251_8bit.so) is set up to output RAW8, it will output RAW8 if you request either HAL_PIXEL_FORMAT_RAW8 or HAL_PIXEL_FORMAT_RAW10.

So if you update the camera_config.xml to the 10bit driver and just keep the HAL_PIXEL_FORMAT_RAW10 in the stream config (then sync and reboot), you should be getting a 10 bit RAW image from the camera. But since the camera server is currently expecting 8 bit image, if you just interpret the image as 8 bit, it will appear garbled, so you will need to handle the 10 bit image (decide what you want to do with it) in the camera server.

Hello @david-moro ,

mvVISLAM library supports a mask file in the following format : 8 bits per pixel PGM with white (255) being where NO features should be found and black (0) where it is OK to have features. For a frame of aspect ratio 4:3, typically a mask of size 160x120 pixels should be used, although you can have a higher resolution mask if you want to. The mask will be resized to the actual image size or the smaller resolution mask will be properly queried (i do not know which one, but it should not matter). I would just suggest keeping the aspect ratio of the mask the same as the actual image.

You can test this easily by creating a mask that splits the image in half and seeing if features get detected in masked out region. Please note that the mask only affects detection of new features -- if the feature was detected in unmasked region and then the feature ends up in the masked region, it will still be tracked.

It does not look like a mask file is an option in voxl-qvio-server at the moment. You could try to add it yourself, here is an example how it was used in another project : https://github.com/ATLFlight/snap_vio/blob/master/src/snap_vio.cpp#L345C19-L345C19 . If mvVISLAM is not able to open the mask file you provided (wrong path, etc), it should output an error message like this : AR ERROR: arFileOpen(): Failed to open file: <path to mask file name>

I hope this helps!

Alex

The ipk is available here now : http://voxl-packages.modalai.com/stable/voxl-hal3-tof-cam-ros_0.0.5.ipk - you should be able to use the launch file to choose between two modes (5=short range and 9=long range) and fps, which are listed in the launch file.

@Samuel-Lehman , currently VOXL2 (M0154 variant, but not M0054 or VOXL2 mini M0104) has hardware support for display port output, the software is not available (no ETA at this point). VOXL2 is not intended to be used as a desktop computer, although it could be :).

However, if you just want to stream video without wifi / ethernet, you can actually use a cool trick : ADB port forwarding:

• plug in USBC cable from your VOXL2 in to linux pc
• run the following command to create a forwarding tunnel between voxl port 80 to port 8080 of your linux machine :

adb forward tcp:8080 tcp:80

If you have voxl-camera-server and voxl-portal already running, you should be able to view the camera streams in the browser at http://localhost:8080/video.html?cam=hires_color or something like that (depending on your camera name). I just tested it and it works well. You can use voxl-portal as you normally would, when using an network connection.

A few notes

• please make sure to use SDK 1.3.4 or later or at least voxl-portal because we recently added a fix for voxl-portal to work with port forwarding : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-portal/-/network/master?ref_type=heads
• voxl-portal can also display h264 video, but it needs a fix that i am working on and is almost done.

If this approach would work for you, you can test the non-encoded streams and we'll get the fix for h264 into voxl-portal soon.

Alex

@david-moro

If you look at this image, https://docs.modalai.com/M0169/#image-of-working-module it shows the TOF sensor (right) and the M0169 adapter (left).

The weight of the TOF module is 0.72g and weight of M0169 adapter is 0.93g.

This particular combo requires an external power input (4-5V as described here https://docs.modalai.com/M0169/#current-consumption). "External" in this case means additional power source besides the camera connector. The additional power input connects into M0169 via 2-pin JST connector.

The worse case power consumption is ~0.9W, which is a lot for such small module. The current draw plot shows a typical 5hz operation at maximum "exposure", which sends out 8 6-ms pulses of 1.0-1.1A (at 3.3V). so that is 1.1A * 3.3V * (5hz * 8 * 0.006s) = 0.87W. If you select 15Hz, the maximum exposure (emitter pulse time) will go down from 6ms to 2ms, so the maximum total power will remain the same.

This module would definitely require a heat spreader to operate continuously at this power. However maximum power output (even in auto exposure mode) can be limited to accommodate specific power dissipation capabilities. Additionally, the temperature of the emitter is reported from the TOF sensor.

Due to additional complexities that come with this new module (need for external power, which cannot be obtained from the camera connector) and heat management, we are working on the solution that would provide a positive user experience, so hence the delay.

I will check on the ETA

Hi

If you are looking for more low level functionality and a bit easier to experiment with, you may want to check out https://gitlab.com/voxl-public/utilities/voxl-rtsp/-/tree/dev . This tool is good for testing or running simple scenarios. This tool accepts some commands (like exposure control) via command line, so you could add a simple feature to save an image upon a command line input. Please see README to see how the exposure control input works and code for how it's implemented.

Please make sure that voxl-camera-server is disabled when you run voxl-rtsp.

If you still would like to go through the voxl-camera-server approach, we may need some more input from respective devs :).

I hope this helps..

Alex

@groupo , I know "cold" is a vague term, but i would say that if you have a rapid 20-30 degrees C change in temperature from idle (sitting on the ground) to flight, then it the is probably some risk of this issue happening. Rapid i mean within 1-2 minutes.

The number 180 degree is probably a coincidence because after the heading is wrong by more than 90 degrees, the vehicle will start an unstable swirling around (in position control mode). Unstable, meaning, that the trajectory of the vehicle will be a growing spiral (where if the heading is off by less than 90 degrees, it would be a shrinking spiral, but radius of the spiral growing as you get closer to 90 degrees).

When the heading is off between 90 and 180 degrees, the vehicle enters an unstable spiral (when you are in x-y gps position control mode), so probably by the end (crash) the yaw drifts closer to 180 degrees off. However, if you have the log, you can see at which point the behavior of the x-y position controller started to have spiral behavior (first stable spiral and then unstable spiral) and correlate that to the IMU temperature, if you have it, and see if the compass produced any inconsistent readings (like significantly changing magnitude of the magnetic field, which should stay constant regardless of the orientation of the vehicle).

I am not aware of this issue happening at ModalaAI during testing, so we don't have any logs and also default logging may not be sufficient to diagnose the issue to the fullest extent. You would need to look at the EKF status / output to see if there are any red flags.

Another way to debug this would be to look at the EKF's estimate of the IMU biases for gyro during the flight and separately perform the IMU calibration across the temperature range that was tested in flight). Then you could see if indeed EKF could not keep up with estimating the gyro bias drift in real time.

What I mean by "EKF not keeping up with estimating gyro bias drift in real time" is the following. EKF in PX4 has some sort of process model for how the gyro bias evolves over time. I am not 100% sure, but i don't think the EKF actually looks at the IMU temperature and uses it for estimating the gyro bias - it could be a random walk model or something like that (lets assume random walk). If the parameters of the random walk for the gyro bias drift model are conservative (the model assumes that the rate of the gyro bias drift "slowly"), then if the gyro bias indeed drifts faster than EKF assumes it can, the EKF will stop correctly tracking the bias drift and correctly subtracting the gyro bias from the raw gyro measurements.

Note that for x and y gyro bias, it is possible to estimate it using accelerometer readings (gravity vector), but the gravity vector does not provide any information for yaw drift of the vehicle (which is aligned with z axis gyro at hover). The compass comes into play here to help with this, since the magnetic field has horizontal and vertical component (unlike gravity only having vertical component w.r.t Earth). So ideally, the compass should aid with estimating the bias drift of the yaw gyro (z axis), however if the z axis gyro bias drift is larger than expected, the EKF can actually think that compass is malfunctioning, because typically the EKF trusts the gyro/accel more than other secondary sensors like compass and barometer, etc.

So, with this in mind, and keeping in mind that this analysis is an educated guess, the crash(es) may have resulted from bias from an uncalibrated IMU not being properly tracked during a sudden temperature drop of the IMU, resulting in compass being flagged as malfunctioning and heading drifting to the point that the vehicle starts spiraling out of control. However, looking at the actual compass data and imu bias data (if possible), local gps position, can provide more information.

If anyone has logs from their flights (if you want to strip out GPS coordinates, please do so, but having local position estimates would be good to see - look for spiraling behavior), you can share the log to help with the analysis..

Alex

@Denver-Hopkins , great to hear! I think your other board probably has even older firmware, but the fact that voxl2-io-spin utility detected the board, it should be upgradable to the latest firmware using the procedure that you already followed.

Alex

Most likely need to update voxl-mpa-tools to latest : https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl-mpa-tools/-/blob/master/lib/voxl_common_config.c?ref_type=heads#L655 or use voxl-mpa-tools and voxl-qvio-server from the same SDK release

@JP-Drone ,

Please try to upgrade to latest voxl-portal version from here : http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64//

I just tested on iphone displaying individual and multi view, it works. What camera feed are you selecting (name)?

The non "encoded" feeds are compressed to mjpeg format which should play on majority of browsers.

I will check regarding the video recording feature in voxl-portal.

Alex

@briandaniels15,

You should look carefully at the flight log (using px4 flight review) and see what was happening to actual compass data just prior to the crash. You can look at the magnitude of magnetic field that should remain constant regardless of the vehicle orientation.

Also, heading change by close to 180 degrees does not mean that the compass readings themselves changed so much.

Were you flying in very cold weather? If so, it is possible that EKF filter in PX4 was not able to correctly track the IMU gyro biases (quickly changing while IMU was cooling down in flight) and incorrectly reported compass error (because it was disagreeing with IMU).

This may be a similar issue, please check out the analysis: https://discuss.px4.io/t/stopping-compass-mid-flight/42808

Alex

@Jetson-Nano , I am sorry, I don't have this hardware set up right now, so I can't actually test what you are trying to do. I will check with the team regarding who can test it.

Alex

@david-moro ,

With a few fixes, I was able to get the ov7251 fisheye stereo to work. Mainly needed to tweak the calibration procedure a bit, and add a fix to actually run fisheye stereo calibration instead of regular stereo calibration for the extrinsics. I will commit the fixes to the voxl-camera-calibration repository soon. The DFS part ran fine without any changes, after getting a proper camera calibration, but I will double check if I needed to change anything in DFS.

sample outputs:

narrow:

wide (not wearing the same shirt, so the pattern is different )

I probably won't be able to run detailed analysis, but it looks like the DFS pipeline is working for fisheye use case. As you probably guessed, the fisheye setup will have shorter range but wider FOV, which can be estimated mathematically.

If you decide to go ahead and buy the fisheye cameras and try it, I just wanted to mention that there is still some risk in robustness of performance, since we have not really tested it until today. However, the results are promising.

Also, wanted to mention that the OV7251 are very old cameras. We have much better options available now, specifically AR0144 , which is 1280x800 resolution (monochrome), also global shutter. In order to use synchronized AR0144 cameras, you would need to use our new camera front end (M0173). Please see links below:

• https://docs.modalai.com/M0173/
• https://docs.modalai.com/voxl2-coax-camera-bundles/
• the best deal would probably be getting this C28 configuration (M0173 camera front end + dual AR0144 and dual IMX412 hires cameras + all the coax cables) : https://www.modalai.com/products/m0173?variant=48528274489648
• you can also buy these components separately, but the bundle seems like a better deal (even though you did not ask for two hires cameras )
• the M0173 front end will take up camera connectors J6 and J7 on VOXL2 (J8 will still be available)

If you want to wait for 1-2 days i can send you a sample output of the AR0144 stereo DFS.

Finally, there is another option, which would plug in directly into the stereo flex that you already have for OV7251 : https://docs.modalai.com/M0113/ . It is OV9782 global shutter RGB camera 1280x800 resolution. It is similar to AR0144, although OV9782 is older sensor and we do not recommend for new designs. We do support it though.
Here is a sample image from OV9782 stereo pair, so you can get an idea of the difference. This module does not have a fisheye lens, but it is much wider than the narrow OV7251. Also, the higher resolution makes a big difference.

Alex

@david-moro , I just confirmed that my concern regarding stereo input to vio is not actually an issue. The meta information for stereo image type actually has width and height equal to width and height of mono image, but the buffer contains two images. So, QVIO will just pick the first image if stereo is provided.

I will do a quick test to see how well the fisheye ov7251 calibration works.

Alex

@Denver-Hopkins ,

The latest voxl2 io firmware can be found in the voxl2-io package. either:

• https://gitlab.com/voxl-public/voxl-sdk/utilities/voxl2-io/-/tree/master/voxl2-io-tools/firmware?ref_type=heads
• on voxl2 : /usr/share/modalai/voxl2-io-tools/firmware/modalai_m0065_firmware_v0_2_RC1_f94baad1.bin

(even though on gitlab the commit says "adding test firmware", it is actually stable for many months now).

You should update both of your voxl2-io boards (M0065), it seems the one that was just shipped to you had older firmware, but that is ok, since it's easy to update.

Please try

Alex

@david-moro , the calibration procedure itself would be the same as the regular stereo calibration, you would just need to use -f flag with the voxl-calibrate-camera command. https://docs.modalai.com/calibrate-cameras/#stereo-calibration-process

Unfortunately we don't have any performance metrics comparing the fisheye vs non-fisheye ov7251 stereo setups.

I can try calibrating the cameras using our standard calibration procedure, just to make sure that it works, if you are interested.

Regarding using one of the stereo cameras for VIO, yes, that should technically work, but i need to check something - the stereo image is merged (stacked) image that combines two cameras into one image, but QVIO expects a single image with correct dimensions of a single image. Basically, camera server would need to publish a separate image with just left or right in addition to the image with combined stereo. Alternatively, QVIO server could be updated to accept a stereo image. It looks like right now QVIO sever can accept a stereo image (https://gitlab.com/voxl-public/voxl-sdk/services/voxl-qvio-server/-/blob/master/server/main.cpp?ref_type=heads#L609) but a few lines later it checks for image dimensions to match the dims in calibration file (there would be a mismatch because calibration is mono and image is stereo) .. I will need to double check this..

Alex