Huge number of camera frames skipped using voxl-logger

boga

We are gathering data, from m500 drone, to apply our own visual odometry algorithms. We discovered yesterday that a huge number of frames are suddenly skipped in stereo and tracking camera data. Also the hires camera frames are skipped, but in smaller amounts (usually 2 - 4 frames). I have added a link to a zip folder where there is stereo camera images, data.csv and calculations file (camera_calc_data.xls). You can see from the calculations file that there are 965 frames lost before sample/picture number 290 and that the image collection has been "frozen" for 33 seconds. Another place where we can see the same is at sample 567.
https://drive.google.com/file/d/1bh0zA9yBO6Si-yWjcdOsxNplCR-9AVLk/view?usp=sharing

Is this caused by the voxl-logger?
Or is this a camera stream issue?
System overload? Probably not, because the hires camera is still streaming at the same time.

We also did a test where we logged only the stereo camera, but we are still seeing similar behaviour. Although, in that case, there was a 10 second gap instead of 30 seconds.

Moderator

@boga What else are you running? Can you use voxl-inspect-cpu to determine your CPU load? we don't drop frames under normal operation, or VIO wouldn't work. It's most likely too much is being run.

Also, different parts of the system can overload at different times. There is only so much bandwidth to write to flash. You could just be logging too much

boga

@Moderator we have default setup, so it shouldn't be the case. Haven't modified the drone in any way and we are not running anything special. We tried to log only stereo camera data using voxl-logger, still had 10 second and about 300 frame "blackouts". I recommend you to look at the data that I uploaded, you can see the behavior over there.

I also logged the voxl-inspect-cam tool at the same time to see if all frames are there and according to the voxl-inspect-cam that is true. Meaning that in the published camera feed all of the frames are there, which would lead me to think that something happens with the voxl-logger (side of things). As hi-resolution camera logs frames do not drop as suddenly and at the same time, I would think that something is wrong with logging the tracking and stereo cameras?

I haven't had the chance to run the voxl-inspect-cpu while flying and it's a shame that there is no reasonable option to log the voxl-inspect-cpu data. I guess the voxl-portal is not needed, but we were not connected to it anyway. QGroundControl is connected and receiving all of the default stuff + hires camera feed (also default stuff).
voxl-logger is at version 0.3.5

Here is the output from voxl-instepct-services. Which services could I turn off?

Service Name | Version | Enabled | Running | CPU Usage

I am thinking of verifying if I am able to capture the video feed with another tool, to see if there are any blackouts at the same time that we have missing frames in the voxl-logger data. I am thinking of using voxl-ffmeg would you recommend something else? The setup should be as simple as possible, just to verify that we are able to capture all of the data from a camera or cameras.
I will also do the voxl-inspect-cpu test during flying and also I am planning to monitor the process load using top.

Moderator

@boga top is not helpful as it does not account for CPU scaling. it just seems you are running out of bandwidth to write to the flash. Can you just try one camera at a time?

Or, can you try using a high-end SD Card?

boga

Short Answer
We are not logging to the SD card. We are logging to the default location: /data/voxl-logger/ (which is in soldered flash I believe).
Test results from logging one camera:

Tracking - no droputs
Stereo - shorter dropouts than logging all cameras

We can try logging to high-end SD Card.

@Moderator said in Huge number of camera frames skipped using voxl-logger:

top is not helpful as it does not account for CPU scaling

Ok, clear. Then we have to use voxl-inspect-cpu (or a modified version of it which will write the data into columns, so we can log it and correlate with the droputs).

Longer Answer
@Moderator, like I already said, we already tried logging just one camera at a time. Although, I just went through the gathered data again and I can see no dropouts when we are logging only the tracking cam. This was during 1 minute 36 second flight. When we are logging only the stereo camera, we see dropouts, although they have shorter period of 10 seconds, compared to the ones that happens when we are logging all the cameras + visual odometry stuff, in that case the dropouts are 30 seconds long.

We are logging to the default location which is /data/voxl-logger/ . I believe that this is located on a soldered flash? Do you want to tell me that SD card communication could be faster?

We need to log all of the cameras + visual odometry data.

So I have the logger with the following command: voxl-logger --preset_odometry --cam stereo --cam hires_large_color

When I was logging only the stereo or tracking then I executed the command like this (of course):
voxl-logger --cam stereo
voxl-logger --cam tracking

boga

@Moderator, repeating some of my questions, because I did not get the answers.

We are logging the data on the 32GB (UFS 2.0) /data/voxl-logger/ location. That, probably means, that the speed and bandwidth are better than most SD cards, right? Which would also mean we are not running out of bandwidth, when recording from a camera or cameras. Is that so?
I posted the list of services that are running. Do you see any that are unnecessary and could be interfering with the logging process? I mean, which ones we could turn off?
Have you recorded the data using voxl-logger and gone through the logs to see if you have any data missing? Did you look at the files that I attached?
Could this be a voxl-logger version issue? We have an older version of the code (0.3.5), could this be an issue? Maybe there were some fixes introduced
Could you propose other methods for recording and checking the camera feed?
Maybe we should update the whole SDK?

Alex Kushleyev

@boga ,

Can you please double check the following:

amount of free space in the /data partition: df | grep data
check the write speed to the data partition : dd if=/dev/urandom of=/data/test.bin bs=1M count=1024 (this will write a test file of size 1GB). For example, using voxl2 i result i got was (on VOXL1 it will be a bit slower):

#set cpu mode to perf
voxl-set-cpu-mode perf

#perform the write test using randomly generated data (may be limited by speed of /dev/urandom)
voxl2:/data$ dd if=/dev/urandom of=/data/test.bin bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.85171 s, 377 MB/s

(you can / should remove /data/test.bin after the test to free up the space)

Alex