voxl-camera-server bugs with snapshot capture for mapping

AndriiHlyvko

Hi. I was evaluating the voxl-camera-server snapshot feature for the purpose of using the images for othromaps and pointcoud generation. I found multiple things which are not working properly. I am using an external px4 (mro-ctrl-zero-h7) which sends an onboard mavlink stream to a serial port on voxl-mini. Timebased snapshot capture was working fine so I moved to an acutal survey mission outdoors, which is when I discovered none of the images were saved on the voxl. I set up a testbench where I am flying SITL missions on my laptop and use Mavproxy to output the SITL mavlink through USB-TTL adapter to VOXL. To check the snapshot timing information I point the IMX214 camera at my screen which shows the SITL QGC simulation. Here are my findings:

If I use the default mavlink datarate and baudrate the snapshot capturing is not reliable. Sometimes no snapshots get captured at all. When I moved to using a custom mavlink stream on px4 which only outputs command messages and GPS info the snapshot feature started working reliably.
I can only reliably capture camera images at most at 1HZ. If I increase the camera capture command frequency beyond 1Hz snapshots get lost.
GPS Exif image data contains wrong GPS seconds data. Degrees and seconds are correct but seconds is wrong. (SITL GPS is in Zurich and VOXL Exif data claims 37 deg 24' 3.85" N, 121 deg 59' 3.11" W which is Santa Clara )

Here is a link to an archive with the snapshots captured by voxl with the SITL mission file I was running:
https://drive.google.com/file/d/1vA5-F1MUICMC2DUvb5XdrhR2N_YSinao/view?usp=sharing

Eric Katzfey

@AndriiHlyvko Snapshots commands are handled by the voxl-mavcam-manager application. It has it's own mavlink connection to QGC. You can enable debug messages in voxl-mavcam-manager by adding the -d option to see what commands it is getting. The images will have timestamps based on the system time at the voxl, not of the external flight controller.

AndriiHlyvko

@Eric-Katzfey do you know if there a way to run a remote gdb server through adb and then just insert breakpoints in the mavcam manager?

AndriiHlyvko

@Eric-Katzfey I added my own print statements to voxl-camera manager that prints GPS data it grams from mavlink before it inserts the information as exif. It is correct GPS data:

ANDRII GPS mavlink: lat:37.413543, lon:-121.996550
Camera: hires taking snapshot (destination: /data/snapshots/hires-2024-05-16_21:18:18.jpg)
ANDRII GPS mavlink: lat:37.413543, lon:-121.996550
ANDRII grabbed gps: lat:37.413543, lon:-121.996550
Camera: hires writing snapshot to :"/data/snapshots/hires-2024-05-16_21:18:18.jpg"
ANDRII GPS mavlink: lat:37.413543, lon:-121.996550

But then when I inspect the snapshot image with exiftool I see the information is not correct:

---- ExifTool ----
ExifTool Version Number         : 12.40
---- File ----
File Name                       : hires-2024-05-16_21:18:18.jpg
Directory                       : ./snapshots
File Size                       : 641 KiB
File Modification Date/Time     : 2024:05:16 14:18:41-07:00
File Access Date/Time           : 2024:05:16 14:18:41-07:00
File Inode Change Date/Time     : 2024:05:16 14:18:41-07:00
File Permissions                : -rw-r--r--
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
Exif Byte Order                 : Big-endian (Motorola, MM)
Exif Byte Order                 : Big-endian (Motorola, MM)
Image Width                     : 4208
Image Height                    : 3120
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
---- EXIF ----
Image Width                     : 4208
Image Height                    : 3120
Make                            : QTI-AA
Camera Model Name               : QCAM-AA
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Modify Date                     : 2024:05:16 21:18:18
Y Cb Cr Positioning             : Centered
Exposure Time                   : 1/120
F Number                        : 2.4
Exposure Program                : Program AE
ISO                             : 194
Exif Version                    : 0220
Date/Time Original              : 2024:05:16 21:18:18
Create Date                     : 2024:05:16 21:18:18
Components Configuration        : Y, Cb, Cr, -
Shutter Speed Value             : 1/120
Aperture Value                  : 2.4
Brightness Value                : 3.52
Exposure Compensation           : 0
Max Aperture Value              : 2.4
Metering Mode                   : Average
Light Source                    : D65
Flash                           : Off, Did not fire
Focal Length                    : 2.2 mm
Sub Sec Time                    : 340360
Sub Sec Time Original           : 340360
Sub Sec Time Digitized          : 340360
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 4208
Exif Image Height               : 3120
Interoperability Index          : R98 - DCF basic file (sRGB)
Interoperability Version        : 0100
Sensing Method                  : Not defined
Scene Type                      : Directly photographed
Exposure Mode                   : Auto
White Balance                   : Auto
Focal Length In 35mm Format     : 2 mm
Scene Capture Type              : Standard
GPS Latitude Ref                : North
GPS Latitude                    : 37 deg 24' 1.00"
GPS Longitude Ref               : West
GPS Longitude                   : 121 deg 59' 1.48"
GPS Altitude                    : 1 m
Compression                     : JPEG (old-style)
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Thumbnail Offset                : 986
Thumbnail Length                : 15135
Image Width                     : 4208
Image Height                    : 3120
Make                            : QTI-AA
Camera Model Name               : QCAM-AA
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Modify Date                     : 2024:05:16 21:18:18
Y Cb Cr Positioning             : Centered
Exposure Time                   : 1/120
F Number                        : 2.4
Exposure Program                : Program AE
ISO                             : 194
Exif Version                    : 0220
Date/Time Original              : 2024:05:16 21:18:18
Create Date                     : 2024:05:16 21:18:18
Components Configuration        : Y, Cb, Cr, -
Shutter Speed Value             : 1/120
Aperture Value                  : 2.4
Brightness Value                : 3.52
Exposure Compensation           : 0
Max Aperture Value              : 2.4
Metering Mode                   : Average
Light Source                    : D65
Flash                           : Off, Did not fire
Focal Length                    : 2.2 mm
Sub Sec Time                    : 340360
Sub Sec Time Original           : 340360
Sub Sec Time Digitized          : 340360
Flashpix Version                : 0100
Color Space                     : sRGB
Exif Image Width                : 4208
Exif Image Height               : 3120
Interoperability Index          : R98 - DCF basic file (sRGB)
Interoperability Version        : 0100
Sensing Method                  : Not defined
Scene Type                      : Directly photographed
Exposure Mode                   : Auto
White Balance                   : Auto
Focal Length In 35mm Format     : 2 mm
Scene Capture Type              : Standard
Exif Image Width                : 320
Exif Image Height               : 240
Compression                     : JPEG (old-style)
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Thumbnail Offset                : 17013
Thumbnail Length                : 15135
Thumbnail Image                 : (Binary data 15135 bytes, use -b option to extract)
---- Composite ----
Aperture                        : 2.4
Image Size                      : 4208x3120
Megapixels                      : 13.1
Scale Factor To 35 mm Equivalent: 0.9
Shutter Speed                   : 1/120
Create Date                     : 2024:05:16 21:18:18.340360
Date/Time Original              : 2024:05:16 21:18:18.340360
Modify Date                     : 2024:05:16 21:18:18.340360
GPS Latitude                    : 37 deg 24' 1.00" N
GPS Longitude                   : 121 deg 59' 1.48" W
Circle Of Confusion             : 0.033 mm
Field Of View                   : 167.3 deg
Focal Length                    : 2.2 mm (35 mm equivalent: 2.0 mm)
GPS Position                    : 37 deg 24' 1.00" N, 121 deg 59' 1.48" W
Hyperfocal Distance             : 0.06 m
Light Value                     : 8.5

AndriiHlyvko

@AndriiHlyvko lat:37.413543, lon:-121.996550 is 37°24'48.8"N 121°59'47.6"W and not 37 deg 24' 1.00" N, 121 deg 59' 1.48" W

AndriiHlyvko

@Eric-Katzfey Idk if you are still interested but I fixed the bug in the voxl-camera-server. It turnes out whoever wrote it didn't do research on how exif format works. The exif data that was being appended to the image was written in intel byteorder while the camera uses motorolla byteorder. I think your HAL driver needs to check the byteordering when the camera is being set up. But here is the fix that works for the IMX214 camera:

    // degrees, min, seconds is each a Radional
    // a rational is two uint32 numbers
    entry = create_tag(exif, EXIF_IFD_GPS, (ExifTag)EXIF_TAG_GPS_LATITUDE, 3*sizeof(ExifRational));
    entry->format = EXIF_FORMAT_RATIONAL;
    entry->components = 3;

    ExifLong degrees_lat = static_cast<ExifLong>(gps_grabbed_info.latitude);
    
    double fractional = (gps_grabbed_info.latitude - degrees_lat);
    ExifLong minutes_lat = static_cast<ExifLong>(fractional*60);
    
    fractional = (fractional*60) - minutes_lat;
    double seconds = (fractional * 60);
    
    
    
    ExifRational degrees_r = { degrees_lat*10, 10 };
    ExifRational minutes_r = { minutes_lat, 1 };
    ExifRational seconds_r = { static_cast<ExifLong>(seconds * 1000000), 1000000 }; // Increased precision for seconds
    //ExifRational seconds_r = { static_cast<ExifLong>(seconds), 1 };
    //ExifRational seconds_r = { 3375UL, 100UL };
    
    printf("Seconds int:%u\n", seconds_r.numerator);
    printf("Seconds double:%lf\n", seconds);
    
    exif_set_rational(entry->data, EXIF_BYTE_ORDER_MOTOROLA, degrees_r);
    exif_set_rational(entry->data + sizeof(ExifRational), EXIF_BYTE_ORDER_MOTOROLA, minutes_r);
    exif_set_rational(entry->data + 2 * sizeof(ExifRational), EXIF_BYTE_ORDER_MOTOROLA, seconds_r);

Moderator

@AndriiHlyvko thanks! Feel free to submit a merge request. Otherwise, we'll look to pull in manually

AndriiHlyvko

@Moderator sounds good I will try to create a merge request for this sometime early next week

AndriiHlyvko

@Moderator Whats the procedure for creating merge requests that are from someone outside of ModalAI? The repo doesn't allow pushing a new branch. I also tried to send an email a link generated by gitlab and that didn't work.

tom

@AndriiHlyvko Can you try forking the repo, pushing your changes to your fork, then opening a merge request from your fork into the dev branch?

AndriiHlyvko

@tom cool. I created an MR with wrong source branch by mistake so just delete that one. But this one is the right one: https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/merge_requests/42