camera_image_metadata_t framerate is zero from lepton0_color

[paul.foley]

Hello,
I have an image processing application that is getting camera frames from the lepton0_color pipe. However, in the camera_image_metadata_t the framerate is zero. Is this expected or do you have any suggestions on how to fix it? For reference, when my application receives frames from the hires_down_small_color pipe the metadata has the framerate populated correctly.

Below is a snippet of the logs from my application showing the framerate of zero.

_camera_helper_cbWARNING: Camera frame rate of 0 Hz is < required frame rate of 1 Hz. 
Camera Image Metadata:
  Magic Number:     0x564f584c
  Timestamp (ns):  72864071544650
  Frame ID:         632968
  Dimensions:      160x120 pixels
  Size (bytes):    57600
  Stride:           480 bytes/row
  Exposure (ns):   0
  Gain (ISO):      0
  Format:           10
  Framerate (Hz):  0
  Reserved:        0

I am on a Starling 2 Max. Below is my current configuration.

voxl2:~$ voxl-version 
--------------------------------------------------------------------------------
system-image: 1.8.02-M0054-14.1a-perf
kernel:       #1 SMP PREEMPT Mon Nov 11 22:59:41 UTC 2024 4.19.125
--------------------------------------------------------------------------------
hw platform:  M0054
mach.var:     1.2.1
--------------------------------------------------------------------------------
voxl-suite:   1.4.1
--------------------------------------------------------------------------------
Packages:
Repo:  http://voxl-packages.modalai.com/ ./dists/qrb5165/sdk-1.4/binary-arm64/
Last Updated: 2023-03-02 12:58:40
WARNING: repo file has changed since last update,
        packages may have originated from a different repo
List:
kernel-module-voxl-fsync-mod-4.19.125     1.0-r0
kernel-module-voxl-gpio-mod-4.19.125      1.0-r0
kernel-module-voxl-platform-mod-4.19.125  1.0-r0
libfc-sensor                              1.0.7
libmodal-cv                               0.5.16
libmodal-exposure                         0.1.3
libmodal-journal                          0.2.2
libmodal-json                             0.4.3
libmodal-pipe                             2.10.6
libqrb5165-io                             0.4.9
libvoxl-cci-direct                        0.2.5
libvoxl-cutils                            0.1.1
modalai-slpi                              1.1.19
mv-voxl                                   0.1-r0
qrb5165-bind                              0.1-r0
qrb5165-dfs-server                        0.2.0
qrb5165-imu-server                        1.1.2
qrb5165-rangefinder-server                0.1.4
qrb5165-slpi-test-sig                     01-r0
qrb5165-system-tweaks                     0.3.4
qrb5165-tflite                            2.8.0-2
voxl-bind-spektrum                        0.1.1
voxl-camera-calibration                   0.5.9
voxl-camera-server                        2.1.1
voxl-ceres-solver                         2:1.14.0-10
voxl-configurator                         0.9.7
voxl-cpu-monitor                          0.5.3
voxl-docker-support                       1.3.1
voxl-elrs                                 0.4.1
voxl-esc                                  1.5.1
voxl-feature-tracker                      0.5.2
voxl-flow-server                          0.3.6
voxl-fsync-mod                            1.0-r0
voxl-gphoto2-server                       0.0.10
voxl-gpio-mod                             1.0-r0
voxl-io-server                            0.0.4
voxl-jpeg-turbo                           2.1.3-5
voxl-lepton-server                        1.3.3
voxl-lepton-tracker                       0.0.4
voxl-libgphoto2                           0.0.4
voxl-libuvc                               1.0.7
voxl-logger                               0.4.9
voxl-mavcam-manager                       0.5.7
voxl-mavlink                              0.1.1
voxl-mavlink-server                       1.4.4
voxl-modem                                1.1.5
voxl-mongoose                             7.7.0-1
voxl-mpa-to-ros                           0.3.9
voxl-mpa-tools                            1.3.7
voxl-open-vins                            0.4.16
voxl-open-vins-server                     0.3.0
voxl-opencv                               4.5.5-2
voxl-osd                                  0.1.1
voxl-platform-mod                         1.0-r0
voxl-portal                               0.7.5
voxl-px4                                  1.14.0-2.0.94
voxl-px4-imu-server                       0.1.2
voxl-px4-params                           0.6.3
voxl-qvio-server                          1.1.1
voxl-remote-id                            0.0.9
voxl-reset-slpi                           0.0.1
voxl-state-estimator                      0.0.4
voxl-streamer                             0.7.5
voxl-suite                                1.4.1
voxl-tag-detector                         0.0.4
voxl-tflite-server                        0.3.9
voxl-utils                                1.4.4
voxl-uvc-server                           0.1.7
voxl-vision-hub                           1.8.17
voxl-vtx                                  1.1.8
voxl2-io                                  0.0.3
voxl2-system-image                        1.8.02-r0
voxl2-wlan                                1.0-r0
--------------------------------------------------------------------------------

[Alex Kushleyev]

@paul-foley , it looks like our voxl-lepton-server is not setting the FPS field in the metadata : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-lepton-server/-/blob/master/src/publisher.c?ref_type=heads#L338 . I do see above that we do set the framerate in json info about the output pipe (cJSON_AddNumberToObject(json, "framerate", 9);) and the framerate is hardcoded to 9, since i guess all lepton sensors are running at 9fps.

I just made the change to publish the fps : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-lepton-server/-/commit/110b9cce4c021322566e46f36d6878fca55f7d26

You are using an older voxl SDK, so installing latest nightly package (from http://voxl-packages.modalai.com/dists/qrb5165/dev/binary-arm64/ will probably not work). So you probably need to take the source of the version you are running now (1.3.3 : https://gitlab.com/voxl-public/voxl-sdk/services/voxl-lepton-server/-/tree/v1.3.3?ref_type=tags) and apply the same change in the source code and build it.

Since i just pushed to dev branch, a new version of leptop server should appear in our package repo tomorrow, and you can download and try it. it might work, but your SDK is 2 major versions down from where we are now (1.4.x vs 1.6.x), so you may want to just build the patched version 1.3.3 :

git clone https://gitlab.com/voxl-public/voxl-sdk/services/voxl-lepton-server.git
cd voxl-lepton-server
git checkout v1.3.3

#make the change in publisher.c to include fps ...
...

#start voxl-cross docker (you can download it in our downloads section: https://developer.modalai.com/asset) 
docker run -it --rm -v `pwd`:/opt/code -w /opt/code voxl-cross:V4.6 bash

./install_build_deps.sh qrb5165 sdk-1.4
./build.sh qrb5165
./make_package.sh

#then, outside docker, deploy the new package to voxl2 via adb
./deploy_to_voxl.sh

Please let me know if this works for you!

Alex