Posts made by Rowan Dempster

(top is Royale 4, bottom is 5)

Here's a comparison at 5FPS: https://drive.google.com/file/d/1NLrdz0SMWsduWnmzLONM64mnIyciXHVC/view?usp=sharing

When upgrading the voxl-camera-server from Royale 4 vs. 5 I noticed a big change in the output of the A65 ToF. Cleo had tuned our mapping software to the Royale 4 output and performs worse on the Royale 5 output. @James-Strawson looks like you were doing some tuning after changing to the Royale 5 processing pipeline, do you any tips for how to recover the Royale 4 performance on the new voxl-camera-server?

I tried just hard coding the num cams and moving on to the setting of callbacks:
M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = 1;
M_DEBUG("DONE Getting num cams\n");
M_DEBUG("----------- Number of cameras: %d\n\n", numCameras);

M_DEBUG("Setting callbacks\n");
cameraModule->set_callbacks(&moduleCallbacks);
M_DEBUG("DONE Setting callbacks\n");

So now it gets to "setting callbacks" but ABORTS before it gets to "DONE setting callbacks". Here are the logs: dmesg.txt logcat.txt

To be clear, it printed getting num cams, did not print DONE getting num cams

Tracked it down to when get_number_of_cameras is called after the cameraModule is init().

M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = cameraModule->get_number_of_cameras();
M_DEBUG("DONE Getting num cams\n");

Everything work until the last line which aborts.

As a first shot at this I just removed the kill_existing_process and make_pid_file code to allow for multiple camera server processes to be running at once and launched two instances with different tracking camera config files. The first one launched fine but the second one immediately aborts and doesn't really print any debug messages even with -d 0:

DEBUG:   SUCCESS: Camera module opened on attempt 0
Aborted

Hi Modal,

Rowan from Cleo here, in order to figure out which camera is crashing and try to mitigate that Simon and I are investigating how to split the voxl-camera-server into multiple processes, where each process runs a single camera (so 4 processes, two for two tracking cameras, one for the ToF and one for the Hi-Res).

Do the Modal camera developers have any input on the best path to follow for this development? Any technical issues we might run into? And also, will this split have the desired affect of only having the CamEx abort affect one camera, or will all the separate processes still abort?

Thanks!
Rowan

When the ToF is looking at empty space, it seems to return that "no measurement" information via a (0,0,0) point value. But when it is looking at occupied space some of the points returned are still "no measurement" points (0,0,0). Is there any way of separating between returns that just never hit something, or points that hit something but the sensor is not confident enough to report a location for?

For the purposes of occupancy mapping it's very useful to know that subtlety of "unknown yes or no the ray hit something" vs. "known empty space along this ray".

@tom @Vinny No problem Tom and Vinny I can do some research on my own and post back when I find a good method for future users. It's just Ubuntu right so I'm sure there are lots of open source tools for monitoring CPU core KPIs. Was just wondering if Modal had already written any "wrapper" software that packages up and exposes the CPU Core KPIs that Modal knows are important. Perhaps in the cpu-monitor code? I'll make an MR if I find anything useful to add for cpu monitoring :). Specifically I'm looking at exposing KPIs that indicate when a core is not performing at a level that it could be because of the thermal environment it is running in.

Oh and to answer your other question we don't have a clue if throttling will affect our overall system performance, but we definitely want to monitor CPU core KPIs just as a good engineering practice.

@Vinny Yeah being able to check if a core is being throttled is actually exactly what I'm looking for! I was going to use temperature as a proxy for that, but if the core can just tell me explicitly how it is operating that would be ideal. Can you provide me with technical documentation on how to query that information from the cores?

Also regarding https://docs.modalai.com/voxl2-thermal-performance/#thermal-performance that document is a boon of information and very helpful to our company, thank you to whoever wrote it and to you for linking it!

@Vinny Thanks for the response, I realized that I did not specify that this is on a VOXL2 with default CPU/thermal daemon settings. So to confirm, we are good to push the CPU cores over 80C without seeing any drop in CPU performance?

What temperature does a core have to hit before that core starts having its clock throttled?

At Cleo we've been using the https://gitlab.com/voxl-public/voxl-sdk/third-party/voxl-microdds-agent to send odometry data to PX4 UORB land. We've been seeing random cutouts of the writer where the odometry data stops being sent through (the /fmu/in/odom topic is active, but the UORB topic is not receiving), and restarting the voxl-microdds-agent fixes the issue. Do you have any ideas on what might be causing this? We haven't seen any failure logs and we have no reproduction steps.

Hi @Cliff-Wong thank you for the response!

In the version of PX4 that Modal is using (commit: https://github.com/modalai/px4-firmware/blob/b5c45bb232afc6a936b8349f4fa0be1efeb17707/src/modules/ekf2/ekf2_params.c) I see MAG_CHECK as:

/**
 * Magnetic field strength test selection
 *
 * When set, the EKF checks the strength of the magnetic field
 * to decide whether the magnetometer data is valid.
 * If GPS data is received, the magnetic field is compared to a World
 * Magnetic Model (WMM), otherwise an average value is used.
 * This check is useful to reject occasional hard iron disturbance.
 *
 * @group EKF2
 * @boolean
 */
PARAM_DEFINE_INT32(EKF2_MAG_CHECK, 1);

Do you suggest updating the EKF2 code to the more recent version that supports more advanced mag health checks:

EKF2_MAG_CHECK:
      description:
        short: Magnetic field strength test selection
        long: 'Bitmask to set which check is used to decide whether the magnetometer
          data is valid. If GNSS data is received, the magnetic field is compared
          to a World Magnetic Model (WMM), otherwise an average value is used. This
          check is useful to reject occasional hard iron disturbance. Set bits to
          1 to enable checks. Checks enabled by the following bit positions 0 : Magnetic
          field strength. Set tolerance using EKF2_MAG_CHK_STR 1 : Magnetic field
          inclination. Set tolerance using EKF2_MAG_CHK_INC 2 : Wait for GNSS to find
          the theoretical strength and inclination using the WMM'
      type: bitmask
      bit:
        0: Strength (EKF2_MAG_CHK_STR)
        1: Inclination (EKF2_MAG_CHK_INC)
        2: Wait for WMM
      default: 1
      min: 0
      max: 7
    EKF2_MAG_CHK_STR:
      description:
        short: Magnetic field strength check tolerance
        long: Maximum allowed deviation from the expected magnetic field strength
          to pass the check.
      type: float
      default: 0.2
      min: 0.0
      max: 1.0
      unit: gauss
      decimal: 2
    EKF2_MAG_CHK_INC:
      description:
        short: Magnetic field inclination check tolerance
        long: Maximum allowed deviation from the expected magnetic field inclination
          to pass the check.
      type: float
      default: 20.0
      min: 0.0
      max: 90.0
      unit: deg
      decimal: 1

What we are worried about is the mag being used when the measurements are unhealthy.

Hi Modal,

Cleo is looking into using Magnetometer+GPS combo for outdoor heading and positioning. We're wondering how Modal handles the switch between indoor and outdoor flight, specifically how does the EKF2 know when to stop fusing the magnetometer (definitely don't want to try to keep heading via mag inside)?

Thanks!
Rowan

@Eric-Katzfey What about running PX4 in SITL so Ulogs can be replayed offline on one's own laptop for things like EKF2 tuning? Is that part of your workflow?

Has Modal tested PX4 replay functionality (https://docs.px4.io/main/en/debug/system_wide_replay.html) on the VOXL2?

I am running into issues loading the orb_publisher.rules file. When I specify more that ~25 characters of dynamic memory (for the restrict_topics and/or module fields) something else in PX4 segfaults. Seems like I'm hitting the upper limit of memory allocation for PX4 modules, very strange.

Hi I'm seeing that voxl-microdds-agent (and foxy, etc...) are blacklisted on SDK1.2? I thought these packages were released

Blacklist: https://gitlab.com/voxl-public/voxl-sdk/voxl-suite/-/blob/sdk-1.2.0/publishing-scripts/generate_depends.py?ref_type=tags#L8