(top is Royale 4, bottom is 5)

Here's a comparison at 5FPS: https://drive.google.com/file/d/1NLrdz0SMWsduWnmzLONM64mnIyciXHVC/view?usp=sharing

When upgrading the voxl-camera-server from Royale 4 vs. 5 I noticed a big change in the output of the A65 ToF. Cleo had tuned our mapping software to the Royale 4 output and performs worse on the Royale 5 output. @James-Strawson looks like you were doing some tuning after changing to the Royale 5 processing pipeline, do you any tips for how to recover the Royale 4 performance on the new voxl-camera-server?

I tried just hard coding the num cams and moving on to the setting of callbacks:
M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = 1;
M_DEBUG("DONE Getting num cams\n");
M_DEBUG("----------- Number of cameras: %d\n\n", numCameras);

M_DEBUG("Setting callbacks\n");
cameraModule->set_callbacks(&moduleCallbacks);
M_DEBUG("DONE Setting callbacks\n");

So now it gets to "setting callbacks" but ABORTS before it gets to "DONE setting callbacks". Here are the logs: dmesg.txt logcat.txt

To be clear, it printed getting num cams, did not print DONE getting num cams

Tracked it down to when get_number_of_cameras is called after the cameraModule is init().

M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = cameraModule->get_number_of_cameras();
M_DEBUG("DONE Getting num cams\n");

Everything work until the last line which aborts.

As a first shot at this I just removed the kill_existing_process and make_pid_file code to allow for multiple camera server processes to be running at once and launched two instances with different tracking camera config files. The first one launched fine but the second one immediately aborts and doesn't really print any debug messages even with -d 0:

DEBUG:   SUCCESS: Camera module opened on attempt 0
Aborted

Hi Modal,

Rowan from Cleo here, in order to figure out which camera is crashing and try to mitigate that Simon and I are investigating how to split the voxl-camera-server into multiple processes, where each process runs a single camera (so 4 processes, two for two tracking cameras, one for the ToF and one for the Hi-Res).

Do the Modal camera developers have any input on the best path to follow for this development? Any technical issues we might run into? And also, will this split have the desired affect of only having the CamEx abort affect one camera, or will all the separate processes still abort?

Thanks!
Rowan

When the ToF is looking at empty space, it seems to return that "no measurement" information via a (0,0,0) point value. But when it is looking at occupied space some of the points returned are still "no measurement" points (0,0,0). Is there any way of separating between returns that just never hit something, or points that hit something but the sensor is not confident enough to report a location for?

For the purposes of occupancy mapping it's very useful to know that subtlety of "unknown yes or no the ray hit something" vs. "known empty space along this ray".

@tom @Vinny No problem Tom and Vinny I can do some research on my own and post back when I find a good method for future users. It's just Ubuntu right so I'm sure there are lots of open source tools for monitoring CPU core KPIs. Was just wondering if Modal had already written any "wrapper" software that packages up and exposes the CPU Core KPIs that Modal knows are important. Perhaps in the cpu-monitor code? I'll make an MR if I find anything useful to add for cpu monitoring :). Specifically I'm looking at exposing KPIs that indicate when a core is not performing at a level that it could be because of the thermal environment it is running in.