Refactor voxl-camera-server into multiple processes (per cam)

[Rowan Dempster]

Hi Modal,

Rowan from Cleo here, in order to figure out which camera is crashing and try to mitigate that Simon and I are investigating how to split the voxl-camera-server into multiple processes, where each process runs a single camera (so 4 processes, two for two tracking cameras, one for the ToF and one for the Hi-Res).

Do the Modal camera developers have any input on the best path to follow for this development? Any technical issues we might run into? And also, will this split have the desired affect of only having the CamEx abort affect one camera, or will all the separate processes still abort?

Thanks!
Rowan

[Rowan Dempster]

As a first shot at this I just removed the kill_existing_process and make_pid_file code to allow for multiple camera server processes to be running at once and launched two instances with different tracking camera config files. The first one launched fine but the second one immediately aborts and doesn't really print any debug messages even with -d 0:

DEBUG:   SUCCESS: Camera module opened on attempt 0
Aborted

[Rowan Dempster]

Tracked it down to when get_number_of_cameras is called after the cameraModule is init().

M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = cameraModule->get_number_of_cameras();
M_DEBUG("DONE Getting num cams\n");

Everything work until the last line which aborts.

[Rowan Dempster]

To be clear, it printed getting num cams, did not print DONE getting num cams

[Rowan Dempster]

I tried just hard coding the num cams and moving on to the setting of callbacks:
M_DEBUG("SUCCESS: Camera module opened on attempt %d\n", i);

//This check should never fail but we should still make it
if (cameraModule->init != NULL)
{
    M_DEBUG("Calling init\n");
    cameraModule->init();
    M_DEBUG("Calling init done\n");
    if (cameraModule->init == NULL)
    {
        M_ERROR("Camera module failed to init\n");
        return NULL;
    }
}

M_DEBUG("Getting num cams\n");
int numCameras = 1;
M_DEBUG("DONE Getting num cams\n");
M_DEBUG("----------- Number of cameras: %d\n\n", numCameras);

M_DEBUG("Setting callbacks\n");
cameraModule->set_callbacks(&moduleCallbacks);
M_DEBUG("DONE Setting callbacks\n");

So now it gets to "setting callbacks" but ABORTS before it gets to "DONE setting callbacks". Here are the logs: dmesg.txt logcat.txt

[Alex Kushleyev]

@Rowan-Dempster ,

I myself tried to do this some time ago and ran into issues, which i could not resolve. I am sure also someone else at Modal also tried it before me.

If I remember correctly, my issue was that it could not get camera module in the second instance of the camera server, on this line, and i did not pursue this further (did not spend too much time on it though).

https://gitlab.com/voxl-public/voxl-sdk/services/voxl-camera-server/-/blob/master/src/hal3_helpers.cpp?ref_type=heads#L79

So in your case you are able to get the camera module?

To help debug, you should run logcat while you are trying this and see what messages might show up there.

And, finally, just wanted to mention that we don't know if what you are doing is possible, but if you want to keep pursuing this direction, I can help if i know the answer to your questions, but may not be able to spend time working on it right now.

EDIT: in order to see if separate instances of camera server are possible, you could try to see if you can run multiple instances of gstreamer plugin that uses qmmfsrc. I found a doc that should be helpful in testing this, take a look : https://thundercomm.s3-ap-northeast-1.amazonaws.com/shop/doc/1596593567074634/Thundercomm EB5_Multimedia SDK User Guide_V1.1_ie89e.pdf

Specifically, the following command should work, although i did not test it:

gst-launch-1.0 -e qtiqmmfsrc name=camsrc ! video/x-raw\(memory:GBM\),format=NV12,width=1920,height=1080,framerate=30/1 ! queue ! omxh264enc control-rate=max-bitrate target-bitrate=6000000 interval-intraframes=29 periodicity-idr=1 ! queue ! filesink location="/data/vid.h264"

Also, i dont see how you sent the camera id in this example, but it should definitely be possible

See if you can run two independent instances of gstreamer using different cameras. I think it should work, but not sure.

Alex