Minimal example of using camera_cb


  • Dev Team

    A typical YUV frame of dimensions w * h looks like this:

    uint8_t Y[w*h]; //intensity image plane
    uint8_t alignment_gap[w*n]; // if h is not a multiple of 16, there will be a gap of n lines to make (h+n) a multiple of 16. the gap could be zero (in case of 640x480, for example)
    uint8_t UV[w*h/2]; //chroma image plane (half the size of intensity plane)
    

    Note that voxl-camera-server removes the gap if needed, before sending data to the pipe, you can see : https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp#L1213

    Converting YUV to RGB is a pretty expensive process (and especially if you are dealing with 4K frames). There are a few references, for example https://stackoverflow.com/questions/17892346/how-to-convert-rgb-yuv-rgb-both-ways (just google YUV to RGB conversion). However anything you find online is likely not optimized and will use a lot of cpu.

    There is a hardware jpeg encoder in VOXL, i think it is available in voxl-camera-server (see snapshot): https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp . HW Jpeg encoder is very efficient and uses almost no cpu, but i don't think it can run at 4K30 full frame rate. @Alex-Gardner , can you please confirm that the jpeg snapshot is functional and how to use it?

    if you want to work with 4K30 streams, even pushing the YUV data through the pipe will be quite expensive (384021601.5*30 = 373MB/sec of data being passed through the pipe). I have personally not tried to do it (i think it can be attempted by configuring voxl-camera-server to use 3840x2160 resolution). Maybe @Alex-Gardner could comment on the hi-res YUV streams.

    Alex



  • Thanks for the quick reply, I'll have to go through all the links still, but just as a point I failed to mention... I'm not looking at sending a 4k video feed. The general use case will be something along the lines of (pseudo code)

    if object_A is detected:
       save 4k image
    wait x amount of time before taking another image
    

    I just wanted to mention that in case it changes how this would be used. I have the rest of the logic in place for the tflite detection and passing that information around, I just have to implement saving a single image when I call my capture function. My current implementation of this function creates the client and camera_cb, which is where the image saving happens. The camera_cb then stops the client after a single cycle, so I am not using any threading for this client.



  • Hey Pawel, the following article might help. Another name for NV21 is YUV420sp (it is the common format used in Android phones which is another thing to search for). This article takes a YUV420sp image and saves to jpg. I haven't tried it and can't guarantee it will work, but it looks reasonable.

    https://www.programmersought.com/article/99965201309/

    In general, search for yuv420sp to jpeg


  • Dev Team

    If you are not using hires camera for actual detection of object, then using a snapshot option (which provides a jpeg on demand) is the most efficient solution.

    However, you could still pipe 4K images at 30hz or lower FPS (to reduce wasted pipe data flow), and then use a software jpeg encoder on the CPU, like Chad mentioned..

    Also, i just realized that there is a reasonable example of converting YUV to RGB (if needed) here : https://gitlab.com/voxl-public/old-projects/voxl-cam-ros/-/blob/master/src/voxl_cam.cpp#L303



  • Thank you all for all of the information :). The snapshot feature sounds like what I'm looking for. I'm looking through the camera server git at the moment for the code. Is there any example for using this vs a regular stream? I see that in the camera server conf you can enable which type of stream you want. I've set preview and video to false for the hires camera and set snapshot to True. My tflite model is using the tracking camera for it's feed. Going off of the voxl-camera-server hal3-camera-mgr.cpp I see that the server handles the image saving. As for the expected behaviour, would an image be saved when data enters the pipe while the client is still connected?

    I currently have my client set with a simple camera_cb that closes the client when the cb is triggered to only save one image after the client is started. Is the snapshot client setup the same way as the streaming client? When I try to run this client with the pipe name pointing to /run/mpa/hires_preview I get the debug message showing that the client connected, but it appears that the capture buffer is being used instead of the snapshot.

    NOTE: I added the all caps printouts that show that pCaptureResultData is not null on line 1115 of the hal3_camera_mgr.cpp and that it is null on line 1240 when checkin the pSnapshotBufferInfo.

    ------ voxl-camera-server INFO: Client: modal-abr-capture-pipe0 connected to channel: 0
    
    
    2:CAPTURE BUFFER NOT NULL, BUT SHOULD BE FOR SNAPSHOT
    
    
    
    SNAPSHOT BUFFER NULL WHEN TRYING TO SAVE
    
    voxl-camera-server ERROR: Error sending request 0, ErrorCode: -22
    
    voxl-camera-server FATAL: Recieved Fatal error from camera: hires
                              Camera server will be stopped
    ------ voxl-camera-server WARNING: Thread: hires request thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: tracking result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: stereo request thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: stereo result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: hires result thread recieved ESTOP
    ------ voxl-camera-server WARNING: Thread: tracking request thread recieved ESTOP
    
    ------ voxl-camera-server INFO: Camera server is now stopping
    		There is a chance that it may segfault here, this is a mmqcamera bug, ignore it
    
    ------ voxl-camera-server INFO: Stopping tracking camera
    ------ voxl-camera-server INFO: tracking camera stopped successfully
    
    ------ voxl-camera-server INFO: Stopping hires camera
    ------ voxl-camera-server INFO: hires camera stopped successfully
    
    ------ voxl-camera-server INFO: Stopping stereo camera
    ------ voxl-camera-server INFO: stereo camera stopped successfully
    
    ------ voxl-camera-server INFO: Camera server exited gracefully
    
    

    I'm still sorting through where this error is being thrown.


  • Dev Team

    I double checked with other devs on the team and the jpeg snapshot feature in voxl-camera-server is still in development. I don't have a good ETA for the feature.

    However, perhaps we can go back to evaluating voxl-rtsp for your application. If you disable hires camera in voxl-camera-server, you should be able to use voxl-rtsp with hires camera after voxl-camera-server has started for other cameras.

    voxl-rtsp supports outputting jpegs. You can first confirm that you can access hires camera jpegs with voxl-rtsp (I don't think it can do 4K30 jpegs, but lower resolution/frame rate should work). That should work and then you could add an input to trigger voxl-rtsp to actually save a jpeg upon request (instead of every frame). This can be done similarly how exposure/gain commands are accepted by voxl-rtsp via a named pipe.

    Hopefully this can work for you..


  • Dev Team

    .. or if you wanted to continue using voxl-camera-server for 4K, just modify the server to save yuv frames on-demand and convert to jpeg later, as we discussed above..



  • Thank you for all of the ideas. I was able to get the image capture to work by setting my thread to watch for a capture_image flag to start the client, then stop it after the image is saved, instead of threading the callback itself. This let me save an image at a time to text file, then I just scp it off the voxl to parse in python as an RGB image later.

    Cheers


  • Dev Team

    Sounds good.. Just keep in mind that saving a 4K image to a text file will be much slower and will take more space than just saving the raw binary data to a file. Parsing a binary file should be as easy or easier than text.. It's up to you though 🙂



  • @Alex-Kushleyev thanks for the heads up. You make a good point, I'll switch it over to a bin 👌 Thanks!


Log in to reply