Minimal example of using camera_cb

PawelJ

Hi, I've been trying to get a simple camera callback for saving a frame from the hires camera on call, instead of a continuous stream listening to the pipe. Do you have any simple examples of something along these lines? I've been working off of the tflite-server and voxl-streamer, but they are a little more involved and are taking some time to parse through for the relevant sections. I'm setting it up without threading so the pipeline initialization can be called, and the callback will save the pixel data to a jpeg or png, then stop the client so the main script can continue. Any direction is greatly appreciated.

Cheers

Alex Kushleyev

Hi

If you are looking for more low level functionality and a bit easier to experiment with, you may want to check out https://gitlab.com/voxl-public/utilities/voxl-rtsp/-/tree/dev . This tool is good for testing or running simple scenarios. This tool accepts some commands (like exposure control) via command line, so you could add a simple feature to save an image upon a command line input. Please see README to see how the exposure control input works and code for how it's implemented.

Please make sure that voxl-camera-server is disabled when you run voxl-rtsp.

If you still would like to go through the voxl-camera-server approach, we may need some more input from respective devs :).

I hope this helps..

Alex

PawelJ

Thanks for the quick reply! Unforunately I need the camera server running because I will be using vvpx4 and vtflite which I believe require it. However, I was able to get a simple camera_cb client working and subscribed to the /run/mpa/hires_preview. Is there a header you can point me to that defines the frame struct? I can't seem to find what format it should be in. I recall past conversations with some devs mentioning that the 4k camera uses NV21 format, but I can't find any references on how to convert / save it as either a png, or netpbm format. Also, are there any streams for the full 4k feed, or is that something that will need to be added on my end?

Alex Kushleyev

A typical YUV frame of dimensions w * h looks like this:

uint8_t Y[w*h]; //intensity image plane
uint8_t alignment_gap[w*n]; // if h is not a multiple of 16, there will be a gap of n lines to make (h+n) a multiple of 16. the gap could be zero (in case of 640x480, for example)
uint8_t UV[w*h/2]; //chroma image plane (half the size of intensity plane)

Note that voxl-camera-server removes the gap if needed, before sending data to the pipe, you can see : https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp#L1213

Converting YUV to RGB is a pretty expensive process (and especially if you are dealing with 4K frames). There are a few references, for example https://stackoverflow.com/questions/17892346/how-to-convert-rgb-yuv-rgb-both-ways (just google YUV to RGB conversion). However anything you find online is likely not optimized and will use a lot of cpu.

There is a hardware jpeg encoder in VOXL, i think it is available in voxl-camera-server (see snapshot): https://gitlab.com/voxl-public/modal-pipe-architecture/voxl-camera-server/-/blob/master/src/api_interface/hal3/hal3_camera_mgr.cpp . HW Jpeg encoder is very efficient and uses almost no cpu, but i don't think it can run at 4K30 full frame rate. @Alex-Gardner , can you please confirm that the jpeg snapshot is functional and how to use it?

if you want to work with 4K30 streams, even pushing the YUV data through the pipe will be quite expensive (384021601.5*30 = 373MB/sec of data being passed through the pipe). I have personally not tried to do it (i think it can be attempted by configuring voxl-camera-server to use 3840x2160 resolution). Maybe @Alex-Gardner could comment on the hi-res YUV streams.

Alex

PawelJ

Thanks for the quick reply, I'll have to go through all the links still, but just as a point I failed to mention... I'm not looking at sending a 4k video feed. The general use case will be something along the lines of (pseudo code)

if object_A is detected:
   save 4k image
wait x amount of time before taking another image

I just wanted to mention that in case it changes how this would be used. I have the rest of the logic in place for the tflite detection and passing that information around, I just have to implement saving a single image when I call my capture function. My current implementation of this function creates the client and camera_cb, which is where the image saving happens. The camera_cb then stops the client after a single cycle, so I am not using any threading for this client.

Chad Sweet

Hey Pawel, the following article might help. Another name for NV21 is YUV420sp (it is the common format used in Android phones which is another thing to search for). This article takes a YUV420sp image and saves to jpg. I haven't tried it and can't guarantee it will work, but it looks reasonable.

https://www.programmersought.com/article/99965201309/

In general, search for yuv420sp to jpeg

Alex Kushleyev

If you are not using hires camera for actual detection of object, then using a snapshot option (which provides a jpeg on demand) is the most efficient solution.

However, you could still pipe 4K images at 30hz or lower FPS (to reduce wasted pipe data flow), and then use a software jpeg encoder on the CPU, like Chad mentioned..

Also, i just realized that there is a reasonable example of converting YUV to RGB (if needed) here : https://gitlab.com/voxl-public/old-projects/voxl-cam-ros/-/blob/master/src/voxl_cam.cpp#L303

PawelJ

Thank you all for all of the information :). The snapshot feature sounds like what I'm looking for. I'm looking through the camera server git at the moment for the code. Is there any example for using this vs a regular stream? I see that in the camera server conf you can enable which type of stream you want. I've set preview and video to false for the hires camera and set snapshot to True. My tflite model is using the tracking camera for it's feed. Going off of the voxl-camera-server hal3-camera-mgr.cpp I see that the server handles the image saving. As for the expected behaviour, would an image be saved when data enters the pipe while the client is still connected?

I currently have my client set with a simple camera_cb that closes the client when the cb is triggered to only save one image after the client is started. Is the snapshot client setup the same way as the streaming client? When I try to run this client with the pipe name pointing to /run/mpa/hires_preview I get the debug message showing that the client connected, but it appears that the capture buffer is being used instead of the snapshot.

NOTE: I added the all caps printouts that show that pCaptureResultData is not null on line 1115 of the hal3_camera_mgr.cpp and that it is null on line 1240 when checkin the pSnapshotBufferInfo.

------ voxl-camera-server INFO: Client: modal-abr-capture-pipe0 connected to channel: 0


2:CAPTURE BUFFER NOT NULL, BUT SHOULD BE FOR SNAPSHOT



SNAPSHOT BUFFER NULL WHEN TRYING TO SAVE

voxl-camera-server ERROR: Error sending request 0, ErrorCode: -22

voxl-camera-server FATAL: Recieved Fatal error from camera: hires
                          Camera server will be stopped
------ voxl-camera-server WARNING: Thread: hires request thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: tracking result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: stereo request thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: stereo result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: hires result thread recieved ESTOP
------ voxl-camera-server WARNING: Thread: tracking request thread recieved ESTOP

------ voxl-camera-server INFO: Camera server is now stopping
		There is a chance that it may segfault here, this is a mmqcamera bug, ignore it

------ voxl-camera-server INFO: Stopping tracking camera
------ voxl-camera-server INFO: tracking camera stopped successfully

------ voxl-camera-server INFO: Stopping hires camera
------ voxl-camera-server INFO: hires camera stopped successfully

------ voxl-camera-server INFO: Stopping stereo camera
------ voxl-camera-server INFO: stereo camera stopped successfully

------ voxl-camera-server INFO: Camera server exited gracefully

I'm still sorting through where this error is being thrown.

Alex Kushleyev

I double checked with other devs on the team and the jpeg snapshot feature in voxl-camera-server is still in development. I don't have a good ETA for the feature.

However, perhaps we can go back to evaluating voxl-rtsp for your application. If you disable hires camera in voxl-camera-server, you should be able to use voxl-rtsp with hires camera after voxl-camera-server has started for other cameras.

voxl-rtsp supports outputting jpegs. You can first confirm that you can access hires camera jpegs with voxl-rtsp (I don't think it can do 4K30 jpegs, but lower resolution/frame rate should work). That should work and then you could add an input to trigger voxl-rtsp to actually save a jpeg upon request (instead of every frame). This can be done similarly how exposure/gain commands are accepted by voxl-rtsp via a named pipe.

Hopefully this can work for you..

Alex Kushleyev

.. or if you wanted to continue using voxl-camera-server for 4K, just modify the server to save yuv frames on-demand and convert to jpeg later, as we discussed above..

PawelJ

Thank you for all of the ideas. I was able to get the image capture to work by setting my thread to watch for a capture_image flag to start the client, then stop it after the image is saved, instead of threading the callback itself. This let me save an image at a time to text file, then I just scp it off the voxl to parse in python as an RGB image later.

Cheers

Alex Kushleyev

Sounds good.. Just keep in mind that saving a 4K image to a text file will be much slower and will take more space than just saving the raw binary data to a file. Parsing a binary file should be as easy or easier than text.. It's up to you though

PawelJ

@Alex-Kushleyev thanks for the heads up. You make a good point, I'll switch it over to a bin Thanks!

m1baldwin

Just coming back to this; any update on the snapshot feature?

Hi,
voxl-camera-server >= 1.0.2 (live on voxl2/rb5, beta on voxl1) supports 4k snapshots from the hires camera, you can echo "snapshot" into the hires control pipe and it will write the snapshot out to /data/snapshots/, or echo "snapshot [path]" into the pipe and it will write the snapshot out to the given path.

mkwan

@Alex-Gardner I get an error when trying snapshot on the Sentinel, voxl-suite 0.8.1.

Input:

voxl2:/run/mpa/hires$ echo "snapshot" > control

Output:

Nov 04 16:20:41 m0054 bash[24392]: Camera: hires taking snapshot (destination: /data/snapshots/hires-0.jpg)
Nov 04 16:20:41 m0054 bash[24392]: free(): invalid pointer
Nov 04 16:20:41 m0054 systemd[1]: voxl-camera-server.service: Main process exited, code=killed, status=6/ABRT
Nov 04 16:20:41 m0054 systemd[1]: voxl-camera-server.service: Failed with result 'signal'.

Any advice?

@mkwan Can you make a new thread with this issue and include the output of voxl-inspect-services -v