muth
Posts: 27
Joined: Sun Aug 05, 2012 3:32 pm

Advice for a large buffer of Y images from the camera

Thu Jan 10, 2019 9:51 am

Hi,
I'll try to summarize what I'd like to achieve:
A continuous live effect where each displayed line have a delay equal to its position. That means the first line of the image has no delay, the second line has one frame delay, the 500th line has 500 frames delay. I tried to illustrate that with the following: (the rocket is rotating)
Image
I imagine the left part as a large image/frame 'fifo' or rotating buffer, and the process is executed for each new frame from the camera, in order to have a continuous animation.

I manage to create this behavior with Processing as I'm more comfortable with Java. The following code work at ~20 fps with an image size of 300^3 pixels:

Code: Select all

img = new PImage[video_size];
imgDest = createImage(video_size, video_size, RGB);
for (int j = 0; j < img.length; j++) {
	img[j] = createImage(video_size, video_size, RGB);
}
// [...]
void draw() {
  // If the camera is sending new data, capture that data
  if (video.available()) {
    video.read();
    img[imgIndex%video_size].copy(video, 0,0,video_size,video_size,0,0,video_size,video_size);
    for(int i=0; i<video_size; i++){
      imgDest.copy(img[(imgIndex + i)%video_size], 0, i, video_size, 1, 0, i, video_size, 1);
    }
    imgIndex++;
  }
  image(imgDest, 0, 0, video_size, video_size);
}
Now my ultimate wishes are >=60fps, image size of 576^3, but in grayscale. The buffer size is theoretically around 200MB with 8bit per pixel.

I dug a bit in the raspistill code, and the following is where I would like to have advice:

I saw in RaspiTexUtil.c in

Code: Select all

int raspitexutil_do_update_texture(EGLDisplay display, EGLenum target,
                                   EGLClientBuffer mm_buf, GLuint *texture, EGLImageKHR *egl_image)
That a EGLImageKHR is created with for example EGL_IMAGE_BRCM_MULTIMEDIA_Y as target. Is it the right direction to create a large stack of image/texture ? Is an array of 576 EGLImageKHR conceivable ?

To compute the displayed image, I imagine I could draw the complete stack of image/texture and select only the right pixel line using a shader. This part as well is completely new for me. I'm starting learning the basics of GL drawing, but in the meantime if someone has advice to compute the image (by extracting every lines from an images buffer), in an efficient way, I would appreciate a lot.

It was a lengthy post, thanks for reading!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7555
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Advice for a large buffer of Y images from the camera

Thu Jan 10, 2019 10:41 am

GL seems overkill for such a trivial operation, unless you want to do something else with the data other than only displaying it.

Use raspividyuv instead of raspistill and you'll get YUV frames back at full camera rate.
Allocate a buffer for output.
In camera_buffer_callback you get given the YUV frame (buffer->data). Memcpy the one desirwed line from that YUV frame into your output buffer. Display that buffer (video_render or dispmanx).

There's a minor fiddle needed if using video_render as you can't submit the same buffer twice, so you have to double buffer.
The nasty hack is to use zero copy, as then your mapped buffer is the live thing being displayed, so updating it will update the screen (ignoring cache flush operations).
A better solution is to have two buffers with MMAL. If you hold on to both data pointers, then memcpy into both buffers from the callback and alternately submit them (hopefully the buffer will have been returned by the time you submit it again, otherwise bad things will happen).
As a better option still, you should be able to allocate a buffer pool with 2 buffer headers but no payload. Allocate a single image buffer, and assign buffer->data to that image buffer for both MMAL buffer headers. Now when either buffer is submitted it will update a buffer on the GPU, but retain the correct operation with no tearing.

Others have used a very similar trick for stacking images to make star trails. I thought viewtopic.php?f=43&t=82306&start=63#p594649 was done that way, but actually it has saved all the frames and post-processed. I'm sure someone wrote the equivalent code to do it as the frames came back, but I'm not going searching now.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

muth
Posts: 27
Joined: Sun Aug 05, 2012 3:32 pm

Re: Advice for a large buffer of Y images from the camera

Thu Jan 10, 2019 11:27 am

Thanks for your answer 6by9!
I made a fist quick attempt some time ago by putting 576 in the existing buffer pool

Code: Select all

/* Create pool of buffer headers for the output port to consume */
   pool = mmal_port_pool_create(video_port, video_port->buffer_num, video_port->buffer_size);
I will search why I got the following error (seems a 512 maximum here)

Code: Select all

./raspividyuv -w 256 -h 256 -t 0 -fps 60
mmal: mmal_vc_shm_alloc: could not get a free slot in the payload list
mmal: mmal_vc_port_payload_alloc: vc.ril.camera:out:1(I420): failed to allocate 98304 bytes of shared memory
mmal: mmal_pool_initialise_buffer_headers: failed to allocate payload 512/576
mmal: Failed to create buffer header pool for camera still port vc.ril.camera:out:2(OPQV)
^Cmmal: Aborting program
I will get back on this side, you're right GL is overkill, I though about it first for performance reasons.
I should keep all the previous 576 frames. As when new frame arrives, the selected lines to be extracted of a particular frame will be different, until this frame is too old (576 frame-old).
Thanks a lot for your time and the link to the stacked image is very interesting !

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7555
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Advice for a large buffer of Y images from the camera

Thu Jan 10, 2019 12:18 pm

muth wrote:
Thu Jan 10, 2019 11:27 am
Thanks for your answer 6by9!
I made a fist quick attempt some time ago by putting 576 in the existing buffer pool

Code: Select all

/* Create pool of buffer headers for the output port to consume */
   pool = mmal_port_pool_create(video_port, video_port->buffer_num, video_port->buffer_size);
I will search why I got the following error (seems a 512 maximum here)

Code: Select all

./raspividyuv -w 256 -h 256 -t 0 -fps 60
mmal: mmal_vc_shm_alloc: could not get a free slot in the payload list
mmal: mmal_vc_port_payload_alloc: vc.ril.camera:out:1(I420): failed to allocate 98304 bytes of shared memory
mmal: mmal_pool_initialise_buffer_headers: failed to allocate payload 512/576
mmal: Failed to create buffer header pool for camera still port vc.ril.camera:out:2(OPQV)
^Cmmal: Aborting program
576 buffers is going to be a problem - even at VGA that's 265MB, so you'd have to be setting gpu_mem to >300MB to be able to run, and you get mirrored buffers in GPU and ARM memory.
There is also a 512 buffer limit. https://github.com/raspberrypi/userland ... _shm.c#L37

Code: Select all

#define MMAL_VC_PAYLOAD_ELEM_MAX 512
(It is a fairly arbitrary limit, and I believe can be increased at will with only a rebuild of userland required).

mmal_vc_shm.c only comes into play with zero copy though, so no mirror buffers, but it does have to fit into gpu_mem and be less than MMAL_VC_PAYLOAD_ELEM_MAX buffers.
muth wrote:I will get back on this side, you're right GL is overkill, I though about it first for performance reasons.
I should keep all the previous 576 frames. As when new frame arrives, the selected lines to be extracted of a particular frame will be different, until this frame is too old (576 frame-old).
Perhaps I misunderstood you.
So you want frame 576 to be line 1 from frame 1, line 2 from frame 2, etc to line 576 from frame 576.
Frame 577 to be line 1 from frame 2, line 1 from frame 3, etc to line 576 from frame 577.
Frame 578 to be line 1 from frame 3, line 1 from frame 4, etc to line 576 from frame 578.
etc.
So all lines of all frames will get used eventually. In which case that will require you to retain all the buffers.
I'd initially read it that you were only taking one line from each frame, in which case the result is at a framerate of camera fps / frame height.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

muth
Posts: 27
Joined: Sun Aug 05, 2012 3:32 pm

Re: Advice for a large buffer of Y images from the camera

Thu Jan 10, 2019 1:19 pm

6by9 wrote:
Thu Jan 10, 2019 12:18 pm

576 buffers is going to be a problem - even at VGA that's 265MB, so you'd have to be setting gpu_mem to >300MB to be able to run, and you get mirrored buffers in GPU and ARM memory.
There is also a 512 buffer limit. https://github.com/raspberrypi/userland ... _shm.c#L37

Code: Select all

#define MMAL_VC_PAYLOAD_ELEM_MAX 512
(It is a fairly arbitrary limit, and I believe can be increased at will with only a rebuild of userland required).

mmal_vc_shm.c only comes into play with zero copy though, so no mirror buffers, but it does have to fit into gpu_mem and be less than MMAL_VC_PAYLOAD_ELEM_MAX buffers.
muth wrote:I will get back on this side, you're right GL is overkill, I though about it first for performance reasons.
I should keep all the previous 576 frames. As when new frame arrives, the selected lines to be extracted of a particular frame will be different, until this frame is too old (576 frame-old).
Perhaps I misunderstood you.
So you want frame 576 to be line 1 from frame 1, line 2 from frame 2, etc to line 576 from frame 576.
Frame 577 to be line 1 from frame 2, line 1 from frame 3, etc to line 576 from frame 577.
Frame 578 to be line 1 from frame 3, line 1 from frame 4, etc to line 576 from frame 578.
etc.
So all lines of all frames will get used eventually. In which case that will require you to retain all the buffers.
I'd initially read it that you were only taking one line from each frame, in which case the result is at a framerate of camera fps / frame height.
Yes exactly, all lines of all frames will get used. It would be more clear if I could make a video. The output frame-rate is preserved, let say 60fps. It's a mix between a slit-scan effect but for all the lines of the image, and a time delay on each line proportional to the position of the line in the image. It's hard to explain, but it enables rather intuitively funny creativity once people face it.
I would like then to put this effect on a photo-booth, my 576 pixel contraint comes from the printer. Of course, I've yet increased the gpu_mem.

Thanks a lot for pointing the constant MMAL_VC_PAYLOAD_ELEM_MAX , I'll give a try.

muth
Posts: 27
Joined: Sun Aug 05, 2012 3:32 pm

Re: Advice for a large buffer of Y images from the camera

Sun Jan 13, 2019 11:00 am

Hi,
Thanks again 6by9 for your precious advice.
With raspiviyuv.c, render.c and your help I manage to have a first working code. It is far from being publishable, there still a lot of cleanup, addition to do, and some process are maybe the right way to do it.
Parameters so far: image 400x400, RGB, buffer 400 images, camera @120fps. CPU usage is around 20% according top.
There a video : https://youtu.be/jGHT7iofSUk
and a snapshot:
Image

I used the existing MMAL_POOL_T of the video, set with the 400 buffers. On the camera_buffer_callback I get a buffet from the render pool queue. I fill each "i" lines by memcpy a line from camera_pool->header[ i ]->priv->payload. I didn't understood well enough how the queue is working, camera_pool->header[ i ]->data is valid only for the newest buffer.
I have to find a way to only do the memcpy process at 60fps. Must keep the rotating buffer array of the queue filled at the camera frame rate, but we only need to display at max 60fps.
I probably have to fix some white balance parameters as the automatic process runs really fast at 120fps.

I'll post progress here once I have a nicer code.

muth
Posts: 27
Joined: Sun Aug 05, 2012 3:32 pm

Re: Advice for a large buffer of Y images from the camera

Tue Jan 29, 2019 11:58 am

Hi,
I more or less managed to have the behavior I'd like. The following code is really messy, I'm now on cleaning that a bit.
https://github.com/pierre-muth/selfpi/b ... piVidYUV.c
For the delay, I use a png image which each pixel intensity represents the delay in frames (0-255) for each pixel of the output image.
It runs at 90fps, and the rendering loop at about 30fps.

Here is the output file with a simple vertical gradient for time delays.
Image

And the real time effect:
https://youtu.be/KcQxYdWLLnk

Return to “Camera board”