Akronoz wrote:First I observed that latency varies around 5ms, is that caused by ISP or by Linux scheduling?.
Probably a combination of the two. There is contention on the VPU within the GPU for some image processing and the format conversion, and there is also some jitter from the Linux side. The sensor itself is standalone and will just keep on clocking out the frames at the rate determined by the configuration - that's about the only given.
Akronoz wrote:From my first question about CPU/GPU side, when camera_buffer_callback its called and I have acces to *buffer, the data was copied to CPU memory? Its posible to acces it via GPU/OpenGL before that copy to get better latency?
You can improve timings slightly by adding "mmal_port_parameter_set(camera_video_port, MMAL_PARAMETER_ZERO_COPY, MMAL_TRUE)" BEFORE the port is enabled and the buffer pool is allocated. That maps the GPUs buffer into ARM memory and so saves a memcpy of the frame.
PLEASE do a "sudo rpi-update" before trying that one, or sync and build the latest userland tree. There was a deadlock situation discovered in the kernel when using zero copy and that was worked around with https://github.com/raspberrypi/userland ... 2838fc250c
OpenGL isn't going to help - all the framework stuff for that is frame based.
Akronoz wrote:And there is a way to acces the pipeline earlier, for example, getting directly the bayer data before it is converted to pixels by ISP, or when each line was send to the ISP to apply my algorithm line by line on the fly before the frame its completed? or its not possible because all its done by hardware automatically.
Whilst the firmware could, and I think there was even a software stage developed to remote algorithms to the ARM, that isn't an option available on Pi.
All the work being done to expose the raw Bayer data via V4L2 won't help much either as it will all be frame based, so you won't save much time. You also then lose the ISP and all the AE/AWB control loops.
If you felt like a world of pain, then if you switched to using OpenMaxIL instead of MMAL you can specify a value of nSliceHeight on each port. This means you can deliver the output buffer as stripes instead of whole frames. If you specify that as eg nFrameHeight/8 (although it has to be a multiple of 16) and nBufferCountActual = 8, then you'd get 8 buffers delivered per frame with each coming out as soon as they had been produced.
IL is supported on Pi and a few people have used it quite happily, however it is not a nice API and has lots of nasty gotchas, so we'd recommend you stick with MMAL unless it really gives you something extra. Please do be aware though that there is no zero copy option with IL - the copy from GPU to ARM memory is mandatory.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.