longo92 wrote: ↑
Fri Oct 26, 2018 6:19 am
Another question: so if have to perform resize (not cropping) is always better to use the vc.ril.isp than the vc.ril.resize? Because isp uses hardware while resize is implemented in software (then it pass through the CPU), right?
Everything has limitations.
The ISP can process one pixel per clock cycle, and runs at 250MHz, so theoretically can hit 250MPix/s.
A few constraints:
- It is limited at both ends of the pipe, so either one pixel in or one pixel out. So if downscaling then the input resolution will be the limit, whilst the output is the limit if upscaling. Other pipe stages stall for a cycle if necessary.
- The ISP processes in tiles, and each tile has a small amount of context required. That context has to be accounted for.
- It is almost impossible to keep the hardware fully occupied with things to process, so scheduling is an issue.
- Everything has to go to and from memory, and memory bandwidth is not infinite.
- The block is also used by the camera component, and now video_encode.
In contrast, vc.ril.resize uses the VPU (Vector Processing Unit), which is also used for almost all other processing within the VideoCore GPU.
It is a 16 way SIMD unit, so can process 16 bits of data per clock cycle. However resize is going to be about 100 instructions per pixel element (eg Y, U, V, or red, green, or blue), which would translate to a maximum of 40MPix/s for a single channel image if you had sole use of the VPU. Reality is that you don't have sole use of the VPU, and you don't deal in mono images, therefore the realistic maximum would be lower. Memory bandwidth is a restriction too.
longo92 wrote:When yuoi said that now the encoder uses isp for conversion, it means that can it can perform resizing inside (also format conversion)? So there is no need to tunnel isp->encoder?
I haven't added any option for resizing at the front end of video_encode, although in theory it would be possible.
I added it as the codec block requires two versions of the image in particular formats, and the ISP was designed to be able to write out both images at the same time. Doing it on the VPU resulted in too much contention when trying to exceed 1080p30, hence pulling in the ISP hardware to help out. There are a couple of conditions where it can't be used and it has to drop back to using the VPU, but generally it's a win, particularly as it extends the range of pixel formats that can be handled by the encoder.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.