User avatar
phaseOne
Posts: 1
Joined: Fri Jul 13, 2012 10:49 pm

Realtime Geometric Camera/Lens Distortion Correction

Fri Oct 19, 2018 1:54 am

I'm currently using a wide angle lens with the OV5647 image sensor, and I'd like to correct for the geometric distortion it introduces. The application is based on the realtime video streaming capabilities of the Raspberry Pi platform, and as such, any image processing needs to be highly performant. This means that I have at least three options that I am aware of:
  1. Distortion Correction on the ISP
  2. Image Effect on the GPU
  3. GL Shader on the GPU
Based on my research, the BCM2835 GPU's ISP contains a Lens Distortion Correction block (noted in the PiCamera docs and @6by9 in this post). However, I am unable to find any references to it in the userland source or elsewhere. Could someone help point me in the right direction please?

In pursuit of this approach, I saw some memory reserved for the distortion tables in this GitHub issue. Perhaps there's a way to manipulate the lookup tables in memory without modifying the GPU source?

The image effect shader seems to be a non starter due to the locked down GPU firmware.

I noticed that raspistill can apply an OpenGL shader to its output, however this feature does not seem present in raspivid. It seems feasible from a bandwidth and memory perspective as the shader would be relatively simple, and I'm only streaming at 720p with limited H.264 encoding demands (baseline, no prediction, etc.). I've gathered a few references to individuals who have attempted this as well, but I'm unclear on the best path forward here.

It seems my best option at this point is to get a pointer to an OpenGL OES texture from the camera MMAL, run it through a shader, but then I'm unsure about how to feed that texture to the video encoder in a performant way. Any advice here that doesn't involve a copy through the CPU?

Thank you to 6by9, jamesh, and other community members that have contributed to the camera development effort!

References: Implementation Details:
  • I'm using a Pi Zero in a headless configuration.
  • H.264 video is streamed over the network, not to disk.
  • My current software implementation is based in Python, and utilizes the PiCamera package. However, given the complexity and performance demands of this project, I will likely port the code over to C++.
  • I am aware that I will likely need to adjust the lens shading tables as well, though this is a lesser concern.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Realtime Geometric Camera/Lens Distortion Correction

Fri Oct 19, 2018 8:32 am

Yes there is geometric distortion correction in the isp. IIRC it's not enabled by default but the tuner (control sw) for it will be loaded hence the memory allocation.
It's not a high priority to enable setting custom distortion grids, but I can look into it.

There is a thread on feeding gl textures into video encode using vcsm. It actually all fell out fairly easily with mmal whilst IL was slightly more restricted.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Realtime Geometric Camera/Lens Distortion Correction

Fri Oct 19, 2018 8:42 am

viewtopic.php?f=68&t=222930 is using IL but has recommendations for MMAL. I thought it had been implemented but can't see it.
viewtopic.php?f=70&t=223050 was adding an overlay although references using GL.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20767
Joined: Sat Jul 30, 2011 7:41 pm

Re: Realtime Geometric Camera/Lens Distortion Correction

Fri Oct 19, 2018 9:51 am

Worth noting that large distortions corrections in the ISP do slow down the frame rate. ISTR a colleague at BRCM using the distortion block to correct for odd sensor shapes, and we were unable to maintain required framerates. I believe it did depend on the extent of the correction required. That may have been on a VC3 though, perhaps the current block is faster.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Realtime Geometric Camera/Lens Distortion Correction

Fri Oct 19, 2018 4:13 pm

It depends on the level of correction. The pipeline splits the image into tiles and has to have sufficient input per tile to produce the complete output tile. A large correction requires a bigger input area to produce the same output, and the hardware is limited on whichever is greater out of input or output pixel rate.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”