Memotech Bill
Posts: 135
Joined: Sun Nov 18, 2018 9:23 am

Understanding MMAL camera control interactions

Mon Jun 21, 2021 3:10 pm

I am returning to an old project using a Raspberry Pi (v1) camera to digitise some 8mm movie film.

To try and improve the quality and consistency of the digitised frames I am trying to understand how the various MMAL camera controls work. Perhaps some of these are obvious to those familiar with digital cameras, but I have not seen much of this documented.

It is well documented that you can only control the red and blue white balance gains if AWB mode is set to off.

However, there seems to be other interactions, which are not documented:
  • The analogue and digital gains can only be controlled if the ISO value is set to zero. With any setting in the documented range (100-800) the gains seem to be automatically adjusted.
  • Exposure mode only has any effect if the shutter speed is set to zero.
  • Exposure compensation only has any effect if ISO is set to a non-zero value.
Also, the brightness, contrast and saturation controls must be affecting the gain in the complete processing pipeline, but they do not appear to affect the gains reported in the MMAL status callback. It is also noticeable that it is possible to adjust the analogue and digital gains over quite wide ranges while having negligible effect on the final image. Presumably these later controls act to counter the gain changes.

Is it possible to have at least some qualitative documentation of the image pipeline describing where the different controls are applied, and how they interact?

Or do I need to move to using libcamera?

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 198
Joined: Tue Jan 07, 2020 9:15 am

Re: Understanding MMAL camera control interactions

Mon Jun 21, 2021 4:39 pm

Hi, let me try to answer as much as I can:
Memotech Bill wrote:
Mon Jun 21, 2021 3:10 pm
It is well documented that you can only control the red and blue white balance gains if AWB mode is set to off.
Correct.
  • The analogue and digital gains can only be controlled if the ISO value is set to zero. With any setting in the documented range (100-800) the gains seem to be automatically adjusted.
  • Exposure mode only has any effect if the shutter speed is set to zero.
  • Exposure compensation only has any effect if ISO is set to a non-zero value.
Yes on all counts here I think. If you specify an ISO value that is effectively the same as setting fixed gain values, so you no longer have any other control over the analogue and digital gains. If you further fix the shutter (i.e. set a non-zero value) then the AE/AGC algorithm has no levers left and so other settings (such as exposure mode or compensation) will have no effect.
Also, the brightness, contrast and saturation controls must be affecting the gain in the complete processing pipeline, but they do not appear to affect the gains reported in the MMAL status callback.
Brightness and contrast are handled by the tone (gamma) curve, so aren't reported anywhere as a "gain". Saturation is dealt with by a 3x3 matrix so that's not a global gain either (though cranking it up would lead to more colour noise).
It is also noticeable that it is possible to adjust the analogue and digital gains over quite wide ranges while having negligible effect on the final image. Presumably these later controls act to counter the gain changes.
Ultimately these parameters all get clamped somewhere, so once you hit those limits nothing really changes any more.
Is it possible to have at least some qualitative documentation of the image pipeline describing where the different controls are applied, and how they interact?
The principal steps in the handling of brightness and colours are:
1. Exposure and analogue gain (this happens in the sensor)
2. Simultaneous application of global digital gain and colour gains (this is now in the ISP).
3. 3x3 colour matrix.
4. Gamma transform.
I'd also point you at this document, section 5. It's about our libcamera implementation, but it's doing the same kind of things.
Or do I need to move to using libcamera?
No, but it gives you more control and we can explain exactly what is going on. I won't say it doesn't still have some deficiencies, but it might be worth a try (our libcamera-apps would be a place to start).

Memotech Bill
Posts: 135
Joined: Sun Nov 18, 2018 9:23 am

Re: Understanding MMAL camera control interactions

Mon Jun 21, 2021 8:37 pm

therealdavidp wrote:
Mon Jun 21, 2021 4:39 pm
It is also noticeable that it is possible to adjust the analogue and digital gains over quite wide ranges while having negligible effect on the final image. Presumably these later controls act to counter the gain changes.
Ultimately these parameters all get clamped somewhere, so once you hit those limits nothing really changes any more.
It is not really clamping that I am seeing. If you turn the gain right down you will get a black image, or right up you get a white image. However for a surprisingly large range of gains between these values the image is the expected view and largely unaffected by the exact gain value, so over this range the brightness & contrast adjustment must be largely counteracting the effect of the gain change.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11464
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Understanding MMAL camera control interactions

Mon Jun 21, 2021 9:10 pm

Memotech Bill wrote:
Mon Jun 21, 2021 8:37 pm
It is not really clamping that I am seeing. If you turn the gain right down you will get a black image, or right up you get a white image. However for a surprisingly large range of gains between these values the image is the expected view and largely unaffected by the exact gain value, so over this range the brightness & contrast adjustment must be largely counteracting the effect of the gain change.
Have you fixed the exposure time when trying this? If not then AE will be compensating for your specified gains by shifting the exposure time in the opposite direct. It can only adjust within a defined range, so push it too far and it can't and you will get a solid black or white image.

If using raspistill/vid, then add the -set option to print out the gains and exposure time as they change.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Memotech Bill
Posts: 135
Joined: Sun Nov 18, 2018 9:23 am

Re: Understanding MMAL camera control interactions

Tue Jun 22, 2021 7:26 am

6by9 wrote:
Mon Jun 21, 2021 9:10 pm
Have you fixed the exposure time when trying this? If not then AE will be compensating for your specified gains by shifting the exposure time in the opposite direct. It can only adjust within a defined range, so push it too far and it can't and you will get a solid black or white image.
Yes exposure time is fixed, I see the same effect if I fix the gains and vary the exposure time.
6by9 wrote:
Mon Jun 21, 2021 9:10 pm
If using raspistill/vid, then add the -set option to print out the gains and exposure time as they change.
I am using my own code, but catching the values returned in the status callback. That is how I knew that if ISO was non-zero, setting the analogue and digital gains did not work.

cpixip
Posts: 52
Joined: Mon May 01, 2017 7:56 am

Re: Understanding MMAL camera control interactions

Tue Jun 22, 2021 1:49 pm

- you might want to have a look at the picamera-lib which was used by some people successfully to access v1/v2 and HQ cameras for digitizing Super-8 material. At least with this fork of the picamera lib (https://github.com/rwb27/picamera/relea ... OH_article) you can set and fix analog and digital gain.

Might be an easier approach than using libcamera....

For digitizing film material, you want to switch off all automatic functions. This is possible with the picamera lib only to a certain extend, but to an extent which is usable for film scanning.

There is some source code for film scanning out there on the internet - you might want to read through some pages of the Kinograph forum (https://forums.kinograph.cc/). Also interesting for you might be the github-pages of Dominique Galland (https://github.com/dgalland).
Last edited by cpixip on Wed Jun 23, 2021 6:06 am, edited 1 time in total.

naushir
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 106
Joined: Mon Apr 25, 2016 10:21 am

Re: Understanding MMAL camera control interactions

Tue Jun 22, 2021 2:39 pm

Might be an easier approach than using libcamera....
Ignoring the C++ vs python difference (when comparing with picamera), libcamera should provide complete control over all 3A and imaging parameters, and would likely be easier to use than MMAL to control said parameters. If folks think there are controls and knobs that are missing from our libcamera implementation, let us know!

Return to “Camera board”