Disabling ISP processing blocks

[6by9]

THIS IS TO BE CONSIDERED A VERY ADVANCED FEATURE.
You can't do damage, but also don't complain your images look bad.

Various people have hassled over the years to be able to disable ISP processing blocks, predominantly lens shading.
With firmware update https://github.com/Hexxeh/rpi-firmware/ ... bf2902bb54 that should be possible.

I will nudge Dom as he hasn't updated userland with the updated headers. Currently you'll need to modify interface/mmal/mmal_parameters_camera.h

Code: Select all

   MMAL_PARAMETER_CAMERA_RX_CONFIG,          /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */
   MMAL_PARAMETER_CAMERA_RX_TIMING,          /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */
   MMAL_PARAMETER_DPF_CONFIG,                /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+
+   /* 0x50 */
   MMAL_PARAMETER_JPEG_RESTART_INTERVAL,     /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+  MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE,  /**< Takes a @ref MMAL_PARMAMETER_UINT32_T */
};

The 32bit value is a bitmask of stages which is ANDed with the stages enabled by the tuning, although a value of 0 means "don't change anything" instead of "disable everything". A couple I have forced to be on to stop the whole pipeline stalling (eg input & output formatters, resize, and demosaic).

Bit usage for blocks that are enabled and not forced on :
bit 2 - black level compensation
bit 3 - lens shading.
bit 5 - white balance gain
bit 7 - automatic defective pixel correction
bit 9 - crosstalk
bit 18 - gamma
bit 22 - sharpening
There are a couple of others that I haven't documented but are active - turning them off will create very odd results as you'll then be omitting colour conversion stages. That's not to say that disabling any/all of the above will give good results - the blocks have been set up as they are to give best image quality for the standard cameras, and there is no guarantee that the default "bypass" mode of operation in a block is the same in terms of overall gain or otherwise.

A call inserted into an app before the camera component is enabled of

Code: Select all

mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~ 0x08);

will disable lens shading.

If you have a play with this then hopefully you can realise quite how much processing is happening.
I'm not intending to expose any more functionality of the ISP via this route, so please don't ask.

[XFer012]

Hello!

Excuse my noob question: is a "firmware update" like this automatically applied by Raspbian update (apt-get update + apt-get upgrade), or should we manually replace the old files with the new ones (in /boot etc.)?

Thanks A LOT for the new functionality by the way!!!

[jamesh]

It will get to Raspbian eventually, which means the normal update process will work, before then use rpi-update, but beware this get the latest bleeding edge firmware with no other guarantees.

[6by9]

It will get to Raspbian eventually, which means the normal update process will work, before then use rpi-update, but beware this get the latest bleeding edge firmware with no other guarantees.

Bleeding edge firmware and kernel. And it's not only the firmware that can suffer regressions.

[WIX]

Thank you very very much 6b9 !.
I just tried it without success (I don't know what I'm doing wrong but I arrived to pi around 2 moths ago so my knowledge about pi is very limited).
First I was updated firmware OK (using "sudo rpi-update") then I modified my custom camera software without success.
Then I work only with raspivid with same results.
I fall in 2 scenarios when I set MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE.
1-After camera component creation and before connect camera component output to other component mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~ 0x08) return OK but nothing happen on video.
2- After camera component connection to other component any intent to write to MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE fail with error 3 : "mmal: Argument is invalid" . Please let me know if you note I'm doing anything wrong.
Thank you very much again!!.

[6by9]

I just tried it without success (I don't know what I'm doing wrong but I arrived to pi around 2 moths ago so my knowledge about pi is very limited).
First I was updated firmware OK (using "sudo rpi-update") then I modified my custom camera software without success.
Then I work only with raspivid with same results.
I fall in 2 scenarios when I set MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE.
1-After camera component creation and before connect camera component output to other component mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~ 0x08) return OK but nothing happen on video.
2- After camera component connection to other component any intent to write to MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE fail with error 3 : "mmal: Argument is invalid" . Please let me know if you note I'm doing anything wrong.

1 should work. 2 won't - plumbing the request through whilst the pipe was running was a greater risk for minimal gain.

Lens shading isn't actually as strong as I thought it was, so the effect is moderately muted. Hold a sheet of white paper up and there should be variation between the centre and edges. AWB will be counteracting that somewhat, although it only applies global adjustment rather than variation over the image.
Try ~0x01000000 instead. That should be one of the colour conversions so is very obvious that something has changed in the pipe.

[WIX]

Try ~0x01000000 instead. That should be one of the colour conversions so is very obvious that something has changed in the pipe.

I tried to set many different values but nothing happens on video image (returns success). I think may be is something dirty on my system because I tried at least 4 different userland and installed many programs. May be a dirty system? I should reinstall from 0 and start from clean system? Other ideas? .

[6by9]

Try ~0x01000000 instead. That should be one of the colour conversions so is very obvious that something has changed in the pipe.

I tried to set many different values but nothing happens on video image (returns success). I think may be is something dirty on my system because I tried at least 4 different userland and installed many programs. May be a dirty system? I should reinstall from 0 and start from clean system? Other ideas? .

"sudo rpi-update", "sudo reboot", "vcgencmd version" and check you've got the Mar 2 2017 firmware.
Fetch the latest userland and rebase - the required MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE enum is now in the repo.
My git diff:

Code: Select all

diff --git a/host_applications/linux/apps/raspicam/RaspiStill.c b/host_applications/linux/apps/raspicam/RaspiStill.c
index 0c3cf3f..099541a 100644
--- a/host_applications/linux/apps/raspicam/RaspiStill.c
+++ b/host_applications/linux/apps/raspicam/RaspiStill.c
@@ -1006,6 +1006,13 @@ static MMAL_STATUS_T create_camera_component(RASPISTILL_STATE *state)
       goto error;
    }
 
+   status = mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, ~0x01000000);
+
+   if (status != MMAL_SUCCESS)
+   {
+      vcos_log_error("Could not set block override - update your firmware : error %d", status);
+   }
+
    preview_port = camera->output[MMAL_CAMERA_PREVIEW_PORT];
    video_port = camera->output[MMAL_CAMERA_VIDEO_PORT];
    still_port = camera->output[MMAL_CAMERA_CAPTURE_PORT];

"./buildme"
"./build/bin/raspistill -t 5000" (I never fully trust whether the installation step works or not so make sure I run the version I want)
~0x01000000 produces some very colourful images.
~0x04 (black level disabled) produces more saturated images with whites blowing out. It's compromised the statistics, so AE/AGC/AWB will all have gone a bit crazy.
~0x08 (lens shading disabled) does produce a vignetting if I hold up a white sheet of paper to the sensor.

So that all appears to be working to me.

[ethanol100]

I just want to confirm that it is working. I did add the ISP block blocking code to RaspiCamControl.c/h instead(similar to the annotation stuff, for easy testing...). As a side node warning do not block 2^25, this will lock up raspistill, and you will need a reboot

[6by9]

I just want to confirm that it is working. I did add the ISP block blocking code to RaspiCamControl.c/h instead(similar to the annotation stuff, for easy testing...). As a side node warning do not block 2^25, this will lock up raspistill, and you will need a reboot

Thanks for testing. I was avoiding putting this in the public API of the raspicam apps as it can screw the images up so comprehensively.

Typo spotted
I had intended to force bits 0x03a010411 (if I've done the conversion right), but managed to write it as

Code: Select all

en1 = value & 0xffff;
en2 = (value >>16) & 0xffff;
en1 |= 0x0411;
en1 |= 0x03a0;

It got through code review too! I'll update and it'll be released at some point.
Suffice to say disabling any of 0x03a00000 will cause issues to various degrees - they're the main resize and output formatter blocks The other blocks forced on are the input formatters, stats generation, and demosaicing.

[ethanol100]

Thanks for testing. I was avoiding putting this in the public API of the raspicam apps as it can screw the images up so comprehensively.

Yes, I agree, this things should not be included into raspistill etc. For me it was just the more fitting part to put them in for some testing. Thank you for your explanation. These simple typos are always the worst.

[ddahms]

FWIW I tried using this new feature to disable lens shading and it worked for me, in a tweaked version of raspistill.

It looks like “lens shading” includes both the software compensations for the red-to-cyan radial color gradient as well as the darker periphery of standard vignetting.

[6by9]

FWIW I tried using this new feature to disable lens shading and it worked for me, in a tweaked version of raspistill.

It looks like “lens shading” includes both the software compensations for the red-to-cyan radial color gradient as well as the darker periphery of standard vignetting.

It is a set of compensations for each of the red, blue, green(red), and green(blue) channels in the Bayer pattern. If there is a difference in the response of the sensor between the colour channels then the compensation for that channel will hold it. There isn't a "global" setting and then independent ones for red and blue.

And none of the processing is in software - it's all programmed into hardware blocks.

[WIX]

Yes. On Raspistill works fine but on Raspivid not visible effect (almost for me and I set everywhere on file). May be Raspivid overrides this setting, where?

[sharix]

Thank you, I compiled it on a Pi B and it works.
Here is an album of a comparison of the default raspistill and one with disabled lens shading corrections, both with and without a stats pass. Camera is the official PiCam v1 with an additional fisheye lens attached in front of the factory lens. Common options for all four photos are: -rot 180 -w 800 -h 600.
Command used to take the photos:

Code: Select all

~/raspistill_disable_lens_shading/userland/build/bin/raspistill -rot 180 -w 800 -h 600 -o test_nols.jpg:
/usr/bin/raspistill -rot 180 -w 800 -h 600 -o test_ls.jpg;
~/raspistill_disable_lens_shading/userland/build/bin/raspistill -rot 180 -w 800 -h 600 -st -o test_nols_st.jpg;
/usr/bin/raspistill -rot 180 -w 800 -h 600 -st -o test_ls_st.jpg

http://imgur.com/a/YVtoT
correction: the third image has wrong description. Shading correction was enabled.

[thanasispap]

FWIW I tried using this new feature to disable lens shading and it worked for me, in a tweaked version of raspistill.

It looks like “lens shading” includes both the software compensations for the red-to-cyan radial color gradient as well as the darker periphery of standard vignetting.

It is a set of compensations for each of the red, blue, green(red), and green(blue) channels in the Bayer pattern. If there is a difference in the response of the sensor between the colour channels then the compensation for that channel will hold it. There isn't a "global" setting and then independent ones for red and blue.

And none of the processing is in software - it's all programmed into hardware blocks.

Sorry but I don't get it. Lens shading correction affects color channels? What exactly did you mean "difference in the response".

If I put camera v2 on a white target with white light illumination and with lens removed from sensor then with awb I take this picture;

https://drive.google.com/file/d/0B3WDu1 ... sp=sharing

and this;

https://drive.google.com/file/d/0B3WDu1 ... sp=sharing

This green-ish spot in center is a product of sony's image processing pipeline?

[jamesh]

FWIW I tried using this new feature to disable lens shading and it worked for me, in a tweaked version of raspistill.

It looks like “lens shading” includes both the software compensations for the red-to-cyan radial color gradient as well as the darker periphery of standard vignetting.

It is a set of compensations for each of the red, blue, green(red), and green(blue) channels in the Bayer pattern. If there is a difference in the response of the sensor between the colour channels then the compensation for that channel will hold it. There isn't a "global" setting and then independent ones for red and blue.

And none of the processing is in software - it's all programmed into hardware blocks.

Sorry but I don't get it. Lens shading correction affects color channels? What exactly did you mean "difference in the response".

If I put camera v2 on a white target with white light illumination and with lens removed from sensor then with awb I take this picture;

https://drive.google.com/file/d/0B3WDu1 ... sp=sharing

and this;

https://drive.google.com/file/d/0B3WDu1 ... sp=sharing

This green-ish spot in center is a product of sony's image processing pipeline?

No, it's the Pi's ISP that is doing the lens corrections. And yes, it acts on all the colour channels since there is no 'white' channel, just two greens, one red one blue (Bayer pattern).

[6by9]

This green-ish spot in center is a product of sony's image processing pipeline?

A product of physics.
https://en.wikipedia.org/wiki/Vignettin ... vignetting
The amount of vignetting varies depending on frequency (colour) of light, hence needing separate compensation for the red, green, and blue channels (due to the Bayer pattern you have 2 green channels).

AWB is expecting to get a lens shading corrected image. It is trying to make the average scene the "correct colour", so give it an image which has vignetting may well result in the centre effectively having negative vignetting to make the average correct.

[thanasispap]

According to what you have written above, by disabling the lens shading ISP should fix the problem, right?

Also, today I updated to the latest firmware. In order to use the "Disabling ISP Blocks" feature, do I have to compile MMAL or userland again? Because I updated the mmal_parameters_camera.h header (I dynamically link MMAL with my app) and tried to disable various ISP blocks and, although I get MMAL_Success, I don't actually notice any difference at the image I get at all (I mean in general, not only the greenish spot).

[6by9]

According to what you have written above, by disabling the lens shading ISP should fix the problem, right?

If your lens arrangement has absolutely no vignetting, then disabling the lens shading will fix the problem.
If there is any amount of vignetting still present, then AWB may cause further issues.

Also, today I updated to the latest firmware. In order to use the "Disabling ISP Blocks" feature, do I have to compile MMAL or userland again? Because I updated the mmal_parameters_camera.h header (I dynamically link MMAL with my app) and tried to disable various ISP blocks and, although I get MMAL_Success, I don't actually notice any difference at the image I get at all (I mean in general, not only the greenish spot).

All MMAL parameters are just enums passed through to VideoCore to process. As long as your app is inserting the correct enum value in for the parameter then that should be fine without recompiling userland.
What date/hash do you get from "vcgencmd version"? If it is before 18th March then you haven't updated correctly and haven't got the fix for disabling YUV stages. This feature is currently only on firmware obtained by rpi-update, not apt-get.

[jbeale]

"If your lens arrangement has absolutely no vignetting"

I am not sure but I understand some sensors* incorporate single-pixel-scale lenses (micro-lens array) mounted against the sensor itself, and this may also be constructed to compensate to some extent vignetting from the intended external lens assembly. Therefore if you remove the main lens and use something else, you may not have a "vignette-free" result even if the external lens is perfect. This is all just my own speculation and I have no idea if the Sony sensor is actually designed this way.

* See for example http://image-sensors-world.blogspot.com ... ation.html

[thanasispap]

You were right, I did have an older firmware of Mar 3. I update using rpi-update and now I have:
Mar 24 2017 13:57:27
Copyright (c) 2012 Broadcom
version 7e9eb8ef90a1cfb42aca39c27807d2d145a403a1 (clean) (release)
I did try again but no luck, tried to disable various blocks and still I don't get any different image.

*EDIT:
I also recompiled userland and installed it again, but still no luck. I am using video port, not still port, on my app, does this affect the result? I was thinking if this ISP omit is available only on still port.

[6by9]

"If your lens arrangement has absolutely no vignetting"

I am not sure but I understand some sensors* incorporate single-pixel-scale lenses (micro-lens array) mounted against the sensor itself, and this may also be constructed to compensate to some extent vignetting from the intended external lens assembly. Therefore if you remove the main lens and use something else, you may not have a "vignette-free" result even if the external lens is perfect. This is all just my own speculation and I have no idea if the Sony sensor is actually designed this way.

* See for example http://image-sensors-world.blogspot.com ... ation.html

Indeed the micro lens arrangement can have an effect - I had mentally included it in my comment but not explicitly stated it. The sensor manufacturer will generally make the assumption that you are using a lens, and very plausibly make some compensation for that in the micro lens array.
I'm not an optics expert, but I believe that this is what sensor manufacturers specify as the chief ray angle. Sony do provide a graph of CRA vs image height in the datasheet - it varies from 0 to 30degrees. They also provide a shading profile in the Module Design Reference Manual, including how it varies with with non-optimal CRA. (I'm not going to link to them, but they are out there and have been linked to before).

The optical system (including the micro-lens array) has to be considered as a whole. Slapping any old lens on the front will give varying results. This is one reason I was reluctant to expose disabling lens shading in the first place as it is there for a very good reason, and subsequent algorithms rely on it doing the right thing.

[thanasispap]

Thanks for your time, it did work finally. I had to pass the parameters before enabling the the camera component, now the options work as well as combinations of them. Is there any reference of what each bit does beside the bits mentioned above?

[6by9]

Thanks for your time, it did work finally. I had to pass the parameters before enabling the the camera component, now the options work as well as combinations of them.

First post updated to state that.

Is there any reference of what each bit does beside the bits mentioned above?

No, because disabling them has no useful purpose. The images are processed in various colour spaces, and the other blocks are mainly doing the required conversions between them. Feel free to play though.