ROI and metering

[jpers]

How does metering algorithm (algorithms) behave when the "roi" option specifies only a part of the image to read?

a) does it work on the whole image, and ROI is applied as a last step?
b) does it measure the inputs only on the ROI (assuming then that ROI is the first step)?

Thanks,

Janez.

[jpers]

Latest version OpenCV 3.2.0. But the better one is Deep Learning
a) Yes
b) Yes Using houghLineS, but not houghlinesP

What?

My question has nothing to do with OpenCV, deep learning, or Hough transform.

I simply want to know how metering algorithms work when ROI is set (e.g. raspivid --roi).

Janez.

[jamesh]

I'm not sure...! Trying to remember where the roi is implemented in the ISP, but not got it dso far. 6x9 isn't in the office so cannot ask him. Will try and remember to ask him when he is back in. My vague recollection is that the crop is done fairly early on in the process, by the CSI peripheral, so the metering is done on the crop not the whole image, but for real confirmation you will need to wait!

[jpers]

My vague recollection is that the crop is done fairly early on in the process, by the CSI peripheral, so the metering is done on the crop not the whole image, but for real confirmation you will need to wait!

Thanks! Fingers crossed that that turns out to be true, it will solve a major problem we have in our application

[6by9]

I'm not sure...! Trying to remember where the roi is implemented in the ISP, but not got it dso far. 6x9 isn't in the office so cannot ask him. Will try and remember to ask him when he is back in. My vague recollection is that the crop is done fairly early on in the process, by the CSI peripheral, so the metering is done on the crop not the whole image, but for real confirmation you will need to wait!

Not in the office, but lurking.

It's complicated.
The CSI-2 peripheral can crop, but that isn't used (you may be remembering zoom modes, but those were done on the sensor, not in CSI-2 rx).

Crop is actually done as part of the high res resize almost right at the of the pipe, so the stats block is seeing the whole image.
It does appear, however, that at least auto gain and exposure control is taking in the crop region details when computing metering areas - anything cropped off will therefore be ignored for metering purposes. I can't see a similar compensation being applied for auto white balance though.

[jpers]

It does appear, however, that at least auto gain and exposure control is taking in the crop region details when computing metering areas - anything cropped off will therefore be ignored for metering purposes. I can't see a similar compensation being applied for auto white balance though.

Thanks. We have problems with underexposure/overexposure using wide angle lens, so this helps a lot. Don't care about wb.

I will test it out, and report back.

Thank you.

Janez.

[jpers]

Crop is actually done as part of the high res resize almost right at the of the pipe, so the stats block is seeing the whole image.
It does appear, however, that at least auto gain and exposure control is taking in the crop region details when computing metering areas - anything cropped off will therefore be ignored for metering purposes. I can't see a similar compensation being applied for auto white balance though.

For anyone that stumbles upon this thread: After running some experiments with Raspberry Pi 3, picamera V1 and early March 2017 Raspbian jessie boot image, I can confirm that auto gain/exposure metering is done relative to the ROI window. Did not test the white balance.

Therefore, using wide-angle lens, the raspicam can be used as a virtual pan-tilt camera by manipulating ROI. Although not related to metering, some practical notes:

• 1. Choose and fix the sensor mode before playing with ROI. For picamera v1 there is a nice table here: http://picamera.readthedocs.io/en/release-1.10/fov.html Did not find corresponding table for picamera v2.
  2. You will probably want to use modes without binning and full FOV, which (at least for picamera v1) leaves only one choice, that is mode 2.
  3. If using raspivid, fix the mode using -md 2 argument, and then -roi xmin,ymin,width,height. The coordinates are normalized (in range from 0 to 1, where 0 is top left, and 1 is bottom right corner).
  4. important To get proper crop-out, you have to take into the account sensor aspect ratio and explicitly specify -w and -h! The ROI is essentially digital zoom, without this your ROI will be resized to full sensor size, and aspect size will be wrong.
  5. Therefore, if you want to crop 1280x720 ROI starting at pixels 100,200 with sensor mode 2 (2592x1944 with 4:3 aspect ratio) you calculate ROI parameters as follows: xmin=100/2592, ymin=200/1944, width=1280/2592, height=720/1944 and you will use -w 1280 -h 720. Now your output image will have correct dimensions and aspect ratio.

Typed all this from memory, hopefully did not make a mistake somewhere.

Janez.

[6by9]

1. Choose and fix the sensor mode before playing with ROI. For picamera v1 there is a nice table here: http://picamera.readthedocs.io/en/release-1.10/fov.html Did not find corresponding table for picamera v2.
[/list]

You're looking at V1.10 of those docs. V1.11 included the V2 camera modes. It's now up to V1.13, or you can view latest which also includes a load more information on what is actually going on under the bonnet.

Please remember that mode 2 does limit you to 15fps.

4. important To get proper crop-out, you have to take into the account sensor aspect ratio and explicitly specify -w and -h! The ROI is essentially digital zoom, without this your ROI will be resized to full sensor size, and aspect size will be wrong.
5. Therefore, if you want to crop 1280x720 ROI starting at pixels 100,200 with sensor mode 2 (2592x1944 with 4:3 aspect ratio) you calculate ROI parameters as follows: xmin=100/2592, ymin=200/1944, width=1280/2592, height=720/1944 and you will use -w 1280 -h 720. Now your output image will have correct dimensions and aspect ratio.

I haven't done exhaustive tests there, but quick tests and my recollection of the setup is that the crop to match the output aspect ratio is done first, and then the roi is applied. That means if you ask for -w 1280 -h 720 (or any aspect ratio different to the sensor mode), the input image of 2592x1944 has 729 lines cropped off top and bottom to give 2592x1458. You can't get an image of those cropped lines regardless of how you set roi.
That being done, then the roi range of 0-1 is always with regard the aspect ratio corrected image and not the full FOV.

Doing the same as your example but asking for a 32:9 image of 1280x360 would give an roi with w/h 0.494 and 0.185. Doing that I get an image stretched vertically. 0.494,0.494 gives an undistorted image.
I did try your numbers of 0.494 and 0.370 but couldn't convince myself whether it was right or wrong hence using an even wider aspect ratio.