alpash1
Posts: 2
Joined: Mon Mar 05, 2018 10:22 am

Best approach to HDR photos with camera v2 module (imx219)

Wed Jun 06, 2018 5:59 pm

Hi everyone,

I'm hoping to do HDR photos with the pi sensor for a scientific application, and I'm slightly confused about what the best approach is.
I'm also very new to HDR so apologies if the question is a simple one.

Is it best to take the different photos (that will be combined to make the HDR image) with different exposure compensation values, or manually vary the shutter speed for the different photos using the camera.shutter_speed parameter?

Also, is it best to use Raspiraw to get raw images for combining to HDR?


Thanks a lot,
Ali

ethanol100
Posts: 540
Joined: Wed Oct 02, 2013 12:28 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Thu Jun 07, 2018 12:15 pm

You have several possibilities to do HDR images, can you explain a bit which scientific application you have in mind? How much time do you have to capture the images?

If you for example look at creating HDR images through a microscope of an object which does not change much, you could create a series of images like:

Code: Select all

for i in -24 -18 -12 -6 0 1 6 12 18 24; do raspistill -ev $i -o hdr_series_${i}.jpg; done

This would try to capture images with Exposure Values of -4,-3,-2,-1,0,1,2,3,4. The command -ev does increase/decrease the EV value in steps of 1/6 EV.

You then can use a software like Luminance-HDR to load the images. You will need to set the exposure values during import and can then create a HDR image.

Depending on the dynamic range you are after, 3 or 5 images could be sufficient.

This lets the camera decide if it changes exposure or ISO value to reach your target brightness.

HermannSW
Posts: 787
Joined: Fri Jul 22, 2016 9:09 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 1:15 pm

Hi ethanol100,

I did read Wikipedia pages
https://en.wikipedia.org/wiki/High-dyna ... ge_imaging
and
https://en.wikipedia.org/wiki/High_dynamic_range.

Then I took a HDR image with smartphone, and it took less than a second.
Each raspistill you propose does take multiple seconds, and you showed to take 9.
An important limitation for HDR photography is that any movement between successive images will impede or prevent success in combining them afterwards.
This for me would mean that taking multiple frames with raspivid with different exposures per frame would be far better to avoid movement, and seems to be done by smartphone HDR recording.

In posting "Re: multi exposure filming"
viewtopic.php?f=43&t=212694&p=1313054#p1313054
I implemented "--shutter2" option allowing to specify two shutter times, one for even numbered frames, and the other for odd numbered. I need to undestand the effect of "-ev ..." to exposure times and can get that understanding by doing I2C command decoding between Pi and camera. With that understanding I could implement "--shutter9" option allowing to pass 9 exposures, and taking frame number X with exposure value (X mod 9). That way you can record a video, and do HDR processing from 9 consecutive frames of that video afterwards. In that posting framerate was 49fps meaning that 9 frames being captured in 0.184s, which makes it much more likely to keep camera steady for such a short time. Even when capturing with 8MP you get 15fps so that taking 9 consecutive frames takes 9/15=0.6s.

What do you think, is it worth to implement --shutter9 or similar option allowing raspivid HDR image taking?

P.S:
Here you can see top stripes from 4 consecutive frames with 2000/8000/2000/8000 μs exposures from "multi exposure" posting:
Image
Last edited by HermannSW on Fri Jun 08, 2018 1:28 pm, edited 2 times in total.
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20751
Joined: Sat Jul 30, 2011 7:41 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 1:20 pm

HermannSW wrote:
Fri Jun 08, 2018 1:15 pm
Hi ethanol100,

I did read Wikipedia pages
https://en.wikipedia.org/wiki/High-dyna ... ge_imaging
and
https://en.wikipedia.org/wiki/High_dynamic_range.

Then I took a HDR image with smartphone, and it took less than a second.
Each raspistill you propose does take multiple seconds, and you showed to take 9.
There are different ways of doing HDR, and on a phone they have very close control of the sensor, which is not available on the Pi camera stack. So you won't get phone levels of HDR speed. And of course, the more frames you have the better the final result, and more frames take more time.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

HermannSW
Posts: 787
Joined: Fri Jul 22, 2016 9:09 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 1:31 pm

So isn't taking 5s 8MP video with such "--shutter9" option good enogh?
Each 9 consecutive frames will be recorded in 0.6s, and keeping camera stable for that time should be doable even with hand.
In 5s you get 75 frames, so 66 different sets of 9 consecutive frames to choose from.

P.S:
> and on a phone they have very close control of the sensor, which is not available on the Pi camera stack.
>
https://github.com/Hermann-SW/userland uses I2C command injection bypassing GPU and has very close control of the sensor!
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5805
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 1:58 pm

HermannSW wrote:
Fri Jun 08, 2018 1:31 pm
So isn't taking 5s 8MP video with such "--shutter9" option good enogh?
Each 9 consecutive frames will be recorded in 0.6s, and keeping camera stable for that time should be doable even with hand.
In 5s you get 75 frames, so 66 different sets of 9 consecutive frames to choose from.
9 frames in 0.6s?
So you're assuming that all exposure times will always be less than the 66ms readout time for the full res mode - that's not unreasonable, but will not hold in many situations.
Several bits that are missing:
- with the default setup the sensor will be stopped after each capture and restarted, therefore it requires a dropped frame. Halve your frame rate.
- if the sensor is kept streaming then changes to the exposure time or gain are not applied to the next frame out of the sensor - it's typically delayed by 2 frames.

If my memory serves me correctly and you want to capture a sequence of images with different EV settings, then MMAL_PARAMETER_CAPTURE_EXPOSURE_COMP combined with MMAL_PARAMETER_CAMERA_BURST_CAPTURE will set the ev between successive captures in the minimum amount of time. The standard MMAL_PARAMETER_EXPOSURE_COMP control only runs during preview. I couldn't comment on whether this technique was used elsewhere.
HermannSW wrote: P.S:
> and on a phone they have very close control of the sensor, which is not available on the Pi camera stack.
>
https://github.com/Hermann-SW/userland uses I2C command injection bypassing GPU and has very close control of the sensor!
And plenty of scope for things going wrong due to no arbitration between GPU and ARM for the use of the I2C peripheral. This can never be recommended as a solution, certainly without a long list of caveats.
Unless I've missed it then you also have no indication of when the sensor started a frame, therefore you can't determine when your new setting will be reflected in the output image. So actually not very close control at all.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20751
Joined: Sat Jul 30, 2011 7:41 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 2:14 pm

HermannSW wrote:
Fri Jun 08, 2018 1:31 pm
So isn't taking 5s 8MP video with such "--shutter9" option good enogh?
Each 9 consecutive frames will be recorded in 0.6s, and keeping camera stable for that time should be doable even with hand.
In 5s you get 75 frames, so 66 different sets of 9 consecutive frames to choose from.

P.S:
> and on a phone they have very close control of the sensor, which is not available on the Pi camera stack.
>
https://github.com/Hermann-SW/userland uses I2C command injection bypassing GPU and has very close control of the sensor!
To do HDR properly requires some major changes to the camera stack. Intercepting I2C doesn't get close to the level of control of ISP and the control loops required.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

HermannSW
Posts: 787
Joined: Fri Jul 22, 2016 9:09 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 2:32 pm

OK, good to know.
So ethanol100's proposition with loop of raspistill commands does not work?
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20751
Joined: Sat Jul 30, 2011 7:41 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Fri Jun 08, 2018 2:40 pm

HermannSW wrote:
Fri Jun 08, 2018 2:32 pm
OK, good to know.
So ethanol100's proposition with loop of raspistill commands does not work?
If you have multiple frames, taken at different exposures then postprocessing to a single image, and you could do something. Doing it quickly in the ISP which combines the multiple frames and the postprocessing) is not currently possible.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

alpash1
Posts: 2
Joined: Mon Mar 05, 2018 10:22 am

Re: Best approach to HDR photos with camera v2 module (imx219)

Tue Jun 19, 2018 10:36 pm

Hi everyone,

Thanks so much for these replies - so much combined wisdom here! I lost track of this thread and didn't realise anyone had got back to me...

So, I'm trying the approach of multiple shots at different exposure_compensation using PiCamera (-25,-12,0,12,25) but the contrast doesn't seem to be hugely different between even -25 and 25. Any ideas on how to get more contrast variation? I think I remember seeing something about night mode having more variation?

Also, would be be good to get the shot-taking as fast as possible. Here's the main thing I've thought of so far:
Pre-set wb gains and first exposure time as lighting is always the same, and samples very similar. Skip waiting time

@HermannSW - I'd definitely be interested if I could capture the frames faster using -shutter9. I guess it can basically change settings faster on the cam?

Also, am I right in thinking ISO100 would have least noise and so be best option, or is it more of a trade-off between low ISO and faster shutter speed (so less blur)?


Thanks so much everyone,
Ali

ethanol100
Posts: 540
Joined: Wed Oct 02, 2013 12:28 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Wed Jun 20, 2018 11:39 am

I got better results setting exposure time manually capturing a test image first:

Code: Select all

raspistill -ag 1 -dg 1 -o test.jpg

Looked for the settings used in this image:

Code: Select all

strings test.jpg | grep gain
which delivers something like:

Code: Select all

#ev=-1 mlux=-1 exp=1030 ag=256 focus=255 gain_r=1.933 gain_b=1.242 greenness=4 ccm=7596,-3496,2,-928,6540,-1512,494,-2756,6364,0,0,0 md=0 tg=256 256 oth=0 0 b=0 f=256 256 fi=0 ISP Build Date: Jun  1 2018, 14:04:35 VC_BUILD_ID_VERSION: 478d13bd2bb41a9c77c03d981a23b31e872b3683 (clean) VC_BUILD_ID_USER: dc4 VC_BUILD_ID_BRANCH: master
.

Then I wrote a script like this, which takes 5 images with ev 0,1,2,0,-1,-2:

Code: Select all

#!/bin/bash
raspistill -t 30 -awb off -awbg 1.933,1.242 -ss $((1030*1)) -ag 1 -dg 1 -o 01.jpg
raspistill -t 30 -awb off -awbg 1.933,1.242 -ss $((1030*2)) -ag 1 -dg 1 -o 02.jpg
raspistill -t 30 -awb off -awbg 1.933,1.242 -ss $((1030*4)) -ag 1 -dg 1 -o 03.jpg
raspistill -t 30 -awb off -awbg 1.933,1.242 -ss $((1030/2)) -ag 1 -dg 1 -o 04.jpg
raspistill -t 30 -awb off -awbg 1.933,1.242 -ss $((1030/4)) -ag 1 -dg 1 -o 05.jpg
To capture the image series, it still takes something like 5-10s to capture.
Image Image Image Image Image

Loading the images into luminance hdr did load the ev values automatically. Where I could save a hdr image or create a mapped images like this:
Image

Its far from perfect, as the grass as moved quite a bit in the lower right corner, etc. But you can create something with a higher dynamic range.

Which kind of objects are you trying to capture?

HermannSW
Posts: 787
Joined: Fri Jul 22, 2016 9:09 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Wed Jun 20, 2018 7:56 pm

Hi ethanol100,

very impressive what you did, and that you showed a method using -ss instead of -ev.

I replicated what you did, can you please take the 5 images and show what luminance hdr makes of it?
https://stamm-wilbrandt.de/en/forum/hdr/01.jpg
https://stamm-wilbrandt.de/en/forum/hdr/02.jpg
https://stamm-wilbrandt.de/en/forum/hdr/03.jpg
https://stamm-wilbrandt.de/en/forum/hdr/04.jpg
https://stamm-wilbrandt.de/en/forum/hdr/05.jpg

This is what I did according your instructions:

Code: Select all

[email protected]:~ $ raspistill -md 4 -w 1640 -h 1232 -ag 1 -dg 1 -o test.jpg
[email protected]:~ $ strings test.jpg | grep gain
ev=-1 mlux=-1 exp=27779 ag=256 focus=255 gain_r=0.800 gain_b=2.691 greenness=3 ccm=6022,-2314,394,-936,4728,310,300,-4324,8126,0,0,0 md=0 tg=254 317 oth=0 0 b=0 f=254 317 fi=0 ISP Build Date: Apr 16 2018, 18:20:48 VC_BUILD_ID_VERSION: af8084725947aa2c7314172068f79dad9be1c8b4 (clean) VC_BUILD_ID_USER: dc4 VC_BUILD_ID_BRANCH: HEAD 
[email protected]:~ $ raspistill -t 30 -awb off -awbg 0.8,2.691 -ss $((27779*1)) -ag 1 -dg 1 -o 01.jpg
[email protected]:~ $ raspistill -t 30 -awb off -awbg 0.8,2.691 -ss $((27779*2)) -ag 1 -dg 1 -o 02.jpg
[email protected]:~ $ raspistill -t 30 -awb off -awbg 0.8,2.691 -ss $((27779*4)) -ag 1 -dg 1 -o 03.jpg
[email protected]:~ $ raspistill -t 30 -awb off -awbg 0.8,2.691 -ss $((27779/2)) -ag 1 -dg 1 -o 04.jpg
[email protected]:~ $ raspistill -t 30 -awb off -awbg 0.8,2.691 -ss $((27779/4)) -ag 1 -dg 1 -o 05.jpg
[email protected]:~ $ 

I could add "--shutter5" to my userland fork of raspivid.
It would take the specified shutter value and produce video with these frame shutter speeds:

Code: Select all

5*i+0: shutter/4
5*i+1: shutter/2
5*i+2: shutter
5*i+3: shutter*2
5*i+4: shutter*4


This way any 5 consecutive frames can be taken to create HDR image, and 1640x1232 can be captured at 40fps, so each 5 frames would be taken within 0.125s and not in more than 5s. Would this "--shutter5" option make sense?

P.S:
Perhaps a HDR video can be created from that video as well, frame i of HDR video built from frames i+0,i+1,...,i+4 of shutter5 video?
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

ethanol100
Posts: 540
Joined: Wed Oct 02, 2013 12:28 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Wed Jun 20, 2018 9:47 pm

The dynamic range of the scene is not great and is quite blurry. It is difficult to find a good setting. The result does not look good.

Image

This is the link to the "real" HDR image, you can use i.e. GIMP 2.10 to open it.

metalmajor
Posts: 33
Joined: Fri Oct 20, 2017 9:40 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Tue Aug 28, 2018 12:09 pm

What is the current situation with HDR capture on V2.1?

So I understand you can't do it by setting some parameter in the mmal, and then you get a HDR image from the preview pin for example?
And what are the future plans for making this work? A new camera module(3.0?) or a software upgrade(and is there a timeline for that?)

And I also understood that you can only do it when you are merging the images together yourself, And you need 3 to 5 images for this merge, with different EV settings, and they are only applied after 2 frames, so at 30 fps, that is maybe taking like 10 frames you need to wait to capture 5 frames and then you need to have some code that will merge the 5 frames together. So maximum is then 3 FPS maybe. Is that correct?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5805
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Best approach to HDR photos with camera v2 module (imx219)

Tue Aug 28, 2018 12:28 pm

metalmajor wrote:
Tue Aug 28, 2018 12:09 pm
What is the current situation with HDR capture on V2.1?

So I understand you can't do it by setting some parameter in the mmal, and then you get a HDR image from the preview pin for example?
And what are the future plans for making this work? A new camera module(3.0?) or a software upgrade(and is there a timeline for that?)
No intention of natively supporting HDR.
There was some support in the Broadcom days, but it's never been brought up to production quality.
metalmajor wrote:And I also understood that you can only do it when you are merging the images together yourself, And you need 3 to 5 images for this merge, with different EV settings, and they are only applied after 2 frames, so at 30 fps, that is maybe taking like 10 frames you need to wait to capture 5 frames and then you need to have some code that will merge the 5 frames together. So maximum is then 3 FPS maybe. Is that correct?
If you're lucky you might possibly get 2fps - capture one frame, request new exposure/gain, wait 2 frames, capture one frame, etc. And that would be assuming that all the exposure times are below 33ms and you're using the binned mode. If using the full 5 or 8MPix stills mode then the sensor is only running at 15fps.

HDR is a very involved process. It can be done in the Bayer or YUV domains, although the Bayer domain in theory has some advantages. Integrating with sensors is also involved.
If you have a proposal as to how your algorithm works and what frames it wants, then you may get some advice on how to minimise the capture times. In all honesty these sensors are not the best designed for doing HDR captures.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

metalmajor
Posts: 33
Joined: Fri Oct 20, 2017 9:40 pm

Re: Best approach to HDR photos with camera v2 module (imx219)

Tue Aug 28, 2018 12:41 pm

Thanks for this answer!

Just one question then: in some press articles about the "new 8MP pi camera module" you see a lot of times "supports BME-HDR". What is that then in fact? Or is that just not true?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5805
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Best approach to HDR photos with camera v2 module (imx219)

Tue Aug 28, 2018 12:58 pm

metalmajor wrote:
Tue Aug 28, 2018 12:41 pm
Just one question then: in some press articles about the "new 8MP pi camera module" you see a lot of times "supports BME-HDR". What is that then in fact? Or is that just not true?
Full quote from http://www.sony-semicon.co.jp/products_ ... 219_e.html
Additionally, combining with a rear-end ISP supporting of BME-HDR enables future scalability of the high-dynamic-range video imaging.
The rear-end ISP does not support BME-HDR.

TBH I've not even looked at how the BME-HDR mode presents the data over the CSI2 bus. Having looked in the datasheet for the sensor there is no mention of BME, HDR, "high dynamic" or smilar terms.
It does support two banks of settings for exposure, gain, frame length, etc, and switching between the banks (manually only it appears). The sensor does return all the register settings in the embedded data line, so it would be possible to extract that and ensure that therefore determine whether you have the correct frame settings or not. I'll point you at raspiraw where you have full control over the sensor. It's not something that I expect to be adding to the firmware.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”