User avatar
schoolpost
Posts: 26
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

PyDNG - RPI Camera RAW to DNG

Tue Apr 11, 2017 5:52 am

Hello Everyone!

Excited to share this small project I've been working on:
https://github.com/schoolpost/pydng

(raw left, jpeg right )
Image

Requires: Python 2.7 & Numpy

PyDNG lets you convert your picamera RAW containing .jpgs into Adobe DNG RAW files
for use in a wide variety of image editing applications.
( Photoshop, Lightroom, RawTherapee, etc.. )

Bayer data gives you the most flexibility to really tune your image for the best results.
Paired with Lossless compression when output from Adobe DNG Converter
or 3rd Party application like SlimRAW & enjoy an ~2:1 compression at no cost! ( 4-6MB )

Features:
-10/16-bit encoding options
-Flat field correction ( lens shading )
-Hot pixel correction
-Dark frame subtraction

Some future considerations:
-Copy EXIF Data from .jpg to .dng
-Fix magenta tinted highlights
-Improve 10bit/8bit encoding in DNG_10.py
-Apply calibrated color matrix values
-Automated calibration script to let users generate dark/shade/pxl from input images

While the output is a mostly functional dng, for best compatbillity users should convert their DNG's through AdobeDNGConverter to fix any problems. ( and get some free lossless compression as a bonus )

Hopefully this is of some use to somebody out there.

User avatar
schoolpost
Posts: 26
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Tue Apr 09, 2019 2:52 am

Wanted to share my latest update to this project. Version 2.0!

https://github.com/schoolpost/pydng

Store 10-bit bayer data from the PiCamera V2 in the most widely used RAW image format available.

This version improves on all aspects of the original, with improved pre-processing, file encoding and new lossless compression!

Would love for you all to try it, there are aspects still in need of development for those who wish to contribute:
  • Use camera specific color calibration matrix ( the one currently used looks good but is not by any means accurate) ,
  • Detect flipped / rotated images automatically ( change CFA layout based on rotation )
  • Detecting/Mapping dead/hot pixels
  • Copy EXIF data from jpeg to dng tags
  • Improve tile based LJ92 compression ( for even further file size reduction )
10-bit DNG File Size ( 7,201,848 ) vs. 8-bit JPEG File Size ( 3,805,122 )
Image

Thanks for viewing! :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8425
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PyDNG - RPI Camera RAW to DNG

Tue Apr 09, 2019 9:46 am

Detect flipped / rotated images automatically ( change CFA layout based on rotation )
I can't see this in code with a quick look, but I do see hard coded offsets at https://github.com/schoolpost/pydng/blo ... ng.py#L338
Captures can be done in any sensor mode, therefore there are multiple valid offsets. See https://github.com/6by9/dcraw/blob/master/dcraw.c#L8162

The useful parts of the raw header are "documented" at https://github.com/6by9/dcraw/blob/master/dcraw.c#L6465, and that includes the absolute Bayer order and resolution for the capture - no need to be guessing it based on rotation or offset.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

seasalt
Posts: 4
Joined: Thu Jan 16, 2020 5:48 pm

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 6:23 pm

* I havn't had an in depth look at the code but to my knowledge, jpg's are compressed and by converting them to dng, you are not uncompressing them, therefore they can not be true raw files.

Am I wrong in making this assumption?

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12825
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 6:28 pm

seasalt wrote:
Thu Jan 16, 2020 6:23 pm
* I havn't had an in depth look at the code but to my knowledge, jpg's are compressed and by converting them to dng, you are not uncompressing them, therefore they can not be true raw files.

Am I wrong in making this assumption?
No,
JPG's are not a "lossless" format, meaning there is data-loss when compressing, data that can never be recovered.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8425
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PyDNG - RPI Camera RAW to DNG

Thu Jan 16, 2020 8:35 pm

raspistill with the -r flag appends the raw image to the end of the JPEG. This was done originally as a way of getting raw images out of Android where the camera HAL only allowed you to capture JPEGs. Any JPEG viewer will ignore the data after the EOI flag in the JPEG.

So, yes, this project can produce a DNG from a JPEG(+RAW) file captured with raspistill.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
schoolpost
Posts: 26
Joined: Sun Feb 19, 2017 10:47 am
Location: Canada
Contact: Website

Re: PyDNG - RPI Camera RAW to DNG

Mon May 11, 2020 3:58 am

New version is now available! With support for the new Raspberry Pi High Quality Camera ( IMX477 )
https://github.com/schoolpost/PyDNG

Lots of fixes and refactoring has been done. Among other things we have:

  • support for automatic bayer format order detection for flipped images
  • attached exif data ( exposure, lens, shutter, etc.. )

Simple example script included to convert input file ( JPEG+RAW ) into a DNG file.

Code: Select all


python examples/utility.py color.jpg


Another example available demonstrates how to integrate into common picamera workflow you would do in python.

Code: Select all


import picamera
from io import BytesIO
from pydng.core import RPICAM2DNG



with picamera.PiCamera() as camera:

    stream = BytesIO()
    camera.capture(stream, 'jpeg', bayer=True)

    d = RPICAM2DNG()
    output = d.convert(stream)

    with open('image.dng', 'wb') as f:
            f.write(output)



Return to “Camera board”