I'm currently writing a lens correction and image optimization program that (if everything works as planned) will perform the following functions:

1. Removal of image blurring introduced by the lens, camera AA filter, diffraction, and RAW converter demosaic algorithm.

2. Correction of radial and longitudinal chromatic aberrations.

3. Correction of coma, spherical aberrations, and other similar distortions.

4. Correction of barrel, pincushion, and the most complex forms of mustache distortion.

5. Vignetting correction.

6. Stacking multiple DNG images for focus or exposure blending.

For best results, the approach I'm working on will require the user to shoot a specially-designed target containing an array of point sources of white light with each combination of camera, lens, aperture, and focal length to create a "blur profile" customized to that particular combination of hardware. As an alternative, my goal is to allow users to create and share blur profiles (e.g. for a Canon 1Ds wearing a 17-40/4L at f/4 and 17mm). As long as a given user's hardware matches that used to generate the blur profile, results will be good. If the user's hardware is different (e.g. the lens is softer in the corners at 17mm) than the profiled hardware, results will be poor, and the user will need to make their own custom profile. This would be similar to the owner of a printer finding that factory "canned" profiles work poorly and improving output by making a custom printer profile.

The biggest hassle I'm encountering right now is figuring out the DNG SDK. I've downloaded it and browsed through it, but there's a huge number of classes, and the object hierarchy and their intended usages aren't clear to me yet. I'm trying to figure out how to accomplish the following tasks:

1. Open a DNG file (preferably the demosaiced linear RGB flavor), and read the image and metadata (especially camera model, lens model, focal length, and aperture) into memory.

2. Once the image has been processed, write it to a new linear RGB DNG file that can be opened in any DNG-aware application, with all original tags and metadata correctly preserved.

All of the internal correction operations are done on linear RGB data in the camera's native color space, so that setting white balance, selection of color profile, etc. is unaffected by the image correction operation. The user will have the exact same color processing options, ability to select white balance, exposure adjustments, and output color space selection as when working with the original RAW(s).

Any advice from the experts would be greatly appreciated.

While Eric does hang out here, Thomas doesn't. The best place to ask very specific questions relating to DNG would be the DNG Forum...

Thanks for the pointer; I'll post an inquiry there as well. There doesn't appear to be a whole lot of traffic there though, less than 10 threads in the last 6 months in the DNG SDK forum...

Jonathan,

I can't contribute anythng of technical substance to your inquiries - that's for the kind of highly expert people Jeff mentions, but I am very interested in what you are trying to do and would like to remain in loop about how it is progressing. Good luck with it.

Mark

Hi Jonathan,

The best place to start in terms of the code would be dng_validate.cpp. The dng_validate () routine in there will read in a negative, parse through it, and build the various image stages. This includes decompressing the raw image data, linearizing it if necessary, and render to a TIFF file if you want. From a developer's perspective, I'd suggest putting a breakpoint somewhere in that routine, feeding a DNG raw file to it, and then stepping through to see how the code flows. In terms of terminology, the code comments use the term "stage 1" image to describe the raw image data, "stage 2" to describe the linearized raw image data (in a canonical space [0, 1], or [0, 65535] if you prefer), and "stage 3" to describe the 3-channel or 4-channel data (i.e., after demosaic). If you want to do processing on linearized mosaic data, you want to grab the stage 2 image; see the Stage2Image () routine in the dng_negative class (dng_negative.h). If you want to do processing on RGB demosaiced data in the native camera color space, prior to white balance, you want the stage 3 image; see the Stage3Image () routine in the dng_negative class.

Thank you, that's very helpful. The processing I'm doing is all on linear RGB data scaled 0-1 in floating-point format (this makes a lot of the deconvolution math much more straightforward), so I'll need the stage 3 data and normalize it to a maximum value of 1. Three questions:

Is the stage 3 data always scaled to the same output range (say 0-65535) regardless of the number of bits/pixel in the original RAW file, or does the range of image values vary by camera model?

How do I write the modified stage 3 data back to a new DNG file, while copying all metadata from the original DNG?

I've got the dng_validate project downloaded along with XMPCore and expat so dng_validate will build with no errors or warnings. I'd like to compile dng_sdk (along with whatever it needs from expat and XMPCore) as a either a .NET or COM .dll so I can use its functionality from Visual Basic .NET or other .NET-family languages like C#, etc. How can I do this from Visual C++?

Sorry for the stupid questions, my programming background is VB, not C, and I'm having a bit of trouble getting my head wrapped around all of the DNG SDK components and how they relate to each other.

Yes, the stage 3 image data is always normalized to 16 bits [0,65535], independent of camera model. That is, if you have intentionally clipped a pixel value by overexposing, that pixel component will be 65535. Typically the stage 3 image will be very "green" because it is not yet white-balanced.

To modify a DNG, you'll want to take a look at lines roughly 250 thru 320 of dng_validate.cpp. That's where you'll find sample code that writes a DNG file out to disk; the metadata should be preserved, so you don't have to worry about that. Also, the SetStage3Image () routine in the dng_negative class will let you store your modified negative data back to the dng_negative object.

Note that you'll have to save as a linear DNG since your modified data is already 3-channel (no longer in mosaic form). I believe you can do this by using

host.SetSaveLinearDNG (true)

prior to saving out the image.

it looks like your software is running before any raw converter, otherwise why bother to output into linear DNG ? to have a workflow like : in camera raw file -> dng converter (well only ~5% cameras out there will have native in camera DNG) -> raw converter #1 -> your software -> raw converter #2 ? too many steps... and if you want to run it after the raw converter very few will produce non linear DNG file as their output (if any ? I know several programs that operate (modify the raw data) on DNG files w/o making them linear, but they are hardly widely used fully functional raw converters... )... and if you are running before a raw converter than how do you plan to coexist ? does DNG has a tag saying - hello people, I do contain a corrected data (besides just linearizing), please do not bother w/ certain corrections and optimizations ? just curious... or if you plan to act like DxO (raw -> DxO (corrections) -> linear DNG -> ACR/LR to deal w/ WB and some colors) then you have to be a good in demosaicing first of all... so why don't you just write a fully functional raw converter ?

deja, it depends on the extent of the corrections Jonathan wishes to pursue. Some of the items he describes can be expressed as metadata, via DNG opcodes -- see Chapter 7 of the DNG 1.3 spec:

http://www.adobe.com/products/dng/pdfs/dng_spec_1_3_0_0.pdf

ouch, for example (w/o reading .PDF - you should be able to tell that in a blink - I mean to tell about tags/opcodes, not that I am lazy - that I know already myself) do you have a tag (DNG opcode) saying - please, raw converter, do not do any automatic CA/PF correction, it was done already ?

deja, there is not currently such a tag. This is a more general problem for all images, not just DNG. For example, if you have a JPEG or TIFF file lying around somewhere, it can be difficult to tell whether or not it has had some form of lens corrections applied to it. A standard metadata tag would be useful to indicate whether specific forms of processing had been applied to the image, in this case. My understanding is that such tags are under current consideration for standardization, but that could take a while.

OK, thank you that is what I was looking for. Linear RGB DNG export is exactly what I want.

Short answer: I don't want to spend a bunch of time reinventing the wheel when there are many very good RAW converters already out there. OTOH, DXO doesn't really have a lot of competition for really good lens corrections, it's way overpriced, and it's crippled because you are locked into using "canned" blur profiles for cameras and lenses, and if DXO doesn't have profiles for your camera or lens, you're screwed--you can't use it.

If your lens' blur characteristics differ from the DXO-profiled lens, then the "canned" profiles aren't going to work all that well anyway. If a printer manufacturer only allowed the use of their canned print profiles, and didn't allow users to make their own profiles, they'd be laughed out of the market. But that is essentially DXO's business model.



The corrections I'm doing need to be done after demosaic. The deconvolution algorithm I'm working on analyzes a linear RGB image of an array of circular light sources (ideally about 5-10 pixels in diameter in the RAW image) and analyzes the blur characteristics found. The blur in the demosaiced image is a combination of lens blur, AA filter blur, and blur introduced by the demosaic algorithm of the RAW converter. By analyzing the total blur caused by all elements of the image capture process (lens, camera, and RAW converter) all of the blur from every blur source can be removed at once.

The basic workflow is this:

1. Run DNG Converter to convert RAWs to linear-RGB DNG files. Essentially all this does is fills in the missing color channel values from the Bayer matrix (e.g. adds R & B values to a G pixel), and then scales the values 0-65535. This can be done on a batch of files.

2. Process the DNGs through my program to correct lens, AA filter, and demosaic blur, remove CA/color fringing, correct fisheye/barrel/pincushion/mustache distortion, eliminate vignetting, etc., and eventually stack multiple focus and/or exposure-bracketed images into a single output DNG. This can be a batch process also.

3. Open the processed DNGs and edit as desired.

The RGB data is still in the camera's native color space, so you still have total flexibility to set white balance, adjust exposure, process colors, etc. the same as with the original RAW, as long as the RAW converter you use can read DNGs. Using my program will require an extra step in the workflow, but my goal is to make the benefits greatly outweigh the slight inconvenience.

My solution for now at least is simply to look for lens correction opcode tags in the source DNG file, and omit them when writing the destination DNG file. If a tag is ever defined to indicate that lens corrections have already been applied, then I'll add it to the output file...

you will endup w/ a subpar demosaicing in the first place that way... that is the problem here - you will be optimizing not the best possible demosaiced data... not a problem in many cases for many people, but aren't you after the ultimate quality ?

Only if you consider ACR a "subpar" RAW converter. And you're not necessarily limited to ACR; any RAW converter that can export to linear-RGB DNG can be used for demosaicing. The majority of the lens corrections have to be done on demosaiced RGB data; for example, CA corrections (which involve shifting the locations of color channels relative to each other) cannot be saved back to RAW because the pixel locations no longer match the Bayer matrix pattern. Correcting barrel/fisheye distortion has the same problem, only worse because the degree of pixel-shifting es even greater.

It just makes sense to demosaic first, and then apply corrections and adjustments.

Umm, DNG converter has the full ACR demosaicing algorithm embedded in it, rather than a simplified, lower performance version? That would surprise me. But then, I get surprised on a regular basis.....

But the fundamental problem you run into is actually that in order to do lens correction, you have to interpolate pixels. So if you do the correction as a separate step to the demosaicing, you have two layers of interpolation on top of each other. Of course, you can do that, but not, IMHO, a good idea. Which is why I believe that DxO will only write a linear DNG - interpolation for demosaicing and for lens correction are a single step for them. But that's only what I believe

Sandy

the question is about the quality of demosaicing that DNG converter has ?

The testing I've done indicates that opening a linear-RGB DNG (demosaiced in DNG converter) and processing the source RAW directly in ACR makes zero difference--if both files are processed with the same settings, the outputs will be an exact pixel-for-pixel match. Where did you get the notion that DNG converter didn't use the same demosaic engine as ACR? If you think otherwise, do your own comparison.

As to the single interpolation vs double objection, that's a load of crap. Demosaicing compares Bayer pixels to their immediate neighbors to guess the values of the missing channels. That requires looking at the pixels before the color channels are shifted around relative to each other (as in CA correction), or the whole concept of "adjacent pixels" gets flushed down the toilet. You have to demosaic first, then you can juggle the location of the color channels. Doing otherwise is a recipe for crappyocrity, not a quality enhancement.

OK, I deduce you have firmly held views on this subject(!)...................I guess we await your working code with eager anticipation so we can get to see all that quality shine through.

Good luck.

Sandy

Actually, CA correction can be done, and done better, during demosaic rather than after.

http://www.ojodigital.com/foro/perfectraw-...erpolacion.html

Even so, geometric corrections might be better done after demosaic. I'll be interested to see what comes out of your project.

Hi Sandy, yes, you are right: the DNG SDK only has a simple bilinear interpolation demosaic routine implemented. (We have been meaning to update it with something more serviceable, but have not worked that into the already rather-packed dev schedule ...)

Hi Eric - I can sympathize with the dev schedule!!!!

Even a "text book" AHD implementation would be really cool though......and make doing things such as the OP wants to do a lot easier.

Regards,

Sandy

well, too bad for ACR then...

The samples in that thread didn't look that impressive to me; in the side-by-side comparison of the makeup brushes, AMAZE didn't appear to do as well as the other converter it was compared to. Correcting CA during demosaic may not be worse than doing so after, but I didn't see any proof it's significantly better.

The side-by-side with the brushes is a comparison to AMaZE prior to CA correction, the CA corrected version is lower down in the thread (post #11).

Here's a direct comparison with ACR at 400% (using a different image). I set the ACR 4.6 CA correction slider so as to minimize CA artifacting in the blinds (there was no setting that eliminated it altogether), and it still crops up in the church steeple and the tree limbs/branches. Anyway, regardless of the specific implementation, it seems to me that any geometric correction that can be applied in a linear space such as RAW is better done there rather than after the nonlinear operation of demosaic. For instance, CA can corrupt the demosaic process itself, depending on the algorithm used (cf the HPHD examples from RawTherapee in the Ojodigital thread).

Jon, if I understood well you plan to demosaic the RAW data with the DNG routines, and that means you should apply white balance first. You can demosaic with no WB applied, but this will probably affect the efficiency of the demosaic process in terms of getting right colours and detail.

On the other hand, applying a WB doesn't mean you cannot modify it later, but you will have to deal carefully with the highlights in the white balancing process: to preserve them will mean you will move the saturation point to some unknown (or difficult to know) place in the histogram. And this point needs to be known if additional exposure or white balance adjustments are to be done to the output DNG file.

BTW if you would like any advice on how to blend different RAW files (relative exposure calculation, and minimal progresiveness blending) feel free to ask.

Regards

I'm still not convinced that correcting CA during demosaic is the magic bullet you think it is. Other than the CA, the differences in the conversions were minimal, mostly due to slightly different tone curve or exposure settings.

The problem with ACR and most every other RAW converter's CA correction is that it's using a fairly simple mathematical formula (something along the lines of a simple gamma curve) to calculate the pixel shift distance needed to realign the color channels and correct the CA. The problem is that this formula is an approximation many lenses do not follow very well; therefore for many lenses there is no setting that creates the proper curve to correct the CA completely. The best you can do is spread the error fairly evenly so that no part of the image looks too horrible. The approach I'm using creates 3 spline curves with 32-64 data points each (calculated separately for each color channel) so that the color channels can be aligned to within 1/10 of a pixel or so from the center of the image all the way to the corners, regardless of whether the lens' distortion curve is simple or extremely complex.

Most differences between good conversions are minimal, up to tone curve and color space conversion. We're talking about the last little bit of data near Nyquist, otherwise there isn't much to discuss.



There isn't much point in aligning to 1/10 pixel spacing when 2/3 of the data has been interpolated in different directions using colors that are as much as a pixel or more away from where they should be. No mathematical formula with a small number of parameters is going to correct for the fact that a color has been interpolated incorrectly in the horizontal direction in one place, and incorrectly in the vertical direction nearby, due to the mismatched color registration caused by CA. That is why it is important to do the CA correction before demosaic.

Any formula for correction that you may have can be applied before demosaic with the right approach.

Actually it doesn't. If you export a linear-RGB DNG from DNG Converter or Bridge, and then process that DNG and the original RAW file in ACR with the same settings, the result is a pixel-for-pixel match, and you have exactly the same flexibility for setting exposure, white balance, and color adjustments either way.



Thank you. Would you be interested in perhaps combining our projects, to create one DNG processing tool that does everything?

But how do you use Bridge to demosaic a non white balanced RAW file?
At least using DCRAW, the AHD algorithm works quite differently if applied on white balanced RAW data than when used on non white balanced data. Colour artifacts and differently interpolated detail appears when the RAW values are not balanced, surely Emil can explain why these issues happen:

White balance prior to demosaicing:


White balance after demosaicing:





That's OK, but your workflow is based on demosaiced data; what I would like to achieve is a pure Bayer blend.

Regards

Bridge can run ACR directly w/o PS

I know, but it allows not to white balance the RAW file? and it allows not to convert to an output colour profile?
My ACR doesn't.

Take for instance the one column of pixels in the pole atop the church dome; it is either GRGRGR data or BGBGBG data. Either B or R needs to be interpolated from an adjacent column. If the data is approximately white balanced, the interpolation error is moderate. If the data is not white balanced, the white balance after interpolation multiplies the interpolation error in the chrominance data by a potentially large factor, resulting in large chromatic shifts.

In the particular case of AHD, the interpolation uses color differences G-R and G-B; while the white balance is a multiplicative shift G -> G, B -> c*B. Typical scenes with not a lot of saturation, G-B is smoothly varying and small in amplitude (it is akin to the Cb channel of YCrCb color space). If the data is not white balanced, then AHD is interpolating RAW data with chromaticity G-B/c. If c is large and G~B ie approximately gray, AHD is trying to interpolate data that looks to it as though the chromaticity is very large and rapidly varying across edges. Of course it's all fake but AHD doesn't know that, and so its errors have large (fake) chromatic content along edges, that persists when the white balance is finally done after demosaic. It ultimately traces to the mismatch between chromaticity being additive while white balance is multiplicative. The order of operations matters when considering error propagation.

That's why, again, it's best to do as many corrections as possible while the data is in a linear space, ie prior to demosaic. Though I imagine Jonathan remains unconvinced.

Open Bridge.

Select a RAW.

Open RAW in ACR.

Click the Save button.

Select DNG as the output format.

Check the linear option.

The file is saved as a demosaiced linear RGB, no white balancing. The conversion settings you choose (white balance, etc) are not applied to the data, only stored as metadata tags. The only thing done to the RAW data is that it is scaled 0-65535 regardless of the number of bits/sample from the camera, and each Bayer RAW pixel value has two interpolated values added to it to create an RGB pixel value. You still have total flexibility to white balance as you choose, color processing is identical to the original RAW, you can select whatever output color space ACR supports. If you compare a conversion of the linear DNG to a conversion of the original RAW with the same settings, they will match exactly.

Which is why I don't buy the whole "WB and CA correction must be done before demosaic" argument. If that was true, the conversion of the linear DNG would have color artifacts and other problems not found in the direct RAW conversion. But don't take my word for it; I have two DNG files created from the same original RAW. The first link is to the normal, non-demosaiced DNG. The second link is to the demosaiced linear RGB DNG version of the same RAW. Open them both in ACR, and do a side-by-side comparison converting both with the same settings. The results will be identical, at least if you use the same version of ACR used to convert the linear DNG. (I'm still running CS2.)

http://www.visual-vacations.com/images/200..._0004-Bayer.dng
http://www.visual-vacations.com/images/200...0004-LinRGB.dng

For downloading convenience, it may be better to download only the Bayer DNG and convert your own linear version for comparison purposes. Or use one of your own RAWs and skip the download thing altogether.

Obviously, ACR is using an algorithm far less stupid than what you are describing; the proof is in the DNGs in my previous post. A strategy needed to work around a limitation imposed by one particular stupid demosaic algorithm does not generalize to iron-clad proof that the work-around is necessary or even desirable in all cases for all demosaic algorithms. Look at the reflection of the umbrella in the baby's right eye. There's a small amount of CA visible. It is fixable with the lens correction sliders in ACR set at about -5,+7. The correction is equally effective in both versions of the DNG, even though the linear version has demosaicing done before white balancing. If your argument was valid, the Bayer version would be more correctable than the linear RGB version of the DNG.

Just a clarity on terminology first: By white balance I understand that only the primaries are adjusted and no cross-term multiplication happens yet (as in a color correction matrix). In my previous job when we would make specialized HD video cameras we used to do white balance before deMosaic for several reasons, so I would concur with Emil on this one, and, there were other reasons also for doing it before deMosaic. Additionally, there were some operations that we did relegate after deMosaic, so we did not want to do every operation before deMosaic.

White balance as I'm using the term is just multiplying all linear color channel values by a fixed value (all B values by 1.105 and all R values by 1.027, for example).

Okay, your and mine terminology is the same. I just find that term "white balancing" so confusing, because, people use it for both a diagonal matrix and a full CCM matrix.

Accepting for the moment what you are saying, all it demonstrates is the particular set of choices Adobe has made. If you think that the programming of ACR is optimal for image quality, I've got news for you. Many choices for the programming of ACR seem to be made to fit an overarching philosophy of speed of processing and convenience, rather than IQ. For instance, if WB is held until after demosaic, then moving the CA slider does not require redoing demosaic; definitely a convenience, since demosaic delays screen update by several seconds. The current demosaic engine in ACR has all sorts of chromatic artifacting; your observation might explain why that is so. ACR does a lot of smearing of chroma data to cover it up; that smearing starts to fail in the presence of noise, which one can see in any high ISO conversion of ACR 5.x and before.

Your examples to demonstrate this assertion haven't exactly been overwhelmingly persuasive. As a counterexample to the church tower, I give the baby's hair in my RAW; there's no color artifacting in it anywhere.

BTW, there appears to be a gaping hole in your mathematical argument:



IOW, if B/c is much smaller than G, then color artifacting is inevitable, and this must be avoided by applying WB to the RAW data prior to demosaic to keep B/c as close to G as possible. You're ignoring the possibility that there can be a large variation between the scaling of color channels even when the desired WB is selected. if your assertion is correct, if you do not choose a perfectly neutral white balance (perhaps to enhance the character of the light in a sunset image or retain the colors of the stage lighting at a concert), then high-contrast-edge color artifacts will become a problem in direct proportion to the non-neutrality of the selected WB setting. Setting a deliberately "wrong" WB is no different than deferring the selection of the "right" WB until after demosaic. And shooting subjects with highly saturated colors (such as a blue flower or red laser beam on a neutral black background) can result in edges and fine details where adjacent Bayer pixel values diverge widely in the manner you described in your example, even if the RAW data has a perfectly neutral WB applied.

(G-B/c) is going to fluctuate just as much when B is small and c~1 as when G~B and c is "large". If your demosaic algorithm requires WB to be done first to avoid situations where B/c diverges greatly from G, it's going to have major problems handling situations where B/c is small and G is large even after WB. Any strategy you employ to avoid color artifacting in this instance can also be used to make pre-demosaic WB unnecessary.

do you see that corrected in LR3B (and ACR v6.x when released) ? there will be new DNG converter by then too, may be w/ changes.

I've only played with LR3ß a little; what I've seen so far looks to be a substantial improvement over previous versions of Adobe's conversion engine. High ISO in particular looks a lot better, especially the chroma blotching is gone.

I must stress that adhering to strict theory is not always the goal in industry. The more important issue is that the image looks pleasing to eye even if violates some linearity considerations, etc. Color correction via CCMs is done frequently on non-linear (gamma-corrected) data with coefficients that are derived from linear theory as it offers some advantages in color fidelity near saturated colors. IIRC even some SMPTE specs have "incorrect" coefficients and perhaps primary placements for such reasons and historic considerations.

Jesus, I had no idea my Bridge could do DNG conversions, either with and without demosaic. Good to learn new things. I am quite surprised ACR didn't generate colour artifacts when doing a non white balanced demosaicing since I always thought it was based in the AHD algorithm. However it seems to produce less sharp results:




Anyway Jon I think there is no need to be so agressive in some of your comments to Emil's points. We are all enjoying and sharing information.

Regards.

I suspect the DNG conversion in Bridge is the same as in DNG Converter stand-alone, in which case Eric gave the answer above:




That would explain why it's softer. Now, bilinear interpolation isn't exactly a linear operation, but to the extent that it is WB before or after wouldn't matter since linear operations commute with rescaling operations. Also, one would have to know the internal workings of a converter to know whether WB is being done before demosaic -- who's to say that the WB multiplier isn't applied and then undone after demosaic, so that you are free to WB afterward in ACR/LR? Well I suppose Eric could say...

I was incorrect before in suggesting that doing WB before demosaic would require redoing demosaic every time the WB is changed. There is presumably an optimal value, that one can apply and then un-apply afterward, then the WB slider can act on un-WB data that has had the optimal demosaic, that need not be redone.

Jonathan, there is a difference between the WB multiplier, which is a constant, and the RGB mosaic data, which is varying across the image.

I think that what Eric was saying is just that demosaicing in the DNG converter is not the same as ACR or bridge.

Re the white balance question, it depends on how you're doing white balance. AHD type algorithms are dependent on being able to measure "distance" in color space. If your change to white balance changes distance, then yes, you have to do the demosaic again, at least in theory. Eric has stated in the past that ACR/LR does NOT run a a AHD algorithm. Although what that means isn't too clear - there is a school of thought that ACR/LR runs a AHD derivative, but in a non-RGB space. I don't think there's much evidence one way or the other, but that would explain why ACR/LR is subject to maze type artifacts. However, we do know that in an Adobe workflow, white balance is before demosaicing - that's why a DNG has two color matrixes.

Sandy

I'm not excessively concerned about getting perfect sharpness directly after demosaic as long as there aren't any color artifacts and suchlike; the goal of the deconvolution I'm doing is to correct the sum of all blurring from the lens, AA filter, and RAW converter combined.