I have just uploaded a new version of Histogrammar, a free software to plot detailed histograms from 16-bit images in their native 0..65535 range. Ideal to study histograms in deep, compare RAW developers, processings,...

It adds an option to plot the histogram in logaritmic scale that I have never seen in any other software and I find very interesting to check dynamic range of scenes and sensor (the image has to be provided in linear format for this, as the ones developed using DCRAW).

Hope you find it useful.

Regards.

Histogrammar V1.0 click on "DESCARGAR HISTOGRAMMAR V1.0"

Thanks for listening!
Marc

Did I?

Guillermo,

The histogram program looks very nice, but when I click on the link I just get some document ion in Spanish but not the actual executable program. Am I doing something wrong?

Bill

I found the link on the upper right and downloaded the program. Comments to follow.

Bill

Oh yeah sorry, "DESCARGAR HISTOGRAMMAR V1.0" is the download link into that page.
I have just uploaded a new version with some minor errors corrected. Try the new one if you find problems, it should look like this:




Thanks for testing.

Guillermo,

The high resolution histogram is impressive, but with 16 bit files derived from a 12 bit raw image, there are gaps in the data. Only 1 of 16 values have data and the rest are blank, since the 16 bit image is derived from the 12 bit by multiplying the latter by 4. Raw values 1,2,3,4 .. 4095 = 16, 32, 48, 64, .. 65536. It might make sense to allow the user to adjust the bin width used to construct the histogram frequencies. With a bin width of 16, you would capture all the values, but a larger width could be used so tht the histogram would fit on the width of the screen.

Bill

mmmm you are a bit wrong, let me explain: RAW files are 12 bit as you say, so displayed on a 16-bit range (0..65535) should have 15-hole gaps between every two tones or levels.
But we are making a double mistake:
1. White balance completely disorganises this 1+15 distribution by scaling all levels by a non integer multiplier.
2. Second, and much more important: after developing, interpolated values are calculated in the true 16-bit range, i.e. 0..65535, so they completely fill all empty gaps we had before the development stage.

Find here some samples of linear histograms from TIFF files generated using DCRAW.
- Peaks correspond to non-interpolated levels (i.e. levels captured by the sensor). R, G and B peaks perfectly match due to absence of WB in this case (I wanted to compare pure camera histograms).
- Between peaks (and also in the peaks themselves) spread all the rest of interpolated levels, completely filling the gaps.

It's not until the gamma compensation when again holes will be generated in the lowest part of the histogram, making it look like a gruyere piece of cheese.

Histograms correspond to the same scene as viewed from:
- Canon 5D: completely linear 12-bit
- Leica M8: non-linear 8 bit
- DMR back: already 16-bit in origin so no peaks will be visible:


5D





________________________________________________________
M8





________________________________________________________
DMR






All those 3 RAW files were developed with no WB applied (-r 1 1 1 1 option in DCRAW) and hence that awful green colour.
In a normally developed RAW file, the WB scales differently each channel and this happens:




Non interpolated peaks now mismatch and again gaps between every two captured levels are filled with interpolated new levels.
That is why I always state that ETTR doesn't provide more tonal richness, as all levels are filled in any case, but provides more tonal precission as peaks corresponding to captured levels get closer (after the exposure correction down), and thus interpolated levels are calculated more accurately.

And this is all thanks to the 12-bit to 16-bit conversion, and I have heard no one saying this!!! In that way, ETTR in a camera with 16-bit native RAWs like the DMR back, will improve SNR as usual, but willl not improve at all the tonal precission of the capture as all overexposed levels will agregate after the exposure correction down to the same levels they would get in a non exposed to the right shot from the begining.


It is always best to start from the whole 0..65535 range, and zoom out with the X-axis zoom option provided. Pressing that Zoom out 4 times will do what you are looking for (0..65535 -> 0..4095). Every press scales down the histogram by 2, which equals to substracting one bit of the image file resolution.

Thanks a lot for the programme, it's a nice tool to play with and it helps to understand what's happenning with the image when you apply different conversions and modifications.



A real gruyère cheese does not have holes in it! It's swiss and in Switzerland nobody will pay for air ;-)
It's emmenthal cheese that has lots of holes.

Allan

Correct, in fact ideal histograms would look like Gruyère cheese!

ehhhhhh what about this?
Gruyere cheese
I think there are both, with and without holes. But you are the experts

This is Emmental cheese, with large holes (see real Gruyère, without holes)!
Gruyère village is just 60km from my office...

Ok, I believe you.

Just another site for real Gruyère

Cheese in Switzerland is a BIG subject

Allan

Guillermo,

Thanks for the detailed reply. Floating point white multipliers do spread out the histogram somewhat. They shift the red and blue values to the right by a fixed amount (for daylight white balance), but they don't really disperse the values. Here is an example from my Nikon D200 with raw conversion done with Iris rather than DCRaw.

Here is a raw file converted into 16 bit with no white balance or gamma adjustment. There are gaps in the data and the RGB channels are aligned as you predicted:

Click to view attachment

Here is the file with white balance. There are still gaps since the red and blue data are merely shifted to the right by a fixed amount and are no longer superimposed:

Click to view attachment

And here is the file with white balance and gamma adjustment (gamma = 2.2):

Click to view attachment

There are still plenty of gaps in the data. I don't really see the filling in that you described.

By the way, I did note that Adobe Camera Raw seems to do things a bit differently.
In the histogram, the channels are superimposed and there is no spreading of the data. Any thoughts on this?

Click to view attachment

The above histogram was from the shadow area which undergoes little compression with the gamma correction. Towards the right of the histogram, the gamma causes the values to be closer together. This does not occur with linear data. I think that is why the Swiss cheese appearance appears in the shadows but not the highlights.

Click to view attachment



I don't see the degree of filling in that you describe. However, it is really a moot point since the eye can distinguish 100 levels per f/stop at most, and the 2048 levels in the brightest f/stop of a 12 bit raw image can not be seen by the human eye, which in effect blends many of them together.

Yes, I do see that the zoom control effectively does what I was asking for. All in all, a very nice tool. Now you need to make a similar tool for HDR 32 bit floating point images

Bill

I think you don't see the degree of filling because, as a Spanish motto says, your RAW developer is "more weird than a green dog"

Please try to develop the same RAW file on DCRAW with something like:
dcraw -v -w -o 0 -q 3 -T -4 pic.nef

You will see all those levels completely filled with interpolated values. In fact, it is a bit stupid (no idea why Iris does) not to make use of the entire 16-bit range for 2 reasons:
1. 16 bits means more precision than 12 bits: if the demosaicing algorithm calculates a value that falls between two 12-bit range levels, why should we round it to one of them losing our additional precission?
2. Even if human eye cannont distinguish so many levels, most pictures will get into PS for heavy edition after developing. The more differentiated levels you have the more robust they will behave in curves, levels, and so forth adjustments to avoid solarization and other artifacts.

Another possibility for such a strange histogram is that the images you are analysing are not linear but already have a gamma compensation curve applied. Are you sure about this point? a colour profile conversion for instance implies a gamma compensation without the user having to introduce any parameter value, and a short window view will show levels equally spaced as the ones you show.

Anyway, Histogrammar well served in this case to find out how Iris soft works. It looks interesting, I will have a look at that softwarre tomorrow.

Regards.


PS: there is a new version of Histogrammar. Called the same V1.0 but with some corrections and additional information applied.

I forgot about your question about ACR version (BTW did you notice the very different % level filling between your Iris and ACR developments?), and I have developed a RAW file and got the same results as you: vertical lines with R, G and B perfectly matched:




But it has to be that way if my theory is correct: gamma compensation has expanded the shadows of the histogram, creating holes where there used to be a continous raw of RGB values. Now they look like aligned as the gamma compensation affected equally the 3 channels. It's my interpretation.

The only thing that disturbs me a bit is the absence of peaks as those found on DCRAW's -q 3 development. It seems ACR's algorithm is quite differente from that of DCRAW, as the same RAW file on DCRAW (just R channel presented for simplicity) shows:

DCRAW linear:




DCRAW gamma corrected (trough sRGB colour profile conversion):




which still shows the peaks found on the linear histogram.

Guillermo- thanks for this!

A request: would it possible to make portable versions of all your apps? No install, no (or optional) registry entries, all configurations saved as an *.ini in the program folder, instructions for manual install of any libraries etc. ?

Andre

IRIS is not a RAW converter. It is a tool for astrophotography that can load RAW data literally, and perform mathematical operations on it, including the ability to interpolate full RGB from CFA images. By default it will not get any in-between values at all for interpolation or WB, as it works on integers, and loads RAW data without gaps. You have to multiply the data by some value first to get in-between values. There are 3 CFA interpolation methods, one creates one intermediate value, one creates 3, and one fills the histogram. All three methods have spikes where the original data values were.

IRIS will do linear "HDR", but you need to export the result to take advantage of better demoasaicing algorithms. IRIS RAW decoding is based on DCRAW, but the auther has not used the AHD demosaicing from DCRAW.

I did develop the raw file with DCRaw as you suggested and the levels were completely filled in as you predicted.

I do not think that the raw converters under discussion get 16 bit precision from a 12 bit file--they are merely filling in blank spaces with interpolated data. Why else would anyone go to the trouble of developing a camera with 16 bits? It does make sense for the converter to use the 16 bit space for intermediate results, but we are not getting true 16 bit precision.

The highlight areas of a 12 bit file contain wasted bits, since such heavy duty editing that would lead to posterization is not done in real world photography. For example, Nikon cameras have a compressed raw format which reduces redundant highlight levels. The resulting file has 683 levels rather than the 4096 values in the original file, and Nikon calls it visually losless. Some people claim that they see a difference in the highlights but have produced no convincing demonstration confirming their claims.



Apparently, Iris does not do the type of interpolation that fills in the blank spaces as John Sheehy pointed out. The files were not gamma corrected as evidenced by their dark appearance in the Histogrammar preview. So far as I know, Iris does not support color profiles.



I did a conversion with Nikon Capture NX and noted the same behavior as with ACR. I also did a conversion with RawMagick, which is a well regarded converter that uses floating point math rather than integers.

Here is the shadow portion of the histogram:
Click to view attachment

and towards the midportion:
Click to view attachment

The missing spaces on the left are present, but irregularly spaced. It could be that they were filled in in the linear file, but the holes appeared during the gamma correction as you predict.

With Histogrammar we see all types of things that were not apparent in low resolution histograms.

Bill

Does the Nikon software convert uncompressed NEFs to compressed? If so, you can take an uncompressed shot with smooth gradients in the highlights, and make a copy compressed, and zoom into them levels-wise (with blackpoint and whitepoint) to see how extreme you need to go to see a difference.

Leica M8 RAWs actually use less levels. They use 256 levels (8 bit) with a LUT, using a curve that preserves levels from the shadows better than from the highlights.

I think I have just found the reason for the discrepancy (peaks in DCRAW, no peaks rest of RAW developers). The key is colour profile conversion: -o 0 option in DCRAW develops with no colour profile conversion (as far as I know: just black offset substracting, WB, 0..4095 range adjust, and Bayer demosaicing). But commercial developers additionally always convert to an output colour profile (sRGB, AdobeRGB...).

We previously saw the peaks in DCRAW's developing with no colour conversion.
This is what we get in DCRAW linear with sRGB colour profile conversion (-o 1):


No peaks (and Zoom=1), the colour profile conversion reallocates all levels in a very uniform way, making the original captured and interpolated values absolutely indistinguishable (this word exists?). By applying gamma over this last histogram we would get a similar result as in ACR and Capture NX: locally-equally spaced matching RGB levels of similar amplitude.

BTW I wanted to ask you something that disturbs me a bit: I almost have no idea about colour profiles, but as far as I know each colour profile (sRGB, AdobeRGB...) has its own standard gamma curve. On the other hand DCRAW allows to develop a RAW file and convert it to a given colour profile, but IN LINEAR MODE, i.e. not applying the gamma correction yet.

Is this conceptually correct? can the gamma curve be applied after a linear colour profile conversion as a separate stage of the conversion process?

David Coffin (author of DCRAW) told me he puts in the output TIFF file converted to a colour profile the needed information so that for instance PS recognizes it as a linear (still gamma=1.0) image. And it seems so: when you load this tiff files into PS (indicating it is gamma=1.0 in the Edit->Colour adjust->RGB custom menu), and then you convert to a colour profile (which can even be the same that we previously set in DCRAW), PS clearly expands the histogram by a gamma curve.
What I wonder is if all this process is correct from the perspective of getting a correctly developed image or we are just getting "numbers" that in some way look like what our photograph should have been.

Regards.

I have just uploaded Histogrammar v1.1 ready for download ("DESCARGAR HISTOGRAMMAR V1.1").
It includes several new features, the most remarkable of which is the possibility to analyse the Zone System of the scene, plotting each f-stop area in 16 greyscale tones (blown areas are plotted red, and black areas in blue).
To obtain a correct result linear RAW developing must be applied (e.g. using DCRAW).

For this escene:




This is the real f-stop distribution (every grey colour means double/half amount of light as the previous/next one):





There is also a Histogrammar tutorial (Spanish).

Hi Guillermo,
Thanks for your efforts and for sharing this program with us. It looks like it could be really useful.
Regards,
Eric

I have added a possibility to plot the classical Ansel Adams Zone System, consisting in only the top 9 f-stops represented in the expanded 16 f-stops Zone System I previously showed.

Both ends are pure white and black, and are supposed to contain no detail at all (this is however not true as with high dynamic range techniques as multiexposition we can achieve easily more than 9 f-stops full of information).