I saw some surprizing results of comparisons between the Sony A900 and other cameras, like the A900's DR being a full stop greater than the D3X' and the 5DMkII's.

I set out to measure the noise and DR myself, and I soon ran into a problem: the measurement results were iconsistent, in some cases plainly contradicting each other; for example the noise measured on the raw channels separatedly is different between the channels.

It appeared that I needed better, i.e. for measurement more suitable raw files. Erik Kaffehr, a fellow poster here and a A900 owner volunteered to create the images; after some fine tuning, he delivered excellent bases for measurements.

(Btw, Erik's contribution was much greater than it is usual when I am asking someone and receiving specific raw files. In the span of several weeks he shot again and again, all together about 100 images. Those, who appretiate this report should say or think a big "thanks" to him.)

Then the surprize came: the measurement results were strange even with the best images. I had to dig deeper in the analysis (thank to Rawnalyze, this was not a problem). Some of the findings were astonishing.

I will create a much bigger essay with painfully pixel-peeping documentation of the background (I created hundreds of screen captures for this purpose), but that is a tiresome business, it will take some more time, and anyway many photographers are allergic to proofs, which are too factual; this is particularly true on these forums.

Here are the findings in condensed form:

1. Noise reduction Off, Low, Normal and High are identical from ISO 100 to 800, ON THE RAW DATA. They correspond something like NR Low @ ISO 1600.

In other words: all reviews used pre-NRed samples.

However, NR Off means almost Off with ISO 1600 (this explains the "sudden" increase of noisiness at ISO 1600).

2. The noise reduction affects mainly the red and the blue channels, much less the green. This is a very primitive noise reduction; it simply eliminates some pixel levels in the affected areas.

3. Due to the nature of this noise reduction, the very dark, noise reduced areas become darker. The consequence is, that not only the visual appearance but the noise measurement too indicates not only lower noise that it would be without NR, but that in deeper shadow, suggesting a greater dynamic range.

The magnitude of this shift is about 0.6-0.7 EV between the red/blue and the green channel. As the green channel itself is not virgin either, the shift must be even greater.

I estimate that in end effect the noise gets shifted and the DR "enhanced" by at least one full stop.

A side effect of the above is the non-linearity of the pixel values, and a WB shift in the low ranges: if the WB is correct in the "normal" ranges, it is certainly incorrect in the very dark range.

The nature of this noise reduction explaines the blotchiness as well. This will be appearant from the coming demonstrations.

Anyway, after dozens of hours squeezing these images I still can not put reliable numbers on the noise of the A900. However, I can say, that the noise is considerable higher and the DR is considerable lower than some reviews have suggested.

It's a choice. You can buy the A900 for $3000, get the highest resolution possible in a current 24 megapixel DSLR and use post-processing noise reduction, or, you can spend $8000 for the D3x, have no choice in whether noise reduction is applied because it is in-camera and you can't turn it off. Sony cameras, since their inception have had the highest noise but also in their class, the highest resolution. That goes back to the A100.

Gabor,

Thanks for sharing information! Are you aware of any forum where users are more open to proof?

Erik

I'm afraid you are mixing up some issues. Although the raw data of Nikon cameras is not as raw as the Canon raw, and the difference contributes to a lower apparent noise, that is not comparable to the noise reduction carried out by the A900 on the raw data, and even less to the costomary noise reduction.

I am working on that anyway. I guess there are more posters on DPReview, who are not afraid of pixel peeping and are interested on such details.

May I HIGHLY suggest www.dyxum.com. It's a Sony/Minolta site, but the people there are wildly open-minded and exceedingly friendly. The founder of the site has repeatedly knocked the A900 for this kind of behavior and is a D700 shooter. Seriously, I know they'd appreciate it. (not to mention a LOT of them have extensive sensor and software knowledge).

DpReview is okay.... I guess, but there are a lot of spaz/troll posters.

Gabor,

Thks for the info and extended tests.

By the way, I had forgotten about that, but do you still need d3x raw files?

Cheers,
Bernard

All right, I'll give it a try; let's see how they digest the news :-)

I am not particularly interested on the Sony issues, but I am interested on all raw processing related issues. I am certainly not interested on bashing or pushing any camera or system, as I have no allegiance of any kind. My present work is a critique, not of the camera but of the professinal evaluations.

Bernard, my collection of raw files contains about 4000 files (beside those tens of thousands of my own cameras, of course) - and this is still not enough to answer some questions.

I don't have *any* raw files, which are suitable for reliable measurements of the D3X noise characteristics. I do need black frames (this is the cheepo request), and special shots for measuring the noise.

I need to say at this point, that I do not think there is any big surprize in the published findings re the D3X, like re the A900. However, if I get suitable raw files, I can document my findings, i.e. publish the raw files and the evaluations. I see this as an "open review", others are free not only to criticize but to counter-evaluate the findings.

I agree with you on the importance of suitable camera evaluations and double checks never hurt.

Now, the fact that Sony does raw level noise reduction is not by itself a problem, and DxOMark measuring this is also not a problem. What the problem is is the fact that the impact this raw level noise reduction has on micro detail is de facto not taken into account in DxOMark results. So Sony ends up getting to good mark on DR, but not the blame on poor micro detail (or at least poorer than it could have been).

From a user point of view cleaner shadows with a bit less detail might not be a problem in itself, but they should be aware that this choice is being made for them by Sony.

Cheers,
Bernard

This is very interesting to me. I for one have been flabbergasted by the DR range of the A900. I can't imagine it being "faked" too much. I totally lack the requisite knowledge to be very helpful in this thread, all I can say is that these two shots the convinced me of the astounding DR to be had:



and



Click for larger version.

From the point of this user, i.e. me, the best is to leave the decision to my discretion. I hate patronization, I can't stand when an asshole decides what the best is for me - even if that costs $5000 less tha if two assholes together would make the decision. The sad fact is, that *all* Japanese cameras show the lack of intellect in the design, and attitudes like this of Sony show, how bad this can get.

While I find these shots very good, I think it is important to note, that your (or mine or anyone else's) judgement on the dynamic range of a scenery is of secondary relevance compared to the actual measurement. Normally I would ask you to please post the raw files and then we talk about the facts - but the issue is, that I could not tell the facts in case of the A900!

Anyway, my experience taught me not to rely on such subjective "DR measurement" (I for myself ceased to make such judgements). But if you want to get into a contest: my dog looks much better at ISO 800 under a 40W lamp in the kitchen than your cat under your reading lamp (I mean this is often the level of proof presented on forums).

Good point! I think I understand your original post now. How do you explain the fact that users seem to have similar experiences to me, even coming from the A700 w/ v4 firmware (which I thought had no noise control at all)? An honest question.....


PS I've noticed the A900 data has flummoxed a few different people when the RAW files are analyzed. I believe David Kilpatrick of PhotoclubAlpha was astounded that it produces images as well as it does because, "the red channel is a mess!!!".

Thanks, Gabor, for all your work and very interesting information. I enjoy reading it, though I'm really more into subjective impressions (in prints) than scientific measurements. And on that subject, I've just finished processing (with a lot of winnowing!) about 3000 a900 frames from a recent trip. What struck me most was how few shots, compared to my former equipment of another famous manufacturer, needed any highlight or shadow recovery in C1.

Please keep up the good work!

Bill

Hum... aren't you a bit over-reacting here? How many % of image quality are we talking about?

Many folks around here have been super happy about the A900 lower ISOs, even people having worked with both the A900 and D3x.

Doesn't it show that the compromise chosen by Sony is pretty good, even if not as close to best in class as some like to think?

Cheers,
Bernard

Partonization is a mild form of terror; one can not over-react to it.

As to the % of image quality: do you mean that one should rescind a percentage of the hardware capabilities for .. for... WHAT in exchange? Perhaps it is an intentional degradation in exchange for the low price? Perhaps an A910 will be released for $200 more, the same camera but without forced NR?


They should read carefully what I wrote in the initial post.


Now, this is the point. A compromise between WHAT? A compromise presumes two or more sides of an issue; something for something. What exactly is the advantage of doing a premature noise reduction without my asking, when doing that later, at my discretion is *at least* as good? What other consideration is here beside image quality? Do Sony collect the stolen pixel levels and sell them someone?

The sensor has certain capabilities, which can NOT be enhanced, only degraded the way Sony chose to go. I am not complaining about the sensor not being good enough, but about the firmware intervening (trespassing) in the process of developing the image. What is happening is cheeting with the appearance on the cost of real quality.

There is absolutely no compromise here, only sacrifice, due to some misguided decision, just like with the A700. I wonder if Sony will do this with every new model, waiting until some customer groups are yelling loud enough.

Well, yes, indeed...

My experience is that I see clearly these effects at high ISO and/or specific test targets. I never shoot really high ISO and if I do it's not in really bad light. My experience is that a lot of shadow detail can be retrieved from the KM 7D, the Alpha 700 and Alpha 900 but less so from the Alpha 100. I cannot compare with other cameras.

One of the issues I have that there is a Sony blotchiness which is very hard to eliminate. I have never seen it in other than test images, but I do these tests to find out the limitations and avoid trying to go beyond what the camera can do.

Best regards
Erik

Following captures do NOT demonstrate the unwanted NR at lower ISOs; these are NR Off, Low, Normal and High with ISO 1600. These show, how brutal evel the lowest level of NR is, performed this way, shown on the red channel on Erik's coffee bucket.

Hi,

This test image was intended to be brutal. The issue that Gabor (Panopeeper) illustrates so well is that the image with NR "off" can probably handled quite well by a late version DPP (a very good but quite different raw-converter) or by noise Ninja. The image with the high NR is beyond rescue.

Best regards
Erik

Regarding the blotches: this requires a bit pixel peeping. The attached captures show the blue channel from the coffee cup, one pixel level at a time. It starts with 129, for everything lower becomes null. Note: this is ISO 800, NR Off, i.e. this is the forced NR. These groups of pixels in the same level are characteristic to the forced NR of the A700 and A900, they have nothing to do with the texture of the cup, they appear everywhere the same way.

The last capture shots the histogram on the side of the cup; notwworthy is the difference between the red/blue and the green: the red and blue are much more "shrunk" around 128, thereby "compressing" the levels, making the noise lower and the appearance darker.

The same is happening in the red channel, but the green is much different, even though it does not remain untouched. This is not restricted to the levels so close to 128, the same happens somewhat higher as well (the pixel values of the ninth and higher stops are between 128 and 143).

(Let's try to get back the attachements)

You are over-reacting...



Euh...

- ease of use through more streamlined workflows,
- better DxOMark results that make the A900 puchasers feel good about their investement
- less apparent noise in low ISO shadows that will prevent people from over-exposing, therefore blowing highlights



Everything is relative Gabor. You are only dissatisfied with the shadows of the A900 because you have found that they could have been closer to your taste (meaning noisy but more detailed). All those users who don't know this are just happy about having clean shadows... See above for some ideas on the value of clean shadows.

Don't get me wrong, I am with you here, noise reduction in camera is not something I find to be a good idea, but I am just trying to put things in perspective.

Cheers,
Bernard

without getting too technical, how are you getting the raw data to compare the true DR to the augmented DR for the a900 ?


also, if the a900 makes too many assumptions in their processing algorithms, why not just hack their firmware? Naturally this would be a real pain in the ass but if you connect with a good enough community of developers (using git or something of that nature), you could make something to accomplish zero change in the raw data, no?

I'm surprised that nobody hacks camera firmware so far that I've seen... It happens with so many other devices, why not cameras?

I'm not a firmware hacker for any device, but I have a sufficiently deep knowledge of CS to begin to pick it up.

EDIT: a google came with this;
http://chdk.wikia.com/wiki/CHDK - Canon hacking wiki
http://www.camerahacker.com/ - enthusiast site

I am not, for I can not. However, if Erik manages to make shots with his A900 and with a Canon at the same time, then I can relate the intensities.


Well, why not? However, the effort is lost when the next firmware version is released and you need it.

There is a firmware hacker/enhancer community, but they are dealing only with certain processors and with certain cameras. Camera makers are not keen to publish any internals.

Bernard, the issue is not so small. At first sight it might appear that this affects only the very-very dark regions, but this is not so: even a medium dark area can have one or two very low component. For example one of the greenish (not pure green) patch on the color checker sheet Erik has been photographing contains greens in the sixth stop and blues in the ninth (or it appears in the ninths). The blues are blotchy; this may make the result blotchy.
Particularly, if one channel gets pushed up due to WB, it can affect much brighter areas as well. Why do you think Sony stopped the unwanted NR with the A700?

In some cases these pixel groups (the blotches) are like perfect mosaics; the pixels of two levels occupy over 90% of the area. Sometimes the groups are not in adjacent pixel levels; this makes their blotchiness particularly strong, when carrying over in the RGB value.

so then you are speculating that a 'true' raw file would have better DR than a NR off file, based on the knowledge of the differences between NR on and NR off and extrapolating because any onboard raw pre-processing must make it- the same or- worse than the original data, yes?

am i missing a crucial step?

1. The dynamic range of the camera is determined by the sensor; that can not be changed.

2. Presently the noise reduction (even with NR Off) causes dark areas appear even darker; this makes the impression, that the DR is greater than it is in fact. This is reinforced by measurements, which are based on the intensity after this noise reduction.

I too am measuring the noise in conjunction with the average pixel intensity on a selected patch; if that pixel intensity is artificially lowered, then the measured standard deviations get assigned to deeper shadows then it should be, i.e. the DR appears greater.

check. thanks for the methodology clarification . I'll be interested to read your report, once compiled

It seems that Sony is guilty of what we called "tuning" their hardware to get a specific reaction from uncritical test sites. For example, Card manufacturers use to send their video cards to specific sites, knowing what software/hardware they used to analyze their cards. They would "tune" the card to perform extremely well on the tests. That latest about one quarter because the test sites caught on really fast to what was happening. That's why they always test synthetic and then back that up with ever revolving real time FPS analysis running each new iteration of game's built in FPS (frames per second) Demo. This is what Gabor is doing, backing up the "test" files by doing his own analysis at a level that cannot be faked.

And even though some people might think "So what, it looks really good, it looks like really high DR, and it looks like really low noise, so who cares?" And the answer is that if you need to manipulate an image that is noise reduced to a point of being unmanipulated, you lose the ability to further manipulate the image to your specific needs. And yeah Sony could put in a switch easily that allowed one to turn off noise reduction completely, but then everyone would be saying, Not that great noise control and DR but pretty good." And then all the people who ran out and bought the A900 might have bought the 5DMKII instead. That's a marketing decision, one that limits our control over our photography for sure.

Hi,

Hacking is not easy, you know...

Best regards
Erik

Hi,

There is certainly some tuning in order to get good reviews. Almost all reviews are done on JPEGs, however, so this issue with NR on raws is not necessarily related to tuning for reviews. The blotchiness reported by Panopeeper does also effect high ISO JPEGS, however.

DPReview had never given good marks for noise to any Sony or Minolta camera, by the way, so would Sony have a strategi to do excessive noise reduction it certainly does not work with DPReview, nor does it work with DxO-mark.

Another way to see it is that without the A900 there would not be any 5DII. Canon would have no reason to release the 5DII unless there was competition. Would Nikon has released the D3x without the A900 and the 5DII being around no one would complain about the price.

Aside from that, the Sony A900 is essentially doing what it is supposed to do, nameley taking pictures and it is quite good at it. It is not really a high ISO camera nor intended for sports. As a landscape camera it works very well.

The noise reduction we are discussing is not a software solution to my best understanding but something done on chip,not necessarily in softfware but probable having to do with preamps before the ADC. That may be related that it cannot be shut off completely. Sony has not been secretive about this in any way, it has been published in their description of the sensor chip.

If you check DxO-mark the 1DsIII and the 5DII are in the same league as the A900 (within some decimals). The Nikon D3x is significantly better. Panopeeper is somewhat sceptical about the claims for the D3x, but we cannot now for shure unless DxO makes their "raw" images public or someone having a D3x makes the images Panopeeper needs for his evaluations.

Best regards
Erik

Hi!

I don't really feeleasy about this. The problem is that I can probably borrow a Canon, but I probably need to change a lot of settings, which I don't know how to do, worse is that the owner probably wants it back in the same shape as it was before. But I may be able to engage a Canon owner in this project.

Best regards
Erik

I am duplicating this post from dyxum.

"Interesting thread. I've got a couple of comments/questions.

[Kiklop,] am I correct in remembering that Sony has been using non-linear RAW data as far back as the A100?

Secondly, I take DxO Mark, like all reviews, with a grain of salt. That being said, it seems strange to me that in light of their brand new A700 review, which goes into some detail about NR on the green channel prior to firmware V4, DxO mark wouldn't notice NR on the blue/red channel in the A700/A900 if it existed. Panopeeper, I have a VERY limited knowledge on your raw analyzer and how it works, so I'm wondering why there is this discrepancy?? Thanks.


Dxo Mark A700 NR article

Hi,

What I gather is that Sony does a different kind of noise reduction than DxO is checking for. DxO is checking for spatial noise reduction, that is some kind of mixing the signal from a center pixel with surrounding pixels, the kind of noise reduction Panopeeper has detected is more on the signal itself.

"2. The noise reduction affects mainly the red and the blue channels, much less the green. This is a very primitive noise reduction; it simply eliminates some pixel levels in the affected areas.

3. Due to the nature of this noise reduction, the very dark, noise reduced areas become darker. The consequence is, that not only the visual appearance but the noise measurement too indicates not only lower noise that it would be without NR, but that in deeper shadow, suggesting a greater dynamic range."

From some of the writings I have seen from Sony may indicate that this design was chosen so the chip would cooperate well with the DRO (Dynamic Rnage Optimizer) that brightens up dark parts.

Best regards
Erik

The on-chip noise reduction is a different class (it should be called "noise avoidance"). However, that is single pixel oriented and never becomes apparent on the image.

The technology is described here (on the "public level"): http://usa.canon.com/uploadedimages/FCK/Im...non_CMOS_WP.pdf

Could the second pass of sensor NR illustrated in this graph be the possible culprit? Much has been made by Sony that they are adding on-chip NR to both the analog and post ADC digital signal, but this is all done on-chip, due to the EXMOR ADC design being on-chip. I'm curious if the D3x differs in 12bit mode.

I have a much smaller "raw base" (raw images files) with the A700 than with the A900; particularly, the Imaging Resource images are not underexposed. Anyway, I have not seen any sign of THAT kind of noise reduction, not even in the darkest regions, what I see in the A900 images. This is immediately visible already on the histograms; the A900 fine histograms proudly display the signs of manipulation (the first one is A700 ISO 100 firmware version 3, the second is A900, ISO 100):

This is really interesting, but too superficial (not as if I would understand it if it dealt with hardware measures). The question is if it could cause the blobs, the tight groups of pixels with the same value.

Thzere is one consideration against that, and for software (firmware) manipulation: the effect of NR with ISO 1600, where it is selectable, is the same as what is happening in ISO 100-800, but the degree depends on the level of NR. I posted captures with the "involuntary" blobs above. The followings are from ISO 1600, NR High; the blobs have the clearest appearance here; with lower level NR it is becoming like what I posted above. The histograms of the darkest patch on the color checker too show the effect of increasing noise reduction: the levels get "pushed together" (the file names in the top white field in the capture indicate the level of NR).

If this is anything like typical LCD displays, a darker black level isn't necessarily a bad thing. I suspect Sony has purposely tweaked the A900 towards landscape and lower ISO uses. After all, this is what most of the people who would buy the camera would use it for. For $2500, it's a very inexpensive 2nd place to the Nikon. I'd be happy to own one.

It's obviously using a different method of NR than you usually find. I've seen the raw files from the A900, and honestly, for the price, it's not a bad way of doing NR, IMO. Whatever they are doing seems to be good enough.

I have found the *perceived* noise characteristics of *my* D3x to be really strange: At 1600 it's unbelievably good, more than enough for fashion/portrait use. After 1600 it's like going off a cliff.

Edmund

There is no healthy patient, only incompetent diagnosis.

you're being elliptically cryptic

Edmund

Just to clarify, you are saying that even at low ISOs, such as 100-400 ISO, you are saying there is some kind of NR being applied in-camera?

That's called crypto-logiC.

(The thing with the diagnosis is an old joke among doctors.)

Pls read the opening message of this thread; it is about nothing else.

I'm curious whether it's being applied on-chip in the second NR portion, or in the BIONZ??

On chip is possible because it is a CMOS !

Thierry

I agree with you Bernard, you put in perspective the well known fact that many people will be happy without knowing that his wife or her husband is seeing another person, or will pick the blue pill and enjoy eating the fake beefsteak in Matrix the movie. But some other people prefer to know what's really under the hood, that's why I think Gabor's analysis are very valuable.

I also made a list of advantages of all these tests and findings:
- The user has the power to _decide_. To decide if switching on/off camera's NR. To decide if purchasing or not a camera where NR cannot be switched off.
- Camera vendors can find out not everyone out there are just followers willing to buy any new camera on the market that _seems_ to perform better than others, so maybe they finally concentrate on making cameras that _do perform_ better than others.
- Sites like DxO Mark, whose RAW analysis are the best found on any public site, can admit their quality measuring criteria is weak against camera vendor RAW tricks, and mend this.
...

The on-chip noise reduction is rather noise avoidance. It cares for single pixels, it does not create blobs.

Moreover, the noise reduction we can see here is selective: in ISO 1600 (and above, I guess) it can be turned on to one of three levels or off. The signs of that actuion are identical to those observable in ine lower ISO shots, but not selectable.

and there are plenty of doctors who do not listen.