Mike Johnston at The Online Photographer has an interesting article on equipment (lens) testing, here:

http://theonlinephotographer.typepad.com/t...s-and-grap.html

This is the typical rumbling of the digitally challenged, who is offended by something he does not understand. The proof is in the pudding; my cat looks much better than your dog, so my lens is better than yours.

Example: does anyone problem with adding vignetting to an image? I dont. However, I do have lots of problems with vignetting, which I don't want to have, but is there; thus the vignetting characteristics of a lens are very important for me - not based on the picture of a nice child. Btw, the correct term is optical vignetting, beside light fall-off.

Maybe reading reviews is as subjective as looking at photographs I took his column to mean that specifications is one thing but at the end of the day the art of photography is about the final result. Sure, if some highly technical photography is required then charts may be useful in making your final decision on which lens to choose; but to look at it from the other direction and say a photograph is poor because of the way the lens measures is crass.

I have a (dwindling) interest in hifi and the same arguments happen there (but much worse). Some show preference for gear based on on technical measurements and others purely on the sound. And at the end of the day if it sound right who cares how it measures?

If I was hung up on MTF charts, edge sharpness, vignetting, soft wide open, noise at high ISO's, etc., etc. I would have given up on photography a long time ago. But all this stuff is for 'pixel peepers' and equipment perfectionists. Nothing wrong there. More power to them. I like my stuff aside from all the technical, microscopic examinations, and if you don't, that's okay too.

Vignetting (as used in the example photo of the child) seems to be most often used to help improve poor composition. In most cases I don't mind it, but I do want the option of being able to capture an image without vignetting (or falloff) if I choose to do so.

It's a lot like sharpness... I realize it's just a crutch for the artistically challenged, but *I* want to be the one to decide when a photo is to be sharp or soft, not the equipment manufacturer.

I couldn't give a dam about Lab tests, it works or it doesn't for me. No doubt the pixel peepers could argue I might as well shoot with a nokia as I am always shooting from aircraft and the vibration kills any resolution advantage of more pixels or better lens. Then again I know my smkIII gives me better files than my smkII or Kodak SLR/n and better also than when I subcontract work which comes back from various other Nikon or Canons. In over 30 years of pro shooting I've never felt the need to shoot brick walls or newspapers, it's the balance of day to day real shooting in different conditions that counts , not the hypothetical max you might get with everything nailed down in your favour.

Kevin.

Same here. Get the shot. Not many have the choice of more expensive, or the right gear, etc. If something interesting is happening now, do your best with what's at hand and get the shot. I have many interesting shots that were not done with the best tool, ideal conditions, etc. However the image itself is interesting and sells the message well.

Even from a technical standpoint, the world is not a pristine lab with balanced lights at 45 degree angles and objects facing at perfect parrallel planes. Screw it. Not every lens needs a red ring to be capable of making good shots. Putting on my engineering hat, a resultant test of a chart only proves that a given lens was better or worse for a particular setup, at a given temperature, humidy, etc. etc. etc. So, yes, the charts do prove some basis to considering purchasing. But in real life, lens' can vary from one to another, and unless you have that very lens used in the test, and know it wasn't knocked around or dropped, you'll still have to do the only thing real to know for sure. Buy it and use it.


Life rarely happens twice, just shoot.

Some of us might be interested in why it doesn't work when the lab tests indicate it should work.

I got the impression that Mike Johston was railing against ill-defined lab tests, subjective quality factors masquerading as lab tests, and MTF charts based on theoretical designs rather than real lenses.

Real and thoroughly executed lab tests and MTF charts, if one understands them, should help one to make informed decisions, not only when buying a lens but when selecting a particular aperture for a particular effect during shooting.

What on earth do you think the misquotation, "The proof is in the pudding" could mean?

Jeremy

I felt like that about hi-fi in the eighties, but you can only struggle against the compact disc for so long!

That's why Michael's reviews on this site are so useful and informative. They don't bother with laboratory testing: he just tells us how he found the piece of kit to handle and to behave in the field, and what his results look like. That's far more useful, if the author is someone you feel you can trust.

Jeremy

I agree with Ray that well done lab tests can contribute significantly to our understanding of the performance of cameras and lenses. Evaluation of photographs is also important, but can be very subjective and depends to some extent on the scene--whether it requires MTF at high or low frequencies.

Unfortunately, I have seen neither photos or test data in this thread and the discussion thus far is not helpful.

Bill

Truth be told, testing more than one lens, say 100, and finding out that 90% of them all had a similar response curve would be more factual. Granted, not everyone has time or money to test 100 lens, or even 10. The manufaturer may run such numbers, and if so, then tests for your particular unit should be a close match to be considered accurate. Again, it's a baseline on a set of standards.

Once you've deviated from that standard, and from the standpoint of scientific testing, it's pointless.

In argument for testing. It's good to test your lens if you duplicate the setup and compare the results, but if a given lens has poor results compared to the baseline, then it can be considered a bad lens. In a visual test, you would have to directly compare a known good lens to actually see the results. One persons sharpness is anothers blur.

You have to have charts to know what your baseline is. Use the charts but only for what they were intended. Then use the lens to see if it's sharp enough for your needs. Most folks find it easier to do the later.

A follow up and a 'test it yourself' here, for those who are interested:

http://theonlinephotographer.typepad.com/t...r-from-rio.html

That's true. One needs some reference point. The charts provide a comparative reference, such as 'lens A' is as good as, or better than 'lens B'. If one already owns 'lens A' and one intends to buy 'lens B', one can carry out some testing on 'lens B' before buying, using 'lens A' as the standard.

However, there is a flaw in the process. How does one know that one's own copy of 'lens A' is typical? It might be an exceptionally fine copy, or it might be substandard.

If it's the former and the copy of 'lens B' that happens to be available for testing is typical, then one would probably reject it. If it's the latter and the copy of 'lens B' were equally substandard, one would probably accept it if the MTF charts had already created the expectation that both lenses are about equal in performance.

One would then be in the position of owning two substandard lenses.

That's the point, lab test mean nothing in the field, you have to shoot over a period of time in the conditions to see if the equipment does what you need. You will soon see if a lens doesn't perform as you need or a camera handles well enough etc A lab test just shows it can take pictures of a chart bracket focused on a tripod etc no real indicator how it produces the goods at a sporting event, wedding, fashion shoot or the Zoo. The net has produced a plethora of mathematicians that decide wether a lens or camera works or not by producing charts and statistics and seldom pictures of anything of note. It's like deciding on a car by the 0-60 mph time. At best a chart will show that a lens camera combination isn't a complete dogs dinner.

Kevin.

I generally ignore all the MTF charts, as they're often too subjective, and often do not correlate to actual physical lens testing. I often read reviews, then rent or borrow the model I'm interested in, and in this way I find out for myself. As to many of the photo magazines, well, it seems they have never tested a lens they didn't think was "great"...I guess that policy is great for generating advertising dollars.

As to pixel peeping:

Think about this: One can use a lens for years, producing thousands of "good" pictures with it, then come to find one day that pixel peeping will show the type of limitations that the glass has, that should've been obvious just looking at the pictures produced.

In other words, one should pixel peep as part of the acceptence testing of a new lens, because often "realworld" photographs will not shake out the optical issues.

Nothing wrong with pixel peeping and why there are those that poo-poo it is way, way beyond me.

I'm a pixel peeper and I'm glad to admit it!

But then, ALL my L glass copies are exceptional and had I not pixel peeped upon receiving them, I could not be able to write this.

To spend $20,000 on L glass and not pixel peep would be the epitome of stupidity. That warranty is just one year long for free calibrations and other required mitigations for faulty glass.

I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do.

However, problems might arise if the tester has not been diligent and become aware from other sources that a particular lens is producing anomalous results and is not typical of the quality expected from that particular model. In such circumstances, the tester should obtain another copy of the lens from another manufacturing batch.



In the absence of MTF charts, this is all one can do. Test the lens for oneself at various apertures. I'm a firm believer in the concept, "Know thy lenses". But it can be a tedious process. There are times when I think I'd like to buy a particular lens, then I think of all the testing that I'll feel obliged to do to ensure I have a good copy, and sometimes I decide it's not worth the trouble and I don't buy the lens.

I'm a modern person of the technological era. I would prefer any lens I buy to ship with a full set of MTF charts that describe its performance, that is, MTF charts that are specific to that individual copy of the lens.

I hope one day China will find an economical method of doing this, but I fear that the additional cost is not the issue. Manufacturers seem to rely upon a certain amount of B/S and hype to sell their products. Simultaneously providing an accurate performance description of their products could be seen as contradictory.

Kevin,
They might mean nothing to you, but they are meaningful to me. Below are a couple of charts from Photozone comparing the performance of the Canon EF-S 17-55/F2.8 zoom with the highly regarded Canon 50/F1.4 prime.

Click to view attachment Click to view attachment

These charts are only an indirect indicator of lens performance. They are more correctly an indicator of 'system' resolution. The camera used was the 8mp Canon 350D, so the results should be relevant for owners of cameras with a similar pixel density, such as the 20D, 40D and 1Ds3 (in the case of the 50/1.4), although there's no information here about the edge performance of the 50/1.4 on the 1Ds3.

The above charts only describe the performance of the lenses at 50% MTF. The choice of the 50% figure appears to be based upon subjective factors in relation to real world images. Lenses which deliver a high resolution at 50% contrast tend to produce real world images that look sharp and detailed. Performance at 30% MTF (for example) would be less relevant to an appearance of sharpness because such low contrast signals tend to get buried in noise, unless one is shooting high contrast line charts, or unless one's real world images contain image components similar to lines charts, such as street signs, advertising billboards, tree branches against a light background, shop signs etc.

Examining these bar charts, one should notice something quite remarkable. The EF-S zoom appears to be as sharp as the 50mm prime, across the apertures they both have in common, ie. from F2.8 to F8. My decision to buy this zoom was sparked by the sight of this chart comparison.

Unfortunately, the circumstances in the small store in a busy shopping centre in Bangkok (where I bought the lens) were not conducive to my testing the lens thoroughly before buying. But later testing has confirmed that this lens is indeed as sharp as my copy of the 50/1.4 prime, and just like the charts indicate, is not quite as sharp at the edges of the frame as the 50mm prime which is, of course, designed for a larger format.

However, one factor which the charts do not (and cannot) address is autofocussing accuracy. I'm disappointed with the autofoussing capability of this lens, even after calibration by Canon back home in Australia.

There you are then, the MTF showed you what a lab test would do, in the real world it does not stack up because the AF is sub par, neither will it show the sloppy MF, or how the elements move when it gets hot or cold, all things that stop you matching the Lab test results. Also the MTF does not tell you of the weird things that happen with point source lights at night with the 50mm f1:4, its like shooting through wet glass. Neither do the MTF charts show the performance of the lens you bought, only the lens tested, if it was a Leica lens you could expect closer tolerances of the construction and materials, neither does a lab test show a lens performance on 3D objects or the bokeh.
The 50 mm is sharp from f 3.5 onwards at f1:4 it's a special effect portrait lens, I don't think Canon planned it that way. I could right a book on the 17 - 40 mm lens, there has never been a chart made that could explain all its variable idiosyncrasies, you just learn them with use.

Cheers,

Kevin.

Well, you could continue to add to that list of things an MTF chart does not tell you, almost ad infinitum. It doesn't tell you what the weather will be like tomorrow, either. Obviously, the charts are only useful for what they do tell you, not for what they do not and cannot tell you.

If resolution behaviour from the centre to the corners of the image at various apertures is important to you and is therefore a major factor to consider when choosing a lens, then a collection of MTF charts might be a good starting point to narrow the selection.

Unfortunately, there are far too few MTF tests being carried out nowadays, and QC variation can be a problem, which is why I'd prefer to see a complete set of charts relevant to each lens before i buy a lens.

Well, even Canon admits that their MTF charts are NOT based on actual scientific testing. Their charts are "guesses", "theoretical"....this is true!

So you really can't assume an MTF has much integrity.

In addition, the many "serious" online review sites OFTEN show results different then what I found with my lenses.

One more thing: Even if an MTF is true, it is a a picture of the characteristics of ONE LENS, and may not necessarily be applicable to all copies of that lens. I have found this to be true often.

At the end of the day, MTF charts are anecdotal at best. Reviews are too, but more accurate, and allow one to "harmonize" results across dozens and sometimes hundreds of findings from actual owners.

I think Mike probably knows as much about lens testing as anyone on the net. I believe he has chosen to do field testing because it works better. The problem with MTF charts is that a) they're either projections of lens performance in an ideal state (the theoretical performance), or B) a chart derived from a particular lens. The problem with the latter is seen in all the complaints from people who have sent lenses back to the manufacturer for sample faults -- so the MTF chart derived from one lens through actual measurement is not necessarily (or even usually) what you'll see from another sample of the same lens.

Mike Johnston's reviews tend to be something like Mike Reichmann's -- practical tests of what he believes to be a good sample of the lens, and then giving you the opinion of a longtime photography journalist about the quality of that lens in different practical situations. MTF charts don't show everything; I have a brilliant copy of a Nikon 14-24, absolutely tack sharp, and last week had several D3 pictures ruined by a veiling sun flare that I couldn't see on the LCD. So now I'll be a little more careful...and I would have already been a little more careful if I'd seen a review from Mike Johnston or Mike Reichmann about this characteristic.

I really do think charts have their place -- and that place is a casual examination before purchase, just like you'd look at Consumer Report's numbers on a new car. But ultimately, you have to go with field performance, and I'll pay as careful attention to the road test opinions from Road & Track as I do the numbers from Consumer Reports. Same with cameras.

JC

At the end of the day, MTF charts are not anecdotal. That's why they are so useful. Unfortunately, we don't seem to have any. You are essentially criticising a non-existent phenomenon. The only collection of MTF charts of actual lenses that I am aware of, is the historical data that still exists on the Photodo site. Other charts, such as those in the Canon Lenswork books, are theoretical, as you've already pointed out.

The MTF bar charts at Photozone are at least current and are results of real tests, but are relevant only to the cropped format sensor.

All the lenses that I own that feature either on the Photodo site of old tests, or the newer Photozone site, have a resolution performance that seems to correspond fairly well with the real test reults on those sites.

The fact that lenses of the same model do vary from copy to copy or batch to batch is definitely a problem, which is why I'd like to see each lens ship with a full set of real MTF test results specific to that copy of that lens.

The Photozone tests are very well done, but they are for cropped sensors as you mention. Since the cropped sensors have a generally high pixel density they give a good figure for resolution in the central portion of the image but leave users of full frame sensors in the dark about edge sharpness.

The Nikkor 70-200 mm f/2.8 AFS has generally been regarded as a stellar lens, but D3 users were confronted with a nasty surprise when they noted poor sharpness at the periphery of the frame with this lens. However, that characteristic is fully documented in the MTF chart that Nikon publishes for this lens.



That would be nice, but is not likely to occur soon. Bad copies of lenses occur even with expensive Zeiss and Leitz lenses and it is a good idea to test one's own lenses with Imatest and compare the results with those reported by Klauss on Photozone.de. In a couple of hours, one can get a good idea of the performance of the lens at various apertures and focal lengths in the case of zoom lenses. To accumulate such data from photographic tests would take weeks or months of testing.

Bill

Bill,
I agree that it's very unsatisfactory to be in a position where one gets an idea of the performance of a lens only after weeks or perhaps even months of use in the field. During such time, one might have taken a few lucky shots or a 'once in a lifetime' shot which is technically marred because one has used the lens at full aperture (for example), expecting it to be sharp because one had heard anecdotal reports on the internet that the lens is sharp wide open. The fact that your copy of the lens is not sharp at full aperture is a useful piece of information you've discovered in the field, and you have this once-in-a-lifetime, lucky shot to prove it.

If one is concerned about technical image quality, it seems to be necessary to always test a new lens one has bought before the expiry of the return period.

Now this testing process might be fun the first time and perhaps even the second and third time and, if one is particularly technically minded, it might be fun for the rest of one's life.

However, speaking for myself, I would rather pay an extra $100 or so on top of the normal price of a lens in order to know what I'm buying and in order to avoid spending probably considerably more than $100 worth of my own time testing a lens and possibly returning the lens, then testing the second copy and perhaps even returning that second copy before going through the whole process again.

In the recent past, I tested 3 copies of the Canon 10-22 zoom before I found one that autofocussed accurately and which was almost as sharp as my Sigma 15-30. (Finding one that was equally sharp might have been a never ending process).

I once spent the equivalent of more than a whole day testing the Canon 400/F5.6 prime. The time I spent included initial comparisons with my 100-400/5.6 zoom at 400mm; repeated tests because at first I couldn't believe that my zoom was sharper at full aperture and thought that maybe there was an autofocussing issue; processing such test images and burning to CD as evidence; driving back to the store to return the lens for another copy (a 2 hour drive since I live outside the city); returning without a second copy because the store didn't have one; (the salesman suggested he could send the lens in for calibration and if it wasn't improved then he'd give me a refund); repeating the testing procedure with the same lens after calibration, only to discover that it was still not as sharp as my zoom at F5.6, although slightly improved as a result of the calibration; another 2 hour drive back to the store to return the lens and collect my refund.

Geez! What a waste of my time! Now, to avoid impressions of exaggeration and hyperbole, I'll admit that the 12 hours of driving involved in this exercise cannot wholly be allocated to the decision to purchase this lens. When I spend two hours driving into town, it's for other purposes as well. I could have done all transactions regarding the purchase of this lens through Australia Post. Instead of a 2 hour drive on 6 occasions, it could have been a 20 minute drive to the nearest Post office on 6 occasions. Add the time spent unpacking and repacking the lens and the time spent chatting with the postmaster, that's probably a total of only 4 hours, not to mention the expense of fuel and wear & tear of the car.

The bottom line is, I spent more than a day of my life attempting to buy a lens that proved to be inadequate for my purposes, and I still don't have a copy of that lens because I'm reluctant to repeat this time-consuming and expensive process of testing.

Consider what could have happened in this case if all lenses were to ship with a comprehensive set of MTF charts. I'd walk into the store with a copy of the MTF charts relating to my 100-400 zoom. I'd tell the salesman that I wasn't particularly happy with the sharpness of my 100-400 IS zoom at F5.6 and that I'd like to check out the 400/5.6 prime which I'd heard is sharper and which I'd expect to be sharper because it's a prime lens.

The salesman removes the set of charts in an envelope attached to the sealed box containing the 400/5.6 prime. I compare the charts of both lenses at F5.6. Both of us can see that my 100-400 zoom is sharper, so I reject the lens on the spot. No need for 12 hours, or even four hours of driving, and a day of my life has been saved for more useful activities, such as expressing my views on LL .

I can think of no other product where this is expected. Cars? Hifi? Fishing gear?

The effects on marketing could be interesting... You rejected three copies of the 10-22 zoom because they were not up to your standard so how should those now be sold? It is now known they are not 'perfect' in performance, so should they be graded as B+ because they are not up to scratch? And price-reduced accordingly? It could follow on that the copy you bought be classed as A++ and have a premium price.
Should a super-quality 10-22 be classed as 'L' quality?
Or a poor-compliance 'L' lens have this classification removed?

Spidermike

I can´t say that your analogy with cars, hi-fi and fishing holds.

The car´s performace isn´t measured to the very fine tolerances that lenses have to be: 0 to 60mph in how many secs and millisecs? and music quality is as much a product of the listener´s ear as the machines whilst fishing, in the end, depends on how hungry/dumb the poor old fish.

A lens does nothing except pass light. If it can´t do that to spec then it has failed. It should not, then, be re-sold at all - it should be scrapped.

Of course this would cost somebody (everybody?) money, but then so be it. Like Ray, there are better things to do in life than buy second-best thinking one has bought best. If you want to make comparisons, then would you willingly fly if you imagined the aircraft was possibly flawed? Of course not; and your ticket costs accordingly, with massive fines if negligence is proven.

I can´t ever remember this happening with either my Haselblad C lenses nor with the Nikkors that I also bought during the period up to the mid-eighties. However, after that, and with later purchases, I have had moments of doubt with Nikkor optics too. I currently own a 2.8/135 which, certainly at 2.8 and possibly throughout the range of stops, fails to equal what I used to take for granted with my earlier 3.5/135. What is the point of 2.8 if it is nothing but a focussing aid? And the rest of the aperture range is inferior too?

In my opinion, it is a wholesale dumbing down of life and expectations. Those with early experience are powerless to do anything about it and those newer to the game take it as the norm. Never in my life had I heard of a return period in case of the lens not being up to snuff; I wonder just how well advertised such a concept really is. But I can see why it now has to exist!

Rob C

True. But sound engineering is crucial to all those products - and they do have QC whether or not is as high as for lenses is a moot point (think piston-bore tolerances on a Maserati engine block). And the very fact that Canon is still in business suggests most users find the variation in quality to be acceptable - so it is in fact just as subjective as cars and hifi.


But it does do that and within Canon's QC limits. If someone doesn't like their QC limits then the choice is simple.


Think of the costs of individual MTFs and the consequently higher rejection rate - these would increase manufacturing costs and would give disproportionate increases in sales cost. You may be happy to pay up, but think of the millions who would be excluded. You talk about dumbing down (not a comment I agree with), but if it wasn't for the mass market it creates, would we really have the money to pay for the research to develop affordable 12, 15, or 22 Mega-pixel cameras?



The more I think about this, the more mind-boggling it becomes: to maintain income you would need to sell all lenses that fall within the current QC limits: a high-performing 'non-L' lens would impinge on the 'L' market; and a high-performing 'L' lens would have to be sold be a 'Super 'L' lens.
You would need to have differential costings to differentiate the model ranges: the lower-QC lenses selling at the current market price, the higher-QC lenses having a price premium to pay for the MTFs on all lenses (plus, no doubt, an added mark-up!)

So would you rather pay a (probably significant) premium to make your shopping easier, or would you prefer to stay with the current QC limits and try out 3 or 4 copies of the lens to get what you would call 'a good copy'? My guess is that the acceptable price differential is not as much as you like to think.

I didn't mean this to be a treatise on marketing but I got carried away.

Yes. You got it. Of course you would want to sell all lenses and grade them accordingly. The principle is, the customer should know what he's buying and not have to spend hours checking to see if he's actually received the quality of product he thought he was buying.

The Canon 400/5.6 prime that I rejected after spending hours of my time testing and comparing with another lens, was not necessarily faulty. I'm sure it would have been sold on to another customer. It had been sent away for calibration and had been returned as being in good order. Another customer might be quite happy with its general sharpness, not realising (and perhaps not even caring) that it was close to the bottom of the QC range of acceptable sharpness for that model.

Don't we all expect to pay more for a quality product? There's only one ultra-wide-angle EF-S zoom, the 10-22mm. Would you not prefer to have a choice of say, 3 different grades at 3 different prices? We could then have endless discussions on internet forums about the differences in resolution amongst the different grades and whether or not Grade A (EF-S 10-22) at double the price of Grade C was worth the extra money.

Incidentally, my hi fi loudspeakers, bought over 20 years ago and still going strong (Celestion SL600), each came with their own frequency response chart. It's supposed to be a real test of the decibel sound level in relation to all frequencies from 20Hz to 20,000Hz, and the charts are slightly different for each speaker in the pair as one would expect.

Although lense are mass produced, and on an assembly line, they are unique creatures. Even the hand built L lenses of Canon fame are built on an assembly line.

So purchasing a lens is about cherry picking, as if one is buying peaches. Lots of okay peaches, a few really sweet really good peaches, and lots of mushy bruised ones too. Lenses are not commodities like cars, trucks, watches coming off the assembly line. This is because there are so many variations in the manufacturing processes that are required to make a lens. And more then that, the margin for acceptability is so narrow.

To me, a lens kit or "stable" is about building it over years, and slowly. This can mean replaceing that 50mm or 24-70 zoom a few times, or getting it calibrated, and this can take a long time. It took me 8 years to finalize my own stable of glass, and so until the makers provide individual MTF charts in each box, I know of no other way to insure good glass.

Now fast forward 5 or 10 years. Will our stable of lenses resolve to the 50mp or 150mp sensors? They won't! Now we'll have to start the process all over again with new glass....

My own experience is that MTF plots are not particularly useful, although they can suggest how resolution varies across the frame at a given aperture.

They are objective, but then again so is the measurement of the diameter of the front element. The question really is whether or not they contain useful information. Unfortunately they tell you nothing about flare, ghosting, CA, bokeh, colour cast, etc. And of course the MTF is measured at only one subject distance and it is known that in many cases a lens can perform quite differently at infinity compared to the mid range, and close range.

Using an MTF to characterise a lens is a bit like judging the appearance of a car by looking at the front alone.

In fact I think it is worse, as some lenses that appear to have good MTF plots are dogs in practice. You could argue that since my experience is based on my lenses, that I had bad samples. But if that is the case - and I do not believe that it is - then MTF plots are pointless because sample variation is too significant. In fact I simply do not trust them for anything other than eliminating obviously bad lenses.

Oh yes, and these days you really need to know how a lens performs on a particular camera, as the lens camera interaction is significant.

I prefer to use reviews from people who are known to be careful workers as chances are they have performed brick wall tests, and used the lens in the field, in various situations. After all, there is no point in using a lens which performs well only in a very artificial situation.

There does seem to be an awful lot of Canon users saying they have had to return a lens to be 'calibrated', and very few Nikon users complaining. I wonder if this is the case, or if there are factors at work here which mask a similar issue with Nikon lenses? I know that Rorslett mentioned about sample variation in 17-35mm zooms in the early days of production. There seem to be no cases of 14-24 lenses being squiffy, at least not online. I wonder if the higher prices of Nikon lenses is due to better QC?

(I use Nikon just in case anyone wonders. )

Rest assurd...Nikon has the same rate of lens issues as Canon...all my Nikon shooting buddies attest to this, and if you look on Nikon sites, you too will see this as true. But to be sure, I do believe that Nikon and Canon make great lenses, and often better then the best Germen models....ever try the Canon 35L? Just one example of near perfection, and a lens that the Germen makes have little or nothing over.

Yes. I understand that, which is why I'm advocating real MTF tests of the lens you buy. Such tests would not be merely suggestive of resolution but would be indicative of actual resolution.



That's exactly what it is not. Judging the appearance of a car by looking at the front would be like judging the quality of a lens only from the existing specifications which the manufacturer offers.



If resolution peformance at various apertures is a useful thing to know, then of course MTF charts contain useful information. If you are not concerned about or interested in resolution and are only concerned about matters such as flare, waterproofing etc., then of course the MTF chart is not providing you with useful information.

MTF charts can provide an indication of the quality of bokeh (where there's little divergence between the sagittal and meridional lines, bokeh is said to be good). MTF tests can also be carried out at different focussing distances. I believe some macro lenses can be sharper at close distances than at infinity. The fact that MTF charts do not tell you everything about a lens is no reason for not having such tests. I'm not advocating that other sources of information about a lens should be discouraged. You seem to be arguing that there's not much point in having an eye test because such a test does not provided useful information on the state of your bowels.



You mean you prefer apples to oranges although you've never tasted an orange? Unless you have acquired one of the lenses which Photodo tested, you probably don't have any MTF charts specific to an individual copy of any lens you own. And even if you did, it wouldn't be much use to you in isolation. However, a system in which all lenses ship with MTF charts would be very useful.

As for reviews from people who are known to be careful workers, how does that help you in a situation where QC variation is common?

As a matter of interest, I bought the Canon 400/5.6 prime after reading Michael's review of it in which he compared it with his copy of the 100-400 IS zoom and demonstrated with shots of brick walls that it was clearly sharper at F5.6.

My copy of that lens clearly wasn't sharper but it's not clear to me to what extent I might simply have an above average copy of the 100-400 IS zoom. The purpose of real MTF tests specific to each lens is to provide a universal reference point to make resolution comparisons accurate and easy. A metre is a metre wherever you are.

Well, perhaps the vendors would improve their tolerances so that we would not have out of spec lenses leaving the factory.

Erik

Just a question, anyone having an idea what Canon means by calibrating a lens? The only thing I can see calibrated in a lens is the aperture control and autofocus mechanisms (although autofocus is mainly done by the camera). The bayonet can probably adjusted to. But what do they do about decentered lenses? Do they disassemble/reassemble the lenses?

Best regards
Erik

Hi,

Anyone knows what "lens calibration" means?

Erik

Hi,

The German monthly "Color Foto" has added focus accuracy to their lens tests, my reflection is:

a) That's a good thing
There are significant variation, which is a bad thing.

I think that MTF tests are a very good indicator of the sharpness of the lens. There are other parameters which at least in some situations are more important than sharpness, especially flare and ghosting.

Some optical problems can be corrected easily in postprocessing, so I may not care that much about those, examples are:

Vignetting
Distorsion
Lateral chromatic aberration

Finally, there is a difference on how a lens works in the "lab" and in the field. A lens that has issues in the "lab" will also have issues in the field. Knowing the strengths and weaknesses of your lens is always helpful.

I think that testing your own lenses is a very good idea. I had experience with two lemon lenses, so I feel the issue is real. To put things in perspective I owned something like 25 lenses. Now days designs are complex and there is pressure on costs which leads to perhaps more sample variations than we had before.

Fixed focals do have some advantages, but my experience is that in many cases my vantage point is fixed. Having zooms allows med to crop in the camera so I don't need to crop in the image.

Erik

In order to improve the tolerance, they might have to do something like an individual MTF test on each lens. If they do this, I'd like to have the result for reference purposes and comparison with other purchases.

If they find a way of improving tolerances through another process, then that's fine with me also because Photodo type MTF tests will then be more relevant to all copies of a particular model of lens.

Ray: You seem to have missed the points I made.



I am confused. My comments apply to any MTF chart.




That is exactly what it is. As I explained quite clearly, an MTF only tells you about part of the optical performance of a lens. My reasons were given in the earlier post to which I refer you.




I am interested in optical performance in the sense of how well it creates images.




That I doubt. Bokeh is all about how the lens images light in regions outside the plane of focus.



But they aren't. You referred to PhotoDo. They are done at one distance.



Yes, but if the tests tell you so little, they become misleading.




You have completely lost me. I am saying that an MTF tells you so little about the optical performance of a lens as to be misleading.



But you referred to PhotoDo as being of use! Are you changing your mind?



I would not want to buy a lens from such a manufacturer.




Ray: You confuse me as you have changed your argument. The point I was making is that MTF charts are not a good basis for deciding between two different lenses, maybe with similar specs but from different manufacturers

To remind you, here is your post that I was referring to:

"I think you've got this the wrong way round, old chap. MTF charts are as objective as you can possibly get, provided they result from the testing of real lenses, like Photodo MTF charts do."

In the context of the above, my response makes sense.

The point you made in your reply - which is quite different - is that an MTF can provide a way to check that a particular sample of a lens is within the manufacturers specs. I do not argue with that as quantities such as colour cast, flare resistance, bokeh, contrast etc would tend to be consistent between samples of the same lens, and all you want is to check for gross deviations in resolution, thus indicating something very wrong.

It is a little hard to respond if you make one argument, and then respond with a totally different one.

As I stated earlier, I have rarely seen discussion of issues with Nikon QC, whereas discussion of Canon QC issues is commonplace. Given that I usually look in Nikon forums, you would expect me to find Nikon issues.

I don't doubt the quality of many Canon lenses.

There's an encoder on the lens focus barrel that tells the camera where the lens is focused to. The wiper for this is sometimes adjustable, allowing the lens to be adjusted to a particular body. There are also electronic adjustments via EPROM by factory service software that are now being implemented in the new higher-end cameras via firmware. This will go a long way to overcoming the situation as it exists.

Due to manufacturing tolerances, there will be differences in the lens flange to sensor distance and the lens comes from the factory adjusted to a mean, which also has manufacturing tolerances. This is the main reason for variations between lens/body combos. The secret behind the focus accuracy of the Leica M series is instead of the lens flange being mounted on a sub-assembly attached to the main body, the body is cast as a single unit. In the case of the M3, the body was machined from a solid billet.

Designing auto focus lenses will always be a bit of a headache due to the necessity to have a certain amount of slop in the tolerances to ensure the elements move freely. Add to this the front element wiggle inherent in a lot of zooms and the advent of image stabilization in the lens, and it's amazing that these things work as well as they do.

One thing I notice in most of these discussions is little or no mention of color and contrast, both of which, to me, are every bit as important as sharpness. I tried one of the Sigma 24/1.8 DG's, and while it was sharp enough, had flat color and harsh contrast characteristics that no amount of post-processing could overcome.

As to the original inquiry, I've never hung an MTF chart on the wall and admired it.

Thanks for good explanation. So calibration just fixes autofocus issues?

I'm actually somewhat confused. As far as I understand the camera is actually doing the focusing and the lens assembly just acts as a servo. Now I understand that servos can both under and overshoot, and that the amount of over/undershoot may be specific to each lens. Any error related to the flange distance should be specific to the camera body and not to the lens, in my view.

Best regards
Erik

Autofocussing accuracy is a major problem which Canon appears to be addressing by adding micro-autofocussing adjustments to their latest cameras. The recently announced 50D has such adjustment capability and any adjustments can be saved in relation to a specific lens.

As I understand, the alignment of the lens elements in their groups is fixed during manufactuire. If some minor misalgnment has taken place, resulting in the lens being less sharp than another copy from a different manufacturing batch, then nothing can be done later to correct this, so I believe.

This is why, in my view, it is always best to start off with a lens that is sharp. Why waste one's time testing autofocus and flare issues if you subsequently discover the lens is not sharp. If the lens is not sharp, there's little you can do about it, except perhaps always use it at F11. In the case of some ultra-wide angle lenses, they are sometimes not even sharp at F16, in the corners on full frame.

Flare and a lack of precise autofocussing, are definitely concerns. However, it is possible to get around them. If calibration doesn't fix the autofocussing problem, Live View allows for great accuracy of manual focus. Flare can often be stopped by holding a card or hat at the edge of the lenshood, blocking the sun's rays.

My Sigma 15-300 has a repution for being susceptible to flare because of its bulbous, protruding front element. It's the nature of the design. I think the Nikon 14-24 also suffers from a similar problem for the same reasons, but perhaps less intrusive due to better internal coating. Sometimes just holding the palm of one's hand between the direct rays of the sun and the edge of the lenshood is sufficient.

I also wonder if such attributes (apart from autofocus accuracy) which are not reflected in MTF charts, have the same degree of QC variability as resolution.

If a lens has a reputation for bad flare, how would you test different copies of the same model in order to select the one which produced the least flare? Has anyone done such a test and found that there is significant variation in proneness to flare amongst different copies of the same model?

With Canon and all the other lenses with built in motors, all moving parts are in the lens, which is what does the focusing. The body just contains the sensors to corroborate with the lens when focus has been obtained. Both the lens and body contain processors and the processor on the main circuit board does the heavy lifting. As with all mass produced products, there will be a certain amount of variance from example to example. Both the body and lens are adjusted to a mean, so how the body/lens combo performs will depend on which end of the tolerance levels both the body and lens are. Also, any anomalies in the communication lines can cause grief, including an inability for the lens and camera to work together. Stabilization lenses have another motor to add to the mix.

Without the lens, flange to focal plane distance errors are moot. It's a symbiotic relationship.

In the never ending quest to out hype the other guy, I think we may be approaching the complexity barrier insofar as current technology is concerned. Like the Mhz, MP, and FPS wars, it may be time to back off on the whiz bang and concentrate on making the current stuff work better. Unfortunately the consumer has been conditioned to constantly chase their tail so the cycle continues.

You can save a lot of money with this poster from Edmunds optics. $18 plus shipping.



This has the old USAF resolution test pattern in various colors and angles all the way out to the corners. When you get a new lens you take a shot at a few different f stops. Pixel peep the raws for resolution. You will know in seconds if you have a dud lens.

Edit: thats a crop from the center. I didn't put in a pic of the whole thing for copyright reasons. You can also read the text in the crop.

Yes. You can do that. There are also free charts available at Norman Koren's site, and of course newspapers are always readily available.

However, as slough has already pointed out, performance at infinity is not always indicative of performance at close distances. When I tested my 400/5.6 prime on real world targets, I tested it at infinity, close to infinity and at around 30m distance. I didn't test it at a distance of 3 or 4 metres because that wasn't the typical distance from which I figured I'd be using the lens.

For all I know, that lens might have been sharper than my 100-400 zoom at a distance of 4 metres from the target.

The issue is really a matter of efficiency and how much you value your time. If you enjoy testing lenses, then you would probably be against the idea of the manufacturer providing you with a complete set of MTF charts at various frequencies, various apertures and various focussing distances.

After all, you might hit the jackpot and find that a lens you've just bought at a regular price has absolutely stellar performance, way above the average, yet you've paid only an average price for it.

I happen to value my time more highly, even though that might seem odd in view of the number of posts I have on LL. I'm willing to pay more for a lens if I can be assured of it's sharpness. Chromatic aberration, flare and bokeh I can deal with. Lack of sharpness I can't.

Hi,

I think that Ray has a point here. On the other hand I think that there are amny sets of MTF:s curves needed to characterize any lens. There would probably be sets of MTF diagrams for different distances and wavelenghts of light.

Erik

Only one?

If one is going to devise a streamlined, efficient and automated system for putting each lens through an MTF test, one might as well take the opportunity to do a thorough test at various apertures and possibly different distances if there's any reason to suppose performance will vary with focussing distance.

If a job's worth doing, it's worth doing well .

But don't the latest Nikon bodies offer micro adjustments too?!?