What do we actually know about how various DSLR's get video output from sensors with far more photosites than there are pixels in the final output?

RED owner Jim Jannard has posted evidence in the RED forums that the 5DMkII does it by reading only every third line, and then using all data in the lines read by binning. One piece of evidence is a zone plate resolution test, which shows a strong asymmetry in aliasing: far more aliasing in the direction across the lines than along the lines, because the binning along each line handles aliasing better than the sub-sampling between lines. (The AA filter cuts of at a spatial frequency suited to still image resolution and so too high to avoid all aliasing at the lower video resolution.)

It seems likely that the same is done in other Canon DSLR's and in Nikon DSLR's. For one thing, full binning of all photosites would greatly improve high ISO performance by not discarding two thirds of the signal, and I have not heard of any Canon or Nikon DSLR so clearly outperforming the 5DMkII for high ISO noise in video mode. Also, full binning would require either
- far higher rates of transfer of signal to the sensor's edge than needed for stills (12MP @24fps of higher) [Edit: maybe this is not so bad, as a whole row can be moved down to the edge in parallel on all columns at once, so only the read off along the edge is at high speed.]
or
- special wiring to bin at the photosites so that only the fewer binned super-pixels need to be transferred.
[Edit: or is it enough to do binning at the sensor's edge, to lower the rate needed to read binned pixels off along the edge?]

But the GH-1 might be different, as it and its sensor seem to have been designed for video from the beginning. (Its sensor is not used in any other 4/3 or m4/3 camera.) Does anyone know, or have evidence? Has anyone done a zone plate resolution test on the GH-1? Has anyone assessed high ISO noise levels of the video modes of various "combo-cams"?

Got a link?

I found the thread about 5DMkII video at the RED forums:
http://reduser.net/forum/showthread.php?t=22517

LL poster and "RED Problem Solver" Graeme Nattress comments there, so maybe he can help this discussion too.
Graeme: do you know of zone plate tests on any other DLSR or m4/3 video mode?

P. S. This interview (from Japanese to Spanish and then robotically to English, so in rather "Zen Koan" style) hints at how the GH-1 sensor differs qualitatively from other m4/3 sensor, for the sake of video:
http://www.quesabesde.com/noticias/entrevi...-tercios,1_5774
http://translate.google.ru/translate?u=htt...en&ie=UTF-8
A robot translated quote:
"GH1 is used in a sensor multiaspecto prepared for high-definition video in the whole process and transmit the signal is digital ... The GF1, by contrast, uses the Lumix DMC-G1 [sensor] and its four-channel analog transmission."

If the 5D2 were able to bin all the pixels required for a 16:9 video image, that would increase signal-to-noise very significantly, wouldn't it? A 16:9 aspect ratio cropped from the 21mp 3:2 aspect ratio would result in 18mp. If it were possible to bin groups of 9 pixels to achieve the 2mp required for progressive HD, that would be fantastic. I'd buy such a camera straight away, if the price were right .

We can only dream! Well the 7D (and the new 1D) do have dual digic 4's ... but somehow I think that's still out of the hardware's league processing wise. The 5DmkII and the 7D are primarily for stills and video (although pretty good if you can get by the aliasing/moire) is an afterthought. We can only hope that someone in Canon engineering will come up with a better compromise than all the line skipping that's currently happening and at least get us at a point where it's a bit better in the aliasing/moire department :/

That plate shot does kind of say it all though

Yes: it should improving sensitivity (usable ISO speed) by a factor of three, roughly. While "line triage" (using all pixels in every third row) is in turn a factor of three more sensitive than sub-sampling in both directions, which is probably what Live View modes too.

And this is why I am curious as to whether the GH-1 sensor, or any recent sensor, does it "right", or whether future sensors will do this. As far as I can tell, it should only require a little bit of extra "color binning" hardware at the bottom of each column of photosites, and maybe modest bump in operating frequency.

As far as I can see on the 5D2, there is no binning going on. It's a strict 1/3 decimation vertically, and horizontally there's some kind of poor downsample because the resulting horizontal resolution is around 75% of what it should be for 1080p, and there's significant aliasing too.

From what I can see on the GH1 there's a combined downsample then upsample process going on, and what appears to be a similar line skipping phenonema. I'd suspect, but can't be sure, that it lineskips to 720p, then upsamples that result somehow to 1080p, which is why the 1080p is both soft and aliasy.

The zone plate is practically the ultimate diagnostic test for what cameras are doing - yet they're rarely used as I'm sure many people think of them as esoteric.

From the demo movies I've seen of the 7D and 1DmkIV, they looks similarly aliasy to the 5D2 with the majority of the aliasing and false colors happening on horizontal detail - ie, the aliasing is worse vertically.

I've (obviously) been able to measure the RED One many times for resolution. http://www.cinematography.net/natress-red-res.html shows the results of such a zone plate test. Low levels of aliasing on a DSLR are not usually a major problem. However high levels, like we see in current vDSLRs eat into the already meagre bit rate allowed for recording the video image. Aliases are more distracting on video on a still as they move in the opposite direction to the movement. So what you can "get away with" on a stills camera is not what you can get away with in motion. The whole point of RED's DSMC line is to not play these skipping / upsampling / downsampling tricks but to read off the full sensor fast enough to allow for high quality motion and high quality stills. And it's very hard to do that!

Graeme

Thanks. That is a pity, though it means that there is plenty of room for future improvement.

Does it make sense that the GH1, with its 16:9 still resolution of 4352 x 2448, is doing line triage (not "decimation" surely: that means taking one tenth!) to 2448/3=816 lines, and then upsampling to 1080p?

With the 5DMkII (and maybe all its kin) I suppose to be precise with words it is downsampling or resampling rather than binning within lines because it is all done after the signal is removed from the sensor, and even probably after A/D conversion ... video being essentially added in firm-ware without hardware revisions.

Yeah, triage not decimation :-). Although we use the word decimation to mean "throwing away" in general, and describe image downscaling as a filter then decimation process.

Looking at the patterns of the aliases on the chart, I'm guessing that it does something like 2448/4 = 816 - crop to 720, then upsample that to 1080p. Hard to tell for certain though.

Within lines, on the 5D2, it's hard to tell what is happening exactly. I've seen what look like demosaic artifacts in the end result image, so it's certainly possible the image could be demosaiced before a scaling to 1920x1080. However, that doesn't fully explain the very poor horizontal resolution. I'd assume each row that is read is dealt with digitally for it's downsampling to 1920, but as for how, I really don't know.

Graeme

I just found this interesting talk by Canon rep. Tim Smith to the Digital Cinema Society about the 5DMkII:
http://digitalcinemasociety.org/content.ph...ting%20Workshop and see Clip Smith1 in particular.

He says that no pixels are skipped, and briefly mentions 6 pixel binning. Could such asymmetrical binning (each block three wide, two high converted to a single RGB pixel, and subsequent down-conversion?) fit the observed asymmetrical aliasing issues?

He also says that the 5DMkII video features were added specifically in response to news agencies like Reuters who wanted their primarily still photographers to also be able to grab video clips for web sites and such, and hence the "auto only" approach was for the simplicity of still photographers with little video experience.

It is also interesting to see the big rigs, with gyros, external video screens and such, used to turn a 5DMkII into a truly professional video tool!

If data was binned in such a pattern:

R G R
G B G

and

G R G
B G B

alternating

I'd expect to see slightly more aliasing vertically than horizontally and very high resolution in both horizontal and vertical direction. Instead, we see a vast disparity in aliasing horizontally to vertical, with the vertical MTF responding to detail way beyond 1080. Horizontally, it's both aliased and soft measuring < 1400 or so out of the 1920. The binning scheme may explain the strong vertical aliasing, but it doesn't explain the low resolution + aliasing you see horizontally. And it doesn't explain the vast difference between horizontal and vertical, only a relatively small difference.

I've done

G R
B G

"binning" from the bayer pattern and you do get some aliasing, but it's hardly like the brutal aliasing we see on the 5D2, and it doesn't exhibit the wild cross color artifacts that we see either.

5D2 sensor is: 5616 x 3744, so the supposed binning would produce 1872x1872 image to be cropped / scaled to 1920x1080. Presumably to get the 16:9 aspect ratio you'd crop to 1872x1080 then scale, but that'd produce a very sharp horizontal resolution, but we don't see that.

Line skipping the 5616 x 3744 would lead to a 5616 x 1248 image, which would be cropped to 5616x1080 then scaled to 1920x1080. That doesn't explain soft horizontal resolution either.

None of the suggested theories fully explain the image facts. However, Canon marketing not withstanding, I reckon the line skipping does explain more of the visual appearance we see than the binning theory. There is still the scaling that occurs after either, and that is where I suspect more is happening. There also does look to be a demosaic stage in their too, from what looks to be visible demosaic artfacts, but they could also be compression artifacts - hard to tell. Binning would not explain demosaic artifacts should that be what they are.

Graeme

The above scaling stuff mixes up aspect ratios a bit so I'm going to re-look at it here:

Also can look at it this way: a 16:9 crop from the 5D2 sensor is 5616x3159. Doing a 2x3 bin on that would be 1872x1579, or doing the line skip would give 5616x1053.

On the zone plate, the major vertical alias is centred around something very close to 1080. I reckon we'd see it in a different spot if the binning was occuring and taking us to 1579 as a first step.

I still can't fully explain the horizontal softness though. The aliases, seem centred around 1920.

Graeme