Popular Photography: How We Test

For decades now, Popular Photography has tested cameras and lenses so that you can make a more informed decision about the equipment you use to capture images. Whether you’re documenting your life or creating art, you need to know not only which camera or lens is best suited to your shooting style, but also how that camera or lens performs so you can use it more effectively.

Our tests are designed to measure not only the quality of the images the camera or lens produces, but also to report on its usability, design, and standing against comparable models. We perform our lab testing in the Popular Photography Test Lab, located in our New York City office. A temperature- and humidity-controlled environment, it ensures the same conditions for every camera or lens tested there. Our lens test bench rests on a set of pneumatic legs, so it floats on a cushion of air to ensure that any vibrations in the building won’t taint our test results.

We perform the lab-based portion of our testing in the same way (or exactingly equivalent ways) for all makes and models, so that each item is subjected to the same rigors under the same conditions. This way, our test results allow you to compare the results from camera to camera and lens to lens. We trust that, armed with this information, people can better decide which model might be best for them, and gain some insight into the best way to use what they eventually choose.

Lighting
We illuminate our resolution, color-accuracy, and noise test targets with Dedolight DLH400D daylight-balanced HMI lights. These lights offer the best approximation of daylight we know of, and are powered by flicker-free ballasts. While many photographers are using daylight-balanced fluorescent lights for portrait and studio work, these don’t reproduce the spectrum of light as well as HMI lights do and are not a true continuous light source, since they flicker. And with the extremely short shutter speeds often required for photographic testing, that flicker would create too much fluctuation for us to accept them as a light source in our tests.

Common Test Settings
We shoot our resolution, color accuracy, and noise tests in RAW, with the camera on a tripod and set to the AdobeRGB color space. We use the camera manufacturer’s 50mm f/1.4 lens at f/8 with shutter speed varying based on ISO. We use single-shot autofocus, centerweighted metering, and bracket to ensure that we obtain a proper exposure, checking against patches in the test targets to select the best exposure.

We convert these images into uncompressed TIFFs for analysis using the RAW conversion software that comes with the camera. We apply only minimal changes from the manufacturer’s default settings in its software. Our reasoning here: Camera makers are selling their customers a package that includes software, and the default settings should be a statement of how they think the camera should perform. In the case of noise reduction, for example, we expect that the manufacturer will set the defaults at settings that make the best tradeoff between retention of fine detail and adequate reduction of noise for that model. Furthermore, any other attempt to adjust that setting would prevent us from having a standardized test procedure.

We should mention that in the case of Nikon, we use one of two different 50mm f/1.4 lenses. For those Nikon bodies that accept only lenses with built-in focusing motors, such as the D5100 and D3100, we use the AF-S Nikkor 50mm f/1.4G. If the DSLR under test can accommodate lenses without their own focusing motor, we use the older AF Nikkor 50mm f/1.4D.

Also of note here: Any DSLR body will behave differently with different lenses. Our AF test (and other tests) provides a way to limit that variable by always using the same lens for a given system. The only exception to this rule is when testing interchangeable-lens compacts (ILCs), since there are currently no 50mm f/1.4 lenses for these systems. Because the lens selection for these systems is still so limited, we have been using the wide-to-normal kit lenses for each. We also have not implemented rigorous AF testing of these systems yet.

Following, we detail each of the individual lab tests we perform here at Popular Photography. We also always use every test camera extensively in the field to see how it performs in everyday shooting conditions and to assess any functions—such as focus tracking of subjects moving toward or away from the camera—that are impractical to test in our lab.