Popular Photography: How We Test

Color Accuracy
To measure a camera’s color accuracy, we shoot the Macbeth Color Checker DC test target using that brand’s 50mm f/1.4 lens at f/8, and at every ISO the camera offers, in whole-stop increments. We do two passes, one with the camera’s automatic white-balance setting and one with a custom white balance set using a gray card. Typically, the automatic white balance setting garners the best results with our lights, but if that’s not the case we make a point to note it in the test.

After shooting the test target, we convert the resulting image into 16-bit and 8-bit TIFFs using the RAW conversion software provided by the manufacturer. We allow fine-tuning of the white balance with a dropper on an 18-percent gray patch of the target if the RAW software allows this, as most do. Other than that, we leave the manufacturer’s default RAW conversion settings intact.

Once the images are converted, we crop down to the center of the target area, within the alternating grey, white, and black patches, so that those repeating patches are not overly weighted in the measurement. We then use Chromix ColorThink software to compare the images of the target we shot against a reference scan of the test target generated on an Epson Expression 10000XL flatbed scanner, which has a color gamut well beyond that of the cameras we test. We update this scan several times a year to ensure that any subtle changes in the color patches of our chart are reflected in the reference we use for the test. As color test targets are exposed to light during the tests, it is natural for the patches in the target to change very slightly. Whenever the target is not in use, we keep it in a protective, light-tight sleeve to minimize its exposure to light.

The ColorThink software generates an average Delta E value for the images we shot, which is what we report in our test results. Delta E is a standard unit of measure for the difference between specific colors, governed by the International Commission on Illumination (CIE); a Delta E score of 1 represents the least amount of color difference that is discernable by a trained observer when two color patches are placed side by side. Most people would require a Delta E of 2–3 before noticing the difference between two colors, maybe even more so if the two patches are not side by side.

(Interestingly, most color films are not necessarily color-accurate. Manufacturers of film often accentuate certain colors over others, or boost saturation overall, in order to create a color palette considered more pleasing by their customers. So digital cameras offer an ability to capture colors in a much more accurate way than film ever did. Given the ease and flexibility of modifying digital images after the fact, the range of colors available to photographers has never been more extensive or customizable than it is today.)

In the near future, we will to migrate to the new Macbeth Color Checker SG, which has more skin-tone patches and has taken the place of the Color Checker DC target in X-Rite’s line of Macbeth color charts.

Noise
When you turn up the sensitivity of the sensor in a digital camera, you get noise—either off-color splotches or white (or overly bright) dots. Manufacturers use a variety of means to minimize the effect of noise, though this process often results in a loss of resolving power, as the noise is either blurred away or otherwise masked.

Our noise test isn’t concerned with how the noise is generated or how the manufacturer attempts to minimize it. Rather, it looks to measure how much variance there is from the grayscale patches in our test target. This is measured in standard deviation from each of the various patches on our X-Rite Gretag Macbeth Color Checker target.

We shoot that target at all the camera’s ISOs, unless there is a sensitivity setting that reduces the pixel count. We typically do not test those settings, which are found primarily in cameras without interchangeable lenses. After shooting, we convert the RAW images into TIFF files using the software provided by the manufacturer, and with the manufacturer’s default noise-reduction settings. We then process the resulting image with the most recent version of DxO Analyzer software from DxO Labs. The software generates the average standard deviation, which we report.