Image quality factors

Image quality is one of those concepts that's greater than the sum of its parts. But you can't ignore the parts if your goal is to produce images of the highest quality. Every image quality factor counts.

This page introduces the key image quality factors and briefly describes how Imatest™ measures them. It is a guide to Imatest organized by image quality factor. Other guides include the Tour (organized by module) and Imatest documentation (the Table of Contents).

To illustrate the quality factors, we use this early morning image of Monument Valley from Hunt's Mesa, near the Arizona-Utah border. A 13x19 print (available for purchase) is breathtaking, though it can't capture the experience of grabbing the camera gear and running for the truck as the storm broke. Hunt's Mesa isn't public land; you need a Navajo guide to get there. Tom Phillips does an excellent job.

The image was extensively edited. The unedited image, straight from the RAW converter (with some tonal adjustment) is shown below.

When people ask me about "digital manipulation" I delight in telling them that I would no more manipulate an image than Ansel Adams. Of course I'm jesting. In Ansel Adams: A Biography, Mary Street Alinder describes how Adams printed his famous Moonrise over Hernandez image. He'd spend the entire morning making test prints — dodging, burning, and (chemically) manipulating them until they met his exacting standards. It took him that long even though he kept meticulous notes. Then he'd spend the afternoon making prints. From a processing viewpoint he would have loved digital. But probably not from a business viewpoint. Thanks in part to his brilliant business manager he earned more in an afternoon of printing than most of us earn in a decade.

There are two key aspects of image quality.

• Factors affected by post-processing. Contrast, color balance, color saturation, etc. These are factors that are adjusted in the camera after the image has been captured and can be enhanced digitally (though casual photographers rarely adjust them). Many of these factors can be analyzed using test charts (color or step charts), which may be placed in scenes (still lives, etc.), but thir effect on subjective image quality can be highly scene-dependent, making it difficult to assign "good" or "bad" rankings. For example, higher contrast images usually score higher in visual quality assessment tests, but high contrast can cause clipping (usually visible as burnt-out highlights) in contrasty (often sunlit) scenes. Imatest results for these factors need to be interpreted carefully. Because of their great importance to photo finishers these factors are the focus of much image quality literature. 
• Factors intrinsic to cameras, lenses, and printers. Sharpness, noise, dynamic range, color accuracy, color gamut, etc. Imatest focuses on these. Key Imatest modules are summarized on the right.

Summary table

This table summarizes the image quality factors described in detail below.

Quality factor	Chart	Module	Comments
Camera, lens
Blemishes, Sensor defects	plain, uniformly lit surface	Blemish	Can be displayed on LDC flat screen monitor with Screen Patterns. Imatest Master only
Color accuracy	GretagMacbeth ColorChecker (24-patch)	Colorcheck, Multicharts
	IT8.7	Multicharts
	ColorChecker SG	Multicharts	Imatest Master only
	Custom "pie" charts	Multicharts	Imatest Master only
Dynamic range, Tonal response, Contrast	Step charts	Stepchart	Transmission charts such as the Stouffer T4110 recommended for DR.  Algorithm
	Reflective step charts	Dynamic Range	More convenient for measuring DR than Stepchart because it doesn't require a transmission chart.
	Special charts: ISO-16067-1, QA-62, EIA Grayscale, ISO-14524 OECF, ISO-15739 Noise	Stepchart	Imatest Master only. Most are available from Applied Image.
	ColorChecker, ColorChecker SG, IT8.7, Step Charts	Multicharts
	SFRplus	SFRplus	Does not measure DR. Highly automated. Measures several factors.
Exposure accuracy	Step chart (reflective)	Stepchart
	GretagMacbeth ColorChecker	Colorcheck
ISO Sensitivity (closely related to Exposure Index)	Step charts	Stepchart	Two ISO sensitivity measurements are displayed when incident light (lux) is entered. Details in ISO Sensitivity and Exposure Index
	Various color and step charts	Multicharts
	SFRplus	SFRplus
	GretagMacbeth ColorChecker	Colorcheck
Lateral chromatic aberration	Slanted edge	SFR	Printable by Test Charts
	ISO 12233, Applied Image QA-77		Printed on photographic media
Lens distortion	Square or rectangular grid or checkerboard,	Distortion	Printable by Test Charts or displayed on LDC flat screen monitor with Screen Patterns.
	SFRplus	SFRplus	Highly automated. Measures several factors. Results in the Image & Geometry display.
Light falloff, vignetting	Plain, uniformly lit surface	Lightfall	Can be displayed on LDC flat screen monitor with Screen Patterns.
Noise	Step charts	Stepchart
	GretagMacbeth ColorChecker	Colorcheck
Sharpness (MTF)	Slanted-edge	SFR Rescharts	Printable by Test Charts  Algorithm
	ISO 12233, Applied Image QA-77		Printed on photographic media
	SFRplus	SFRplus	Highly automated. Measures several factors.
Veiling glare (lens flare)	Reflective step chart with "black hole"	Stepchart
Color moiré	Log Frequency	Log Frequency
Software artifacts	Log F-Contrast	Log F-Contrast
Data compression	Log F-Contrast	Log F-Contrast	Not yet fully supported
Prints
Dmax (deepest black tone)	Custom test chart printed from file, scanned on profiled flatbed scanner	Print Test	Gamutvision extracts these properties from ICC profiles.
Color gamut

Image quality factors for cameras and lenses

 

Sharpness

Sharpness is arguably the most important single image quality factor: it determines the amount of detail an image can convey. The image on the upper right illustrates the effects of reduced sharpness (from one application of the Picture Window Pro blur operation).

Device or system sharpness is measured as a Spatial Frequency Response (SFR), also called Modulation Transfer Function (MTF). MTF is the contrast at a given spatial frequency (measured in cycles or line pairs per distance) relative to low frequencies. The 50% MTF frequency correlates well with perceived sharpness— much better than the old vanishing resolution measurement, which indicated where the detail wasn't.

Sharpness and MTF are introduced in Sharpness: What is it and how is it measured?

The perceived sharpness of a print or display is measured by Subjective Quality Factor (SQF), which is derived from MTF and the Contrast Sensitivity Function of the human visual system.

Sharpness is measured with Imatest SFR, Rescharts Slanted-edge SFR, or the highly-automated SFRplus, using targets you can purchase or print with the Imatest Test Charts module. Concise instructions are found in How to test lenses with Imatest. Detailed instructions are found in Using SFR Part 1 - Setting up and photographing the target and Using SFR Part 2 - Running Imatest SFR.

An alternative method of measuring MTF uses sine pattern charts that increase in frequency logarithmically. This method provides a check on the slanted-edge method; it is more direct but less accurate. It is described Rescharts Log Frequency and Log Frequency-Contrast.

System sharpness is affected by the lens (design and manufacturing quality, focal length, aperture, and distance from the image center) and sensor (pixel count and anti-aliasing filter). In the field, sharpness is affected by camera shake (a good tripod can be helpful), focus accuracy, and atmospheric disturbances (thermal effects and aerosols).

Some lost sharpness can be restored by sharpening, but sharpening has limits. It can't restore detail where MTF is very low (under about 10%). Oversharpening, illustrated on the right, can also degrade image quality (especially at large magnifications) by causing "halos" to appear near contrast boundaries. Images from many compact digital cameras are oversharpened.

Noise

Noise is a random variation of image density, visible as grain in film and pixel level variations in digital images. It arises from the effects of basic physics— the photon nature of light and the thermal energy of heat— inside image sensors.

Noise and its measurement are introduced in Noise in photographic images.

Noise is measured by several Imatest modules. Stepchart produces the most detailed results, but noise is also measured in Colorcheck, SFR, SFRplus, and Light Falloff.

Noise scales strongly with pixel area. It can be very low in digital SLRs, which have pixels at least 5 microns square. But it can get ugly in compact digital cameras with small sensors, especially at high ISO speeds. It is also affected by sensor technology and manufacturing quality.

Typical noise reduction (NR) software reduces the visibility of noise by smoothing the image, excluding areas near contrast boundaries. This technique works well, but it can obscure fine, low contrast detail. Some specialized programs, such as Neat Image, which is trained to recognize the structure of noise, have fewer side effects. Several of the patterns in Test Charts can be printed and photographed (or entered directly into software) to examine the effects of NR algorithms.

Dynamic range, tonal response, and contrast

Dynamic range (or exposure range) is the range of light levels a camera can capture, usually measured in f-stops, EV (exposure value), or zones (all factors of two in exposure). It is closely related to noise: high noise implies low dynamic range. It is also related to the tonal response— the relationship between light and pixel level (shown below). Contrast, also known as gamma, is the slope of the tonal response curve. High contrast (shown on the right) usually involves loss of dynamic range— loss of detail, or clipping, in highlights or shadows— when the image is displayed. (The image file often has a greater dynamic range than the displayed image.)

Dynamic range, tonal response, and gamma are measured (A) by Stepchart using a transmission step wedge, preferably one with a maximum density at lest 4.0 (equivalent to 13.3 f-stops), or (B) by Dynamic Range, a postprocessor for Stepchart that uses results from up to four differently-exposed reflective stephart images. The latter technique is often the most convenient because it doesn't require a special light source in a darkened room.

Dynamic range is a strong function of pixel area, which is proportional to the number of electrons a pixel can store. It is invariably better in DSLRs (which have relatively large pixels; at least 5 microns square) than in compact digital cameras. It can be maximized by setting the camera at the lowest ISO speed.

Displaying images with large dynamic ranges (which can be well over 1000:1; 10 f-stops) can be problematic in printed media, which has a maximum dynamic range of about 100:1 (a little over 6 f-stops; 200:1 at the absolute maximum). Reducing contrast can make the image look flat and dull. Some processing is usually required, like applying an "S" curve or dodging and burning (lightening or darkening) portions of the image. Contrast masking is a particularly effective approach. Tonal quality and dynamic range in digital cameras offers additional tips.

Lenses have an intrinsic contrast— the higher the better. But lens contrast isn't exactly intrinsic. It results from flare light— light originating inside and outside the lens's field of view that bounces between lens elements and off the inside of the lens barrel. Flare light tends to fog the image and obscure shadow detail: it reduces dynamic range. To our knowledge there is no solid standard for measuring lens contrast. But you can measure it, or at least compare it for different lenses and apertures, by running Stepchart on a Q-13 (or equivalent) chart mounted on a white board that extends beyond the camera's field of view. (Recall, contrast = gamma.) There would be less flare light, hence higher contrast, with a black board. Actual lens contrast depends strongly on the scene.

Color accuracy

Color accuracy is an important but ambiguous image quality factor. It can be critical in medical and technical photography, but less so in pictorial (consumer) photography, where many viewers prefer enhanced color saturation, particularly in "memory colors": foliage, sky, and skin. Accurate color is not the same as "pleasing" color.

Whatever the application, it is important to measure a camera's color response: its color shifts, saturation, and white balance effectiveness.

Color response is measured by Colorcheck, using the widely-available 24-patch X-Rite ColorChecker^®  and by Multicharts using the 24-patch ColorChecker, ColorChecker SG, IT8.7, QPcard, CMP DC 003, and custom "pie" charts. These charts may be included in scenes for white balance testing.

Color accuracy may be measured against standard chart reference values or CSV reference files that contain measured color values, which may be altered to reflect customer preferences.

Color accuracy is affected by the Bayer color filter array and by the signal processing and white balance algorithm in the camera or RAW converter. Flare light (veiling glare) in lenses tends to reduce color saturation. Multicharts can calculate a color correciton matrix.

Lens distortion

Lens distortion is an aberration that causes straight lines to curve near the edges of images. It can be troublesome for architectural photography and metrology (photographic applications involving measurement). The simplest approximation is the equation, r_u = r_d + kr_d³ where r_u is the undistorted and r_d is the distorted radius. Depending on the sign of k, it can be either "barrel" (shown on the right) or "pincushion."

Lens distortion and coefficients for correcting it are calculated in Distortion, which also includes 5th order and tangent/arctangent distortion models. SFRplus measures distortion in less detail, along with sharpness and several other factors. Distortion results are in the Image & Geometry display.

Distortion is worst in wide angle, telephoto, and zoom lenses. It often worse for close-up images than for images at a distance. It can be easily corrected in software. Picture Window Pro and PTLens have tools for removing it.

Light falloff (vignetting) and sensor nonuniformities

Light falloff (vignetting) darkens images near the corners. It can be significant with wide angle lenses. It is measured by Light Falloff (Uniformity) and Uniformity-Interactive (an interactive module that can work with the Imatest Image Sensor edition).

Light falloff tends to be worst in wide angle lenses. It often improves when lenses are stopped down. It can be easily corrected in software. Picture Window Pro, PTLens, and several other programs have tools for removing it. Because moderate amounts of light falloff can be pictorially pleasing, it's not always advisable to remove it completely.

Light Falloff (Uniformity) also measures other sensor nonuniformities, including color shading, stuck pixels, local sensitivity variations, spots (from dust), and noise. In Imatest Master it has a particularly rich set of displays.

Blemishes (visible sensor defects)

Blemishes are visible spots or marks in the image, caused by sensor defects or by dust in front of the sensor (typically separated by the Bayer, anti-aliasing, and infrared (IR) filters). They are extremely important in manufacturing. They are measured by Blemish Detect (Imatest Master-only).

Blemish Detect filters the image using a transfer function derived from the Contrast Sensitivity Function of the Human Visual System. Because of this, when filter parameters are set up properly, visible blemishes will be flagged and blemishes beneath the threshold of visibility will be ignored. This can significantly improve manufacturing yields.

Blemish Detect also measures hot and dead pixels.

Exposure accuracy and ISO Sensitivity

Exposure accuracy is not much of a problem with manually-adjustable cameras. You can usually determine it quickly (with the help of the histogram), and if you don't like it, you can change the exposure compensation or the way you meter.

But exposure accuracy can be an issue with fully automatic cameras and with video cameras where there is little or no opportunity for post-exposure tonal adjustment. Some even have exposure memory: exposure may change after very bright or dark objects appear in a scene.

Exposure accuracy can be measured by photographing a gray scale step chart such as the Kodak Q-13 or Q-14 in a scene and analyzing it with Stepchart, or photographing a GretagMacbeth ColorChecker and analyzing it with Colorcheck.

ISO Sensitivity (closely related to exposure accuracy) is a measure of a camera's sensitivity to light. Imatest modules that analyze step charts (which may be included in color charts) display two measures of sensitivity when the incident light in Lux is entered. Details in ISO Sensitivity and Exposure Index

Lateral chromatic aberration

Lateral chromatic aberration (LCA), also called "color fringing" is a lens aberration that causes colors to focus at different distances from the image center. It is most visible near corners of images. It is explained in Chromatic aberration.

LCA is measured by SFR, using edges at a distance from the image center.

LCA is worst with asymmetrical lenses, including ultrawides, true telephotos and zooms. It is strongly affected by demosaicing. It can be fully corrected in software prior to demosaicing, but only partially corrected afterwards. Picture Window Pro has a fairly effective transformation. In the future, information provided by Imatest (detailed LCA profiles) may improve the degree of correction.

Veiling glare (lens flare)

Veiling glare is stray light in lenses and optical systems caused by reflections between lens elements and the inside barrel of the lens. It predicts the severity of lens flare— image fogging (loss of shadow detail and color) as well as "ghost" images— that can occur in the presence of bright light sources in or near the field of view.

Veiling glare is measured by Stepchart using a standard Kodak Q-13 or Q-14 step chart beside a "black hole," i.e., a box lined with black behind a small opening, mounted on a large white board, as described in detail in Veiling glare.

Color moiré

Color moiré is artificial color banding that can appear in images with repetitive patterns of high spatial frequencies, like fabrics or picket fences. The example on the right is a detail of a shirt captured by the Canon Rebel XT with its excellent kit lens. The usual image wasn't used because it doesn't contain a repetitive pattern and because color moiré is difficult to simulate.

Color moiré is the result of aliasing (image energy above the Nyquist frequency) in image sensors that employ Bayer color filter arrays, as explained here. It is affected by lens sharpness, the anti-aliasing (lowpass) filter (which softens the image), and demosaicing software. It tends to be worst with the sharpest lenses.

Color moiré is measured by Rescharts Log Frequency, using a sine chart of logarithmically increasing spatial frequency.

Software artifacts: noise reduction and sharpening

Software (especially operations performed during RAW conversion) can cause significant visual artifacts, including oversharpening "halos" and loss of fine, low-contrast detail. These artifacts result from nonlinear signal processing (so-called because it varies with the signal). Images may be sharpened (MTF boosted) in the proximity of contrasty features like edges and blurred (lowpass filtered) in their absence. This generally improves measured performance (both sharpness and noise/Signal-to-Noise Ratio (SNR)), but it may result in a degradation of perceived image quality, for example, a "plasticy" cartoon-like appearance of skin even though edges are strongly sharpened. This loss of detail cannot be measured with SFR.

These artifacts can be measured by the Log F-Contrast module in Rescharts, which analyzes the chart shown on the right, which varies logarithmically in spatial frequency on the horizontal axis and in contrast on the vertical axis.

Data compression and transmission losses

Data compression and transmission losses can have a significant effect on image quality. The right side of the image on the left has been saved as a low quality JPEG. Banding, loss of low-contrast detail, and "waviness" near edges are visible.

A full analysis of data compression and transmission losses is not yet available in Imatest.

Some of these losses, especially the loss of low-contrast detail, can be analyzed with the Log F-contrast module in Rescharts. A more detailed analysis of compression losses is under development.

Quality factors for printers

Print Dmax

Dmax = -log₁₀(minimum print reflectivity) is a measure of the deepest black tone a printer/ink/paper combination can reproduce. It is an extremely important print quality factor. Prints with poor Dmax look pale and weak. Dmax = 1.7 is a good value for matte prints; 2.0 is a good value for glossy, semigloss, and luster prints. There have been reports that the new Epson Ultrachrome K3 printers have Dmax as high as 2.3 with Premium Luster paper. That would be outstanding.

Dmax is measured by Print Test, along with the Printer's tonal response curve and the color factors described below. Print test requires a decent flatbed scanner, and for best accuracy, a Step chart.

Dmax is affected by the printer, paper, and ink. The response curve is also affected by the ICC profile. Print Test can help with the selection of supplies and diagnosis of printing problems.

Print color gamut

Print color gamut is the range of colors a printer/ink/paper combination can reproduce. It is an important quality factor, though its importance may be somewhat overrated. (This statement is bound to generate controversy.) Relatively unsaturated colors such as skin tones dominate our impression of print quality. Such colors must be reproduced accurately. Gamut affects only highly saturated colors. Overall color response, especially for low to moderately saturated colors, is more important than gamut.

Print color gamut and overall color response are measured by Print Test, along with the density factors described above.

Print gamut is affected by the printer, paper, ink, and working color space. Color response is also affected by the ICC profile and rendering intent. Print Test can help with the selection of profiles, software settings, and the diagnosis of color problems.

Links

Bror Hultgren (ImagIntegration.com), an imaging scientist with 27 years experience at Polaroid, has developed algorithms and software that relates image quality factors (especially MTF, noise spectrum, white balance, and memory colors) to perceived image quality. Bror is available for consulting.

Direct Digital Image Capture of Cultural Heritage from RIT is a gold mine of information. Links to a number of reports on image quality targeted at museums and cultural institutions. The 78-page Final Project Report by Berns, Frey, Rosen, Smoyer and Taplin, July 2005, is probably the best summary unless you have time for Erin P (Murphy) Smoyer's 345 page Master's thesis (about twice the length of the average Ph.D. thesis).

The Research Library Group (RLG) has published an excellent series of articles, Guides to Quality in Visual Resource Imaging (2000). These articles are the predecessors to the above-mentioned RIT Direct Digital Image Capture work.

The European Broadcasting Union (EBU) has a library of technical papers (the EBU Tech 3000 series), some related to TV image quality, for example, T3249, Measurement and Analysis of the Performance of Film and Television Camera Lenses (1995), and T3281, Methods for the Measurement of Characteristics of CCD Cameras (1995).

Volker Gilbert has written an excellent French language description of Imatest. (PDF version)

SMIA (Standard Mobile Imaging Architecture), a consortium founded by Nokia and STMicroelectronics, has published a Camera Characterization Specification for image quality measurements in camera phones. To obtain the spec you must join SMIA. It's free; all you need to do is answer a brief questionnaire.

Paul van Walree has an excellent page on Optics, covering several sources of degradation.

The University of Texas Laboratory for Image & Video Engineering is doing some interesting work on image and video quality assessment, which approaches the problem using information theory, natural scene statistics, wavelets, etc. Challenging material!

Details of several Imatest algorithms are included in Appendix C, Video Acquisition Measurement Methods (pp. 91-125), of the Public Safety SoR (Statement of Requirements) volume II v 1.0, released by SAFECOM, prepared by ITS (a division of NTIA, U.S. Department of Commerce). No credit is given, but the style and illustrations will be recognizable.