Using Colorcheck

New in Imatest 3.7 Temporal noise can be calculated from two identical Colorchecker images. New in Imatest 3.5 ISO sensitivity is calculated when the incident lux level is entered. New in Imatest 3.1 Highly distorted images can be analyzed if automatic patch location is turned off. Several images can be combined (averaged) to facilitate fixed pattern noise measurement. New in Imatest 3.0 Options and Settings I in the Imatest main window allows you to select between traditional ROI selection, which automatically locates patches, and a Multicharts-like method, with adjustable patch locations that are visible during fine adjustment. This can be helpful with distorted images.

What Colorcheck does

Colorcheck analyzes images of the widely-available X-Rite^TM (formerly GretagMacbeth^TM ) ColorChecker^® (available from Adorama and other sources) for color accuracy, tonal response (using the six grayscale patches), noise or SNR (Signal-to-Noise ratio), and ISO sensitivity. Results for tonal response are similar to Stepchart. It is particularly useful for measuring the effectiveness of White Balance algorithms and settings under a variety of lighting conditions. The ColorChecker can be photographed in isolation or as part of a scene. Algorithms and equations can be found in the Colorcheck Appendix. The X-Rite ColorChecker is an 8x11″ inch chart consisting of 24 patches with 18 familiar colors and six grayscale levels with optical densities from 0.05 to 1.50; a range of 4.8 f-stops. The colors are not highly saturated. The quality of the chart is very high— each patch is printed separately using carefully controlled pigments. Patches have a smooth matte surface. It is available from ColorHQ, Adorama and many other dealers. A 2.25″x3.25″ mini ColorChecker is also available. It is useful for including in scenes for evaluating white balance.

Version comparison
Studio:	Tonal response and noise analyses for B&W patches; La*b* color error and Color analysis
Master:	User-supplied reference files; noise analysis for R, G, B, C, M, and Y patches; noise spectrum

A simulated ColorChecker is shown below for sRGB color space (the Windows/Internet standard). The optical densities (–log₁₀(reflectivity)) of the grayscale patches on the bottom row are shown in parentheses. One color, patch 18. Cyan, is out of the sRGB gamut; hence it can’t be reproduced perfectly on most monitors.

1. dark skin	2. light skin	3. blue sky	4. foliage	5. blue flower	6. bluish green
7.  orange	8. purplish blue	9. moderate red	10. purple	11. yellow green	12.  orange   yellow
13. blue	14. green	15. red	16.  yellow	17. magenta	18. cyan
19. white (.05)	20. neutral 8 (.23)	21. neutral 6.5 (.44)	22. neutral 5 (.70)	23. neutral 3.5 (1.05)	24. black (1.50)

What colors are the ColorChecker? Before 2005 the ColorChecker came with insufficient information. Grayscale densities (for the bottom row) were printed on the back of the chart. CIE xyY values (from the familiar 1931 xy chromaticity chart) were included, but these were for CIE illuminant C, whose 6774K color temperature is slightly different from standard 6550K color spaces. A set of RGB values with no specified color space were also included. They should not be used. A ColorChecker purchased in October 2005 had better information: sRGB values and CIE L*a*b* D50 (5000K) values. L*a*b* and RGB values for known color spaces are used in the Imatest analysis. (sRGB is the RGB standard color space for website display.) xy chromaticity values (from the 1931 chart) are far from perceptually uniform, i.e., distances between points on the xy-plane are not proportional to perceptible differences between colors. (Green is greatly exaggerated.) Color differences are better represented in the CIELAB color space, where L* is luminance, a* is color on a green-red scale, and b* color on a blue-yellow scale. A distance of 1 between L*a*b* values represents the minimum perceptible difference (just-noticeable difference, JND) between colors (for relatively unsaturated colors). For colors on the a*b* plane (neglecting L*), this distance is expressed by the equation, ΔC* = ( (a2*-a1*)2+ (b2*-b1*)2 )1/2  ;        where (…)1/2 denotes square root.   More generally, ΔE*ab = ( (L2*-L1*)2+ (a2*-a1*)2+ (b2*-b1*)2 )1/2      (includes L* differences) Although ΔC and ΔE*ab (which are both Euclidian distances) are widely used to quantify color differences, they are not as accurate as the CIE 1994 and CMC equations, which are discussed in detail in the Colorcheck Appendix. Danny Pascale/Babelcolor’s page on the Colorchecker contains everything you want to know about the chart. A great resource! Colorchecker reference values There are several sources, each slightly different. The values used by Imatest are shown below. The GretagMacbeth website (now X-Rite) no longer has downloads available. The very best source is Danny Pascale/BabelColor’s  RGB coordinates of the Macbeth ColorChecker is an outstanding resource that contains X-Rite’s and Pascale’s most recent Colorchecker values (2005). They are very close, with mean ΔE =2. Pascale presents ColorChecker values in xyY, L*a*b*, and a variety of color spaces, based on numerous measurements. He also presents procedures and equations for data conversion. The L*a*b* values are for D50 illumination. BruceLindbloom.com contains equations and tables for converting between various RGB color spaces as well as XYZ, xyY, and CIELAB. Click on Math. His ColorChecker values are based on accurate spectrophotometer measurements of a single sample. He has a useful ColorChecker calculator. Bruce Fraser, Creativepro.com. Calibrating Camera Raw in Photoshop CS contains Bruce’s ColorChecker L*a*b* values from his book, Real World Color Management. I not sure what reference color temperature he uses. Allen Pacheco has published a similar calibration technique. The ColorChecker reference can be selected in Colorcheck. At present you may choose between references 1 (X-Rite) and 3 (Pascale), or you can read L*a*b* or xyY files. The default is the X-Rite L*a*b* values. The grayscale values are derived from the definition of ISO speed for digital sensors, as described in the Colorcheck Appendix.

Photographing the target

• 

  Photograph the ColorChecker. The distance is not critical. There is no need to fill the frame with the ColorChecker image. Filling the frame may reduce accuracy if there is significant vignetting (light falloff due to the lens system, which is often quite large in wide-angle lenses).

  It may be useful to include other charts or scene elements that can affect white balance. A ColorChecker image width of 500 to 1500 pixels is sufficient for the Colorcheck noise analysis. That’s under half of the horizontal field of a 6 megapixel image. More pixels just slow the calculations. As few as 100 pixels is adequate for color and white balance (but not noise) analysis.

  The image on the right contains a ColorChecker and a Q-14 stepchart with a neutral gray surround (similar to the classic 18% gray card), which should help ensure a normal  (auto) exposure. Charts tend to be overexposed when the background is black and underexposed when it is white. Still life scenes can be used as backgrounds for testing a camera’s automatic white balance (AWB) and exposure algorithms.

  Lighting should be as even as possible. Lighting uniformity can be measured with an illuminance (Lux) meter. A variation of no more than ±5% is recommended. As little as possible should come from behind the camera— it can cause glare. An angle of incidence of about 20-45 degrees is ideal. For the noise analysis, the light should not emphasize the texture of the chart surface, which could affect noise measurements made in grayscale patches in the bottom row. More that one light source is recommended. If possible, the surroundings of the ColorChecker should be black or gray to minimize flare light. Middle gray (18-22% reflectance) is best for getting the correct exposure with auto-exposure cameras. Uneven lighting tilts the tops of the gray scale pixel plot (the upper left plot in the first figure, below). Lighting recommendations can be found in The Imatest Test Lab. If the objective of running ColorCheck is to analyze a camera’s white balance and color accuracy (and no noise analysis is required), the ColorChecker does not need to be photographed under ideal conditions. It can be part of a scene that challenges the camera’s white balance algorithm.

• If possible, the exposure should be within 1/4 f-stop of the correct value. L*a*b* color values are only accurate if exposure is correct. The exposure error is displayed in several Colorcheck figures. An explanation of exposure and grayscale levels can be found in the Colorcheck Appendix.
• Colorcheck measures the effectiveness of white balance algorithms; results are sensitive to the type of lighting. Outdoor, flash, incandescent, and fluorescent lighting may produce different results. Lighting and Color by John Beale illustrates some of the differences.
• 

  The ColorChecker may be photographed slightly out of focus to minimize errors in noise measurement due to texture in the patch surfaces. I emphasize slightly— the dark bands between the patches should remain distinct. The texture is quite low. If the lighting is reasonably diffuse (not a point source), noise from the surface texture should be minimal.

• WARNING  The ColorChecker image should not be too large! Images over 3000 pixels wide (6 megapixels total) can slow calculations because Imatest uses double precision math, which consumes 24 bytes per 3-color pixel. For example, an image from a 6 megapixel camera requires 144 megabytes (6x3x8 MB) in Matlab: enough to bog down computers with limited memory (256 MB or less). Be especially careful not to fill the frame with 8+ MB digital cameras.Save the image as a RAW file or maximum quality JPEG, then load it on your computer. If you are using a RAW converter, convert to JPEG (maximum quality), TIFF (without LZW compression, which is not supported), or PNG.  If you are using film, develop and scan it.Open Imatest. Click on the Colorcheck button in the Imatest main window. Open the input file using the dialog box. Imatest remembers the folder name of the last input file opened.

Running Colorcheck

• Click on the Colorcheck button on the left of the Imatest main window.
• Select the file to analyze.

• Multiple file selection Several files can be selected in Imatest Master using standard Windows techniques (shift-click or control-click). Depending on your response to the multi-image dialog box you can combine (average) several files or run them sequentially (batch mode). Combined (averaged) files  are useful for measuring fixed-pattern noise (at least 8 identical images captured at low ISO speed are recommended). The combined file can be saved. Its name will be the same as the first selected file with _comb_n appended, where n is the number of files combined. Batch mode  allows several files to be analyzed in sequence. There are three requirements. The files should (1) be in the same folder, (2) have the same pixel size, and (3) be framed identically. The input dialog box for the first run is the same as for standard non-batch runs. Additional runs use the same settings. Since no user input is required they can run extremely fast. If the order of the files in a batch runs is different from the selection order, click Settings, Options and Settings… (in the Imatest main window) and change the setting in Batch run order. The sequence may be affected by Windows Explorer settings. One caution: Imatest can slow dramatically on most computers when more than about twenty figures are open. For this reason we recommend checking the Close figures after save checkbox, and saving the results. This allows a large number of image files to be run in batch mode without danger of bogging down the computer.

• Crop the image. The method depends on settings that can be accessed by clicking Options and Settings I in the Imatest main window.

For Automatic patch detection: Original method, crop by dragging the cursor so a small amount of the dark boundaries at the edges of the Colorchecker image are visible. There is no harm if the boundaries are cut off as long as most of the patch area remains; precise cropping is not critical. Click outside the image for no crop. Correct orientation isn’t required; Imatest will rotate the image (in multiples of 90 degrees) to the correct orientation. This method works well for well-behaved images with little perspective or lens distortion. The image shown above is representative of a good crop of a well-behaved image.

For Display patch squares in fine adjustment (similar to Multicharts), the initial crop is similar to the original method: drag the cursor so a small amount of the dark boundaries are visible. You’ll be able to to refine the boundaries in the fine adjustment window, which is described in detail in Multicharts. For highly distorted images like the one shown below, you’ll need to adjust the distortion slider and reduce the fill factor below the default value of 0.7. Since this image has geometrical distortion as well as strong barrel distortion, the corner positions had to be adjusted with care.

Fine adjustment of a highly distorted image. Corners, Fill factor,
and Distortion were carefully adjusted (they interact somewhat).

• The Imatest Colorcheck dialog box appears if Express mode has not been selected. It allows you change Title, Colorchecker reference, Color space, Noise display, Color error display, and several other parameters from their default values. It also contains news about the latest updates.

Title: defaults to the file name. You can leave or modify it. Figure selection is in the Plot box on the right.

Colorcheck input dialog box

Colorchecker reference selects the ideal ColorChecker L*a*b* and RGB values. You can choose between any of several standard values (based on measurements) or you can enter your own measured values from CSV files. The Old / New radio buttons affect the X-Rite and Babelcolor reference values (described below). Old selects the original reference values, used prior to the release of Imatest 1.6.17 in July 2006. New (recommended) uses more accurate recent measurements. For most color patches, the difference between the old and new values is small.

Standard values: X-Rite (default): L*a*b* values provided by X-Rite/GretagMacbeth. Old: Values (formerly on their website) measured with Illuminant C (6774K). New: Values supplied courtesy of X-Rite in 2006 (measured in 2005). Either D50 (5000K) or D65 (6500K) values are used, depending on the white point of the selected color space (below). This is the default.Babelcolor: Values measured by Danny Pascale of Babelcolor. If New is selected, Pascale’s D50 values are transformed to D65 using the Bradford transformation for color spaces with a D65 white point. Another excellent choice, close to the X-Rite values. These values, as well as the old and new X-Rite values, are described in Danny Pascale’s RGB coordinates of the MacBeth ColorChecker, which is strongly recommended for all Colorcheck users. Danes-Picta BST4D: A knockoff of the Colorchecker (same geometry with somewhat different colors) from Danes-Picta in the Czech Republic (on their Digital Imaging page). Enter your own values:  These can be (a) measured on your own instruments or (b) values you have determined to be “pleasing” that you wish to use as design targets. (Remember, accurate color is not necessarily pictorially pleasing. Designers of consumer cameras often aim for “pleasing” colors, which typically involves increasing saturation in foliage, skies, and skin.) LAB data file: Read a file containing L*a*b* data. This is the best approach if you have spectrophotometer measurements for your individual chart. A dialog box appears for entering the filename. The file must consist of 24 lines with L*, a*, b* values on each line separated by spaces, commas (,), or semicolons (;). Example (first 3 lines of 24): 38.08, 12.09, 14.39 66.38, 13.22, 17.14 51.06, 0.38, -22.06 (The commas are optional if spaces are present & vice-versa.) If you have an Excel .CSV file with extra rows or columns, you can easily edit it Excel by selecting the key region (3 columns, 24 rows), copying it to a new file, and saving it in .CSV format. L*a*b* data is preferred to xyY data below because it is independent of white point color temperature, hence less error-prone. xyY 5000K data file: Read a file containing xyY data with a white point of 5000K (D50). Procedure and format is the same is the LAB data file, above. xyY 6500K data file: Read a file containing xyY data with a white point of 6500K (D65). Procedure and format is the same is the LAB data file, above. Last (selected) file: Always appears when the previous selection was an LAB or xyY data file, above. If you selected LAB data file or one of the xyY data files, a dialog box will appear after you press OK that enables you to navigate to the reference file. The Colorchecker reference is displayed at the bottom of the L*a*b* color error figure.

Noise display: Pixel noise or SNR  selects the noise display for the lower-left plot of the first figure (noise/SNR for grayscale row 4) and the upper-right plot of the second figure. (noise/SNR for BGRYMC row 3). The effects of gradual illumination nonuniformities have been removed from the results. All displays are derived from Noise in pixels (the third selection below). The five display types are identical to the types in the lower noise detail plot in Stepchart. In the notation below N_i is RMS noise and S_i is signal (pixel) level for patch i.

Noise (%) normalized to White – Black (Zone 19 – 24)	100% * (Noise in pixels) divided by (White patch (19) pixel level − Black patch (24) pixel level) = 100% N/(SWHITE−SBLACK ). Useful because it references the noise to the scene: noise performance is not affected by camera contrast.
Noise (%) normalized to 255 pixels (max of 100)	100% * (Noise in pixels) / 255 = Ni/2.55. Values 0-100. This noise measurement will be worse for higher contrast cameras. (It is affected by the gamma encoding.)
Noise in pixels (maximum of 255)	Noise in pixels ( Ni ). Values 0-255.
Pixel S/N (Signal in patch/RMS noise) dimensionless	(Signal (pixel level)) / Noise = Si/Ni for each patch. Dimensionless.
Pixel SNR (dB) (20*log10(S/N))	20 Log10(Signal/Noise) for each patch = 20 log10( Si/Ni ). Units of dB (decibels). Doubling S/N increased dB measurement by 6.02.
	For this selection, SNR_BW is also displayed. SNR_BW is an average SNR based on White-Black patches (Zone 19 – Zone 24; density difference = 1.45). SNR_BW = 20 log10((SWHITE-SBLACK )/Nmid ), where Nmid is the noise in patch 22 (the 4th patch in row 4; middle gray; nominal density = 0.7).

Color space: You can select among the following. Danny Pascale’s A Review of RGB Color Spaces is recommended for readers interested in an in-depth explanation of color spaces.

sRGB	The default space of Windows and the Internet. Limited color gamut based on typical CRT phosphors. Gamma = 2.2 (approximately), White point = 6500K (D65).
Adobe RGB (1998)	Medium gamut, with stronger greens than sRGB. Often recommended for high quality printed output. Gamma = 2.2, White point = 6500K (D65).
Wide Gamut RGB	Extremely wide gamut with primaries on the spectral locus at 450, 525, and 700 microns. One of the color spaces supported by the Canon DPP RAW converter. 48-bit color files are recommended with wide gamut spaces: banding can be a problem with 24-bit color. Gamma = 2.2, White point = 5000K (D50).
ProPhoto RGB	Extremely wide gamut. Gamma = 1.8, White point = 5000K (D50). Described in RIMM/ROMM RGB Color Encodings by Spaulding, Woolfe and Giorgianni.
Apple RGB	Small gamut. Used by Apple. Gamma = 1.8, White point = 6500K (D65).
ColorMatch RGB	Small gamut. Used by Apple. Gamma = 1.8, White point = 5000K (D50).

Color error display: You can select between

• Standard ΔC and ΔE*_ab (CIE 1976) measurements, CIE 1994 measurements (somewhat more accurate), and CMC mesurements (widely used in the textile industry) color difference formulas.
• CIEDE2000 measurements ( ΔC₀₀ and ΔE₀₀ ). The emerging standard. See Gaurav Sharma’s CIEDE2000 Color-Difference Formula page.

The CIE-94, CMC, and CIEDE2000 mreasurments reflect the eye’s reduced sensitivity to hue and saturation changes for highly saturated colors. Mean CIE-94 numbers are typically around half of the standard measurements; CMC measurements are between the two. CIEDE2000 measurements will be become more familiar as time passes. Patch for noise spectrum (Imatest Master only): You can select any of the patches in the bottom two rows for calculating the noise spectrum. (Row 3 contains B, G, R, Y, M, and C primaries; row 4 contains grayscale values.) The spectra for the Y (luminance), R, G, and B channels is displayed. The third patch in row 4 (middle gray; density = 0.44) is the default. a*b* color error: The upper-right of the a*b* color error plot displays mean color errors (ΔC… and ΔE…) and either the standard deviation of the color error (σ) or the maximum color error (max), selected in this box.

Incident lux (for ISO sensitivity calculations)  When a positive value of incident light level (not blank or zero) in lux is entered in this box, ISO sensitivity is calculated and displayed in the Stepchart noise detail figure. More details are on the ISO Sensitivity and Exposure Index page.

Output

The example was photographed with the Canon EOS-10D at ISO 400, and RAW converted with Canon FVU using automatic color balance.

The first figure: Gray scale tonal response and noise

shows graphic results derived from the grayscale patches in the bottom row, including tonal response and noise. Selected EXIF data is displayed, if available.

The upper left plot  is the density response of the colorchecker (gray squares). It includes the first and second order fits (dashed blue and green lines). The horizontal axis is Log Exposure (minus the target density), printed on the back of the ColorChecker. Stepchart provides a more detailed density response curve. The exposure error is shown in pale red if it is less than 0.25 (the maximum recommended error) and bold red otherwise.

The upper right plot shows the noise in the third colorchecker row, which contains the most strongly colored patches: Blue, Green, Red, Yellow, Magenta, and Cyan. In certain cameras noise may vary with the color. Problems may be apparent that aren’t visible in the gray patches. Note that the x-axis scale (log exposure) is reversed from the plots on the left.

The lower left plot shows the RMS noise or SNR (signal-to-noise ratio) for each patch: R, G, B, and Y (luminance). Several display options are described above. The selected display in the above figure is Noise (%) normalized to White – Black (Zone 19 – 24), i.e., RMS noise is expressed in percentage of the difference between the white and black patches.The average R, G, B, and Y noise for the gray zones (2-5) is also reported.

The lower right region displays EXIF data, if available.

The bottom left plot has several display options, listed above. Pixel SNR (dB) is shown on the left.

SNR (dB) = 20 log₁₀(S_i/N_i), where S_i is the signal (mean pixel level) of patch i and N_i is the noise (standard deviation of the pixel level, with slow variations removed) of patch i.

SNR_BW is an average SNR based on White – Black patches (patch 19 – patch 24; density difference = 1.45)

SNR_BW = 20 log₁₀((S_WHITE-S_BLACK )/N_mid ), where N_mid is the noise in patch 22 (the 4th patch in row 4; middle gray; closest to nominal chart density = 0.7).

Two precautions when working with figures

Too many open Figures Figures can proliferate if you do a number of runs, especially SFR runs with multiple regions, and system performance suffers if too many Figures are open. You will need to manage them. Figures can be closed individually by clicking X on the upper right of the Figure or by any of the usual Windows techniques. You can close them all by clicking Close figures in the Imatest main window. Checking Close figures after save is recommended for batch runs to prevent buildup of open figures.

Clicking on Figures during calculations can confuse Matlab. Plots can appear on the wrong figure (usually distorted) or disappear altogether. Wait until all calculations are complete— until the Save or Imatest main window appears— before clicking on any Figures.

The second figure: Noise detail (Imatest Master only)

shows the density response, noise in f-stops (a relative measurement that corresponds to the workings of the eye), noise for the third Colorchecker row, which contains primary colors, and the noise spectrum of the selected patch.

The upper left plot  is the density response of the colorchecker (gray squares). It includes the first and second order fits (dashed blue and green lines). The horizontal axis is Log Exposure (minus the target density), printed on the back of the ColorChecker. Stepchart provides a more detailed density response curve. The exposure error is shown in pale red if it is less than 0.25 (the maximum recommended error) and bold red otherwise. ISO Sensitivity is displayed if indicent lux has been entered.

The upper right plot shows noise or SNR in the third Colorchecker row, which contains the most strongly colored patches: Blue, Green, Red, Yellow, Magenta, and Cyan. Several display options are described above. In certain cameras noise may vary with the color. Problems may be apparent that aren’t visible in the gray patches.

The third figure: a*b* Color error

illustrates the color error in the device-independent CIELAB color space, where a* is the horizontal axis and b* is the vertical axis. The squares are the ideal (a*, b*) ColorChecker values, set by the ColorChecker reference setting, above. The circles are the (a*, b*) values of the actual measured ColorChecker. The numbers near the squares or circles correspond to the numbers of the ColorChecker patches: 1-6 are in the top row, 7-12 are in the second row, and 13-18 are in the third row. The numbers for patches 19-24 (the bottom row) are omitted because their (a*, b*) values cluster around (0, 0).

The background of the chart shows expected colors (in monitor sRGB color space) for L* around 0.9. It presents a reasonable picture of the hues associated with a* and b* (though they shift somewhat with L*). The light gray curve is the boundary of the color space (sRGB). There is significant scatter in the results because the luminance (L*) of the color patches (not displayed) varies considerably. This 2-Dimensional figure cannot display L*; a limitation is overcome in Multicharts, which has a rotatable 3D L*a*b* display. CIELAB (often shortened to LAB) was designed to be perceptually uniform, meaning that the perceived difference between colors is approximately proportional to the Euclidian distance between them, ΔE*_ab (which includes luminance L*) or ΔC* (color only; omitting L*).

ΔE*_ab = ( (L₂*-L₁*)²+ (a₂*-a₁*)²+ (b₂*-b₁*)² )^1/2;    ΔC* =  ( (a₂*-a₁*)²+ (b₂*-b₁*)² )^1/2

The smallest perceptible difference corresponds roughly to ΔE*_ab = ΔC = 1. (Gaurav Sharma uses 2.3 in the Digital Color Imaging Handbook, p. 31.) CIELAB is far from perfect in this regard. ΔE*_ab for the minimum perceptible color difference increases with chroma (i.e., distance from the origin, (a*_i² + b*_i² )^1/2. More accurate color difference difference formulas, ΔE*₉₄, ΔE*_CMC, and ΔE₀₀, can be selected for display and are included in the CSV and XML output files. If you report any of the more accurate formulas, remember that they are less familiar than plain old ΔE*_ab, so to avoid misunderstanding, be sure to clearly indicate which formula you are using. Mean values of ΔE*₉₄ are about half of ΔE*_ab . Mean values of ΔE*_CMC are between the two. The CIEDE2000 color difference formula, ΔE₀₀, is the emerging standard. It’s the most accurate, but it’s very complex and still relatively unfamiliar. It’s the best choice in the long run.  ΔE*₉₄, ΔE*_CMC, and ΔE₀₀ are discussed in the Colorcheck Appendix.Several values are reported on the upper right of the figure.

• Mean camera chroma (%) is the average chroma of camera colors divided by the average chroma of the ideal Colorchecker colors, expressed as a percentage. The chroma of an individual color is its distance from the origin, C = (a_i*²+b_i*²)^1/2. Chroma is often referred to as saturation because more chromatic colors appear to be more saturated. The mean chroma is,
  

  Chroma = 100% mean((a*_{i_meas}² + b*_{i_meas}²)^1/2 ) / mean((a_{i_ideal}*² + b_{i_ideal}*²)^1/2 ) ;

  i_meas denotes measured values of patch i; i_ideal denotes ideal ColorChecker values ;

  i ≤ 1 ≤ 18   (the first three rows of the Colorchecker) ;  mean(x) = Σx_i / n   for n = 18 values of x.

Chroma is boosted when Chroma > 100%. Chroma is affected by lens quality (flare light in poor lenses decreases it) and signal processing. Many cameras and most RAW converters have adjustments for chroma (usually labelled Saturation).

Chroma is not a strong indicator of image quality because it is often altered during RAW conversion and can be easily adjusted in image editors. Images out of cameras often have boosted saturation to make them more vivid (120% is not uncommon in compact digital cameras), but boosted saturation can cause a loss of detail in highly saturated objects. Saturation over 120% should be regarded as excessive.

• Color errors ΔC and ΔE

ΔC is color error with luminance difference omitted, i.e., color only. ΔE includes luminance difference. Imatest displays two values of ΔC: with and without correction for the saturation boost (or cut) in the camera, discussed below. You may select between three models of ΔC and ΔE for the display (all are included in the CSV/XML output files).

• Standard ΔC and ΔE*_ab measurements, CIE 1994 measurements (somewhat more accurate), and CMC mesurements (widely used in the textile industry) color difference formulas
• CIEDE2000 measurements ( ΔC₀₀ and ΔE₀₀ ). The emerging standard. See Gaurav Sharma’s CIEDE2000 Color-Difference Formula page.

ΔC or ΔE of around 1 (2.3 according to Sharma) correspond roughly to a just noticeable difference (JND) between colors. The three color difference models are described in the Colorcheck Appendix. The CIE 1994 and CMC formulas are more accurate than the standard ΔE formula; the CIEDE2000 formulas are the most accurate.

• Color error ΔC is calculated with and without chroma (saturation) correction. Saturation correction involves normalizing the measured colors so their average saturation is the same as the ideal average saturation. This is done by multiplying a*, b*, and C* by 100/Sat. The corrected values, denoted by i_corr, are substituted for the measured values in the ΔC calculation.

a*_{i_corr} = 100 a*_{i_meas} / Sat ;    b*_{i_corr} = 100 b*_{i_meas} / Sat

C_{i_corr} =(a*_{i_corr}² + b*_{i_corr}² )^1/2

ΔC_{i_corr} = |C_{i_corr} – C_{i_ideal} | = ( (a*_{i_corr} – a*_{i_ideal} )² + (b*_{i_corr} – b*_{i_ideal} )² )^1/2      (Standard (CIE 1976) formula)  

This normalization removes the effects of saturation boost, resulting in a lower mean color error, mean(ΔC_{i_corr}). It is applied because saturation can be easily adjusted by turning a (digital) knob in an image editor. The remaining error, mean(ΔC_{i_corr} ), is a useful indicator of color quality because it cannot be easily removed and it is not a function of mean saturation. It is displayed in two ways on Figure 3:

• The mean and root mean square (RMS) or maximum color errors are displayed.

mean(x) = Σx_i / n   for n values of x.
RMS(x) = σ(x) =  ( Σx_i² / n )^1/2   for n values of x.

x is ΔC_i and ΔC_{i_corr} for Colorchecker patches 1-18 (the top three rows).

mean color error (uncorrected) = mean( ΔC_i )

mean color error (corrected) = mean( ΔC_{i_corr} )

The root mean square (RMS) error, denoted by sigma (σ), gives greater weight to large errors; it may therefore be a better overall indicator of color accuracy. You can select between σ and maximum in the input dialog box.

• Summary: Figure 3 displays the mean and RMS values of the color errors, using either standard (CIE 1976), CIE 1994, or CMC color difference formulas. (The latter two are somewhat more accurate.) CIE 1994 and CMCreflect the eye’s reduced sensitivity to hue and especially saturation changes for highly saturated colors. Hence they are generally lower. The results are summarized below.

ΔC, ΔE display
One of these four columns is selected.				The mean and RMS values of all three rows are displayed.
ΔC*ab sat. corr.	ΔC*94 sat. corr.	ΔC*CMC sat. corr.	ΔCoo sat. corr.	Color difference, omitting luminance, corrected for average saturation.
ΔC*abuncorr.	ΔC*94 uncorr.	ΔC*CMC uncorr.	ΔCoo uncorr.	Color difference, omitting luminance; no saturation correction.
ΔE*ab	ΔE*94	ΔE*CMC	ΔEoo	Color difference, including luminance; no saturation correction.

It’s difficult to assess these numbers until a large database of test results is available. For now I would characterize the values of ΔC*_ab(corr) mean = 5.66 and σ = 6.73 for the EOS-10D (above) as very good to excellent.

When you interpret the results, keep the following in mind. Camera manufacturers don’t necessarily aim for accurate color reproduction, which can appear flat and dull. They recognize that there is a difference between accurate and pleasing color: most people prefer deep blue skies, saturated green foliage, and warm, slightly saturated skin tones. Ever mindful of the bottom line, they aim to please. Hence they often boost saturation, especially in blues, greens, and skin tones.

See the Links, below, for details on CIELAB color space.

The fourth figure: Color analysis

is an image of the ColorChecker that allows you to compare the actual and ideal values. It also displays White Balance error using the bottom row (the grayscale patches).

The outer portion (1) of each patch contains the ColorChecker image as photographed,

The inner portion contains the ideal ColorChecker values corrected for

exposure (2) and uncorrected (3).

1. The outer region, is the patch as photographed. This corresponds to the circles in the L*a*b* color error plot, above.
2. The square in the center, is the ideal value of the patch, corrected for the luminance of the photographed chart. The correction is derived from the second order fit to the gray areas, described above. The average luminance of zones 1 and 2 should be close— zone 2 may be darker in some patches and lighter in others. Zones 2 and 3 correspond to the small squares in the L*a*b* color error plot, above. The exposure error is shown on the same line as the date and color space. Best results are obtained if it is less than 0.25 f-stops.
3. The small rectangle to the right of the central square, is the ideal value of the patch, with no luminance correction. The luminance of zone 2 will be consistently lighter or darker than zone 3 for all patches, depending on the exposure.

In the above example, the ColorChecker image 1 is slightly underexposed; it is darker than the ideal uncorrected color 3. It is also darker than the corrected ideal color 2. This is a characteristic of this camera: it tends to darken oranges and lighten blues and cyans. This can be seen in the above right ColorChecker image, which is slightly overexposed. This image of the orange-yellow patch (row 2, column 6) has about the same luminance as the uncorrected ideal color 3, but is darker than the corrected ideal color. The uncorrected ideal value 3 in the two orange-yellow images above (the example on the left and the full chart on the right) are the same, but appear to be quite different because of the interaction of colors in the eye— an effect that should be familiar to artists and photographers.

This image provides a clear visual indication of color accuracy. The L*a*b* color error plot provides a quantitative indication.

The bottom of the third figure illustrates the White Balance error, which is quantified in three ways. (1) Saturation S in the HSV color model (burgundy). S can take on values between 0 for a perfect neutral gray and 1 for a totally saturated color. (2) Correlated color temperature error in Degrees Kelvin (blue), calculated from Color Science: Concepts and Methods, Quantitative Data and Formulae, by Günther Wyszecki and W. S. Stiles, pp. 2224-228. (2) Color temperature in Mireds (micro-reciprocal degrees) (blue), where Mireds = 10⁶/(Degrees Kelvin) is a more perceptually uniform measure.

White Balance error tends to be most visible in the gray patches (2-5 in the bottom row). It is barely visible for S < 0.02. It is quite serious for S > 0.10, particularly for the lighter gray patches.

The image at the bottom shows  the bottom row (the grayscale patches) with exaggerated white balance error, calculated by boosting saturation S using the curve on the left, keeping H and V constant. Low values of saturation are boosted by a factor of 4; the boost decreases as saturation increases. This image shows the White Balance error much more clearly than the unexaggerated chart. Keep it in mind that this image is worse than reality; small White Balance errors may not be obvious in actual photographs, especially in the vicinity of strongly saturated colors.

The fifth figure: Temporal noise

Temporal (time-varying) noise is the random difference between otherwise identical images. It is measured as the noise (standard deviation) of the difference two identical test chart images divided by the square root of 2 (1.414).

N_temporal = Noise(Image₁-Image₂) / 1.414

Subtracting two images removes fixed pattern noise. The square root of 2 is needed because noise powers add, even though image pixels are subtracted. Dividing by the square root of 2 scales temporal noise to be the same as noise measured in an individual image.

To measure temporal noise, select two images for analysis. In the dialog box shown on the right, select Read two files for measuring temporal noise. The analysis proceeds normally (for the first file), but a fourth figure, displayed if 5. Temporal noise is checked in the input dialog box Plot area, contains the temporal noise analysis. This figure is intended to be self-contained, hence repeats some features of previous plots.

Temporal noise display

• The upper-left plot displays the density response, similar to the upper-left plot of Figure 2.
• The lower-left plot (RMS noise (f-stops)) displays scene-referenced grayscale noise or SNR using the same scale as the lower-left plot of Figure 2. In these plots, noise (or SNR) for a single image is shown as a pale curve.
• The upper-right plot (Pixel SNR (dB) for the third row (for the BGRYMC patches)) displays noise or SNR for the color patches with the scale set by Noise display in the input dialog box. It is similar to the upper-right plot of Figure 2.
• The lower-right plot (Grayscale SNR dB) displays grayscale pixel noise with the scale set by Noise display in the input dialog box (scaled the same as the lower-left plot of Figure 1 with a different x-axis).

Saving the results

 

When the Colorcheck calculations are complete, the Save Colorcheck results? dialog box appears. It allows you to choose which results to save and where to save them. The default is subdirectory Results in the data file directory. You can change to another existing directory, but new results directories must be created outside of Imatest— using a utility such as Windows Explorer. (This is a limitation of this version of Matlab.) The selections are saved between runs. You can examine the output figures before you check or uncheck the boxes. After you click on Yes or No, the Imatest main window reappears. Figures, CSV, and XML data are saved in files whose names consist of a root file name with a suffix and extension. The root file name defaults to the image file name, but can be changed using the Results root file name box. Be sure to press enter. The .CSV summary file contains Excel .CSV output for the tone level, noise, and color calculations. It also contains some additional data, including EXIF data for JPEG files. The XML output file contains similar results.

Checking Close figures after save is recommended for batch runs to prevent buildup of open figures.

Summary .CSV and XML files

An optional .CSV (comma-separated variable) output file contains results for Colorcheck. Its name is [root name]_summary.csv. An example is Canon_EOS10d_ColorCheck_lt_small_summary.csv.

The format is as follows:

Module	SFR, SFR multi-ROI, Colorcheck, or Stepchart.
File	File name (title).
Run date	mm/dd/yyyy hh:mm of run.
(blank line)
Tables	Tables are separated by blank lines.
	The first table contains ideal and measured pixel levels and densities. Zone is the chart patch (19-24 for grays). Gray is 1-6, corresponding to patches 19-24. Pixel is the measured pixel level (maximum = 255). Pixel/255 is the normalized measured pixel level. Px/255 ideal is the ideal normalized pixel level. Log(exp) is (-)patch density. Log(px/255) is the log exposure. WB Err Deg and WB Err Mired are white balance errors in degrees and mireds (micro reciprocal degrees), respectively.
	The second table contains noise measurements for the R, G, B, and Y channels for the the patches in the bottom two rows. Noise is measured in both % of the maximum density patch (19) of the Colorchecker (density = 1.5) and f-stops.
	The third table contains S/N and SNR (dB) measurements, described above.
	The fourth table contains ideal and measured RGB and La*b* values for the 24 patches.
	The fifth table contains several color error measurements (differences with the ideal Colorchecker values) for the 24 patches: ΔC (color difference, neglecting luminance L), ΔE (total difference, including L) using measured (camera) values with and without saturation correction. Details in the Colorcheck Appendix.
(blank line)
Additional data	The first entry is the name of the data; the second (and additional) entries contain the value. Names are generally self-explanatory (similar to the figures).
(blank line)
EXIF data	Displayed if available. EXIF data is image file metadata that contains important camera, lens, and exposure settings. By default, Imatest uses a small program, jhead.exe, which works only with JPEG files, to read EXIF data. To read detailed EXIF data from all image file formats, we recommend downloading, installing, and selecting Phil Harvey’s ExifTool, as described here.

This format is similar for all modules. Data is largely self-explanatory. Enhancements to .CSV files will be listed in the Change Log.The optional XML output file contains results similar to the .CSV file. Its contents are largely self-explanatory. It is stored in [root name].xml. XML output will be used for extensions to Imatest, such as databases, to be written by Imatest and third parties. Contact us if you have questions or suggestions.

Links

RGB coordinates of the Macbeth ColorChecker by Danny Pascale, BabelColor, 2003 (PDF). Excellent survey of color spaces and techniques for calculating the RGB values of the ColorChecker. And don’t forget his other Colorchecker page

Bruce Lindbloom  Outstanding site with equations for converting between color spaces and models. Slightly different L*a*b* values from Pascale.

Earl F. Glynn (EFG) has an excellent description of the ColorChecker. His Computer Lab and Reference Library are a goldmine of information about color, image processing, and related topics.

Lighting and Color by John Beale illustrates how different light sources affect Colorcheck results.

Munsell Color Science Laboratory (at RIT) has an interesting page of color standards data available for download, including the spectral reflectivities of the Colorchecker measured by Noboru Ohta.

CIE Fundamentals for Color Measurements by Yoshi Ohno, Optical Technology Division, NIST.  An excellent review of CIE colorimetry from an IS&T technical conference. Describes the CIE 1931 (x, y) and 1976 (u’v’) diagrams.

Digital Color Imaging by Gaurav Sharma  A comprehensive 1997 review paper that was updated and turned into the first chapter of the Digital Color Imaging Handbook.

Color management equipment/supplies

ColorHQ.com  |  ColorManaged.com

GretagMacbeth^TM and ColorChecker^® are tradmarks of X-Rite, Incorporated, which is not affiliated with Imatest. ColorChecker L*a*b* values are supplied courtesy of X-Rite.