Documentation

Multitest: fixed color & grayscale analysis

Analyze color and grayscale test charts
in a fixed batch-capable module

Note: Multitest was briefly known as ColorTest in Imatest 3.10. It has been changed because it is much more general and powerful,
incorporating all calculations in Multicharts, Colorcheck, and Stepchart.

News– Imatest 4.0: Automatic region detection is available for the Colorchecker and 36-patch Dynamic Range charts, set using the ROI Options button at the bottom of the Imatest main window or the Multitest settings window. ISO 15739 Visual Noise has been added.

Introduction

Imatest™ Multitest (not available in Imatest Studio) analyzes images of a large variety of test charts for color accuracy, tonal response, noise, SNR (Signal-to-Noise Ratio) and ISO sensitivity using fixed interface that can run batches of images. It can measure white balance and color response for a wide range of lighting conditions and scenes. It can also display the tonal response of monochrome charts (or monochrome sections of color charts). A color correction matrix can be calculated.

Multitest can analyze all the charts supported by Multicharts and can produce most of the results plots. It can also incorporates all important functions of the Colorcheck and Stepchart modules: it is similar in operation, but able to analyze more charts and display more results.

Major test charts supported by Multitest
X-Rite ColorChecker (24-patch)24-patch
Colorchecker
 
IT8 target (from Wolf Faust)IT8.7
 
CMP DigitaL TargeT 003
CMP DT 003
 
ColorChecker SG
Colorchecker SG
 
 

dynamic_35_E_200W
36-Patch DR chart

 
Q13 grayscale step chart
Q-13/Q-14
and other
grayscale
stepcharts
    ColorGauge_120W
ColorGauge
(Micro, Nano, etc.)
 
 
ISO-15739 Noise target crop
ISO-15739
 

General mxn
charts (L*a*b*
reference file
required)
 
QPcard_202_180W
QPcard 202/203
 
ST-52 (ISO-14524) crop
ISO-14524
Also: SFRplus 20-patch color, SMPTE/Hale Color Bars (6 patches), TE226 (9×5) chart

A full list of supported charts can be found on Multicharts– Supported charts.

Getting started

Photograph or scan the chart, taking care to illuminate it evenly (±5%) and to avoid glare, which can be problematic in charts with semigloss (SG) and glossy surfaces such as the reflective IT8.7 and ColorChecker SG. Glare can be especially challenging to control with wide angle lenses. To minimize the effects of light falloff in lenses, the chart should typically not fill the image field.

A low-cost lighting setup is described here. For testing white balance, you can photograph the chart in a scene under a variety of lighting conditions. Small versions of the ColorChecker (the Passport, the SpyderCheckr, or the ColorGauge charts) are especially suitable for this purpose. Some rectangular charts may be rotated by multiples of ±90° if they are geometrically symmetrical (these include the two Colorcheckers and the Kodak/Tiffen Q-13/Q-14, but not the IT8.7 (where top and bottom are different).

Before you run Multitest you should select the chart type to analyze. Click on the Setup button, just to the right of the Multitest button in the Imatest main window (shown on the right) or click on Fixed Modules, Multitest settings. These open the Multitest settings window, shown below.

The Chart type dropdown menu (in the Multitest settings window, shown on the right) contains all available chart settings.

After a selection has been made it is saved and displayed in the Multitest button, as shown above-right.

Additional settings can be changed at this time or after an image file has been read and the Region of Interest (ROI) selected (unless you’ve selected Express mode)— but since Chart Type always affects the next run, a selection should be made before a new chart type is run.

Chart selection dropdown menu

Chart type should be selected prior to running Multitest.

  multitest_button
  Chart type (for next Multitest run)
multitest_chart_type

Automatic region detection

This is also a good time to select automatic or manual region detection, located at the bottom of the Multitest settings window and enabled for charts that allow the automatic option (initially Colorcheck and the 36-patch Dynamic Range chart, with more to come).

multitest_autoroiManual/Automatic ROI selection, located at the bottom of the Multitest settings window

To start Multitest, run Imatest, then click the Multitest button. A window will open for selecting the file (or batch of files). If you have selected manual ROI detection, after you make the file selection, Imatest will check to see if the image(s) are the same size as recently-run images (it saves several sizes). If it is the same size you’ll be asked if you want to repeat the same Region of Interest (ROI) selection:

ColorTest_SQ_roi_repeat
ROI repeat window (appears when image is the same size as a previously-run image). Manual ROI detection.

If the image is not the same size as a previous image or if you click No, select new ROI(s) , a window will open asking you to select the region. Do this by drawing a rectangle from one to another corner of the chart you want to analyze. Don’t worry about getting the region exactly right— you’ll be able to refine it.

ColorTest_SQ_region_selection
Initial (coarse) region selection. Draw a rectangle from corner to corner of the chart.
Image from Imaging-resource.com (the Multi-Target from the Samples page for individual camera reviews).

Refine the selection in the fine selection window that opens after you make the coarse selection. The inner squares are different for different charts. (Note: this window will appear if you’ve selected Automatic ROI detection with confirmation with one of the charts that support automatic detection.)

ColorTest_SQ_fine_adjust
Fine region selection, shown for the ColorChecker SG

  • The entire ROI can be moved ( ^ > v < , upper-left).
  • The top, bottom, left, or right sides can be moved ( T^ Tv R> R< B^ Bv L> L< , middle-left).
  • Corners can be moved individually (any of the 8 buttons in each of the four corners, middle-left).
  • Pixel values for the four corners can be entered (X(tl), Y(br), …, where tl = top-left, br = bottom right, etc.). The origin is the upper-left. Be sure to press the Enter key after entering a value.
  • You can choose between Fine, Coarse, or Extra Coarse (XC-25) movement (1, 5, or 25 pixels of movement per click).
  • The display can be zoomed out or in.
  • The display can be lightened to view the dark regions more clearly.
  • For some charts (those with relatively large patches) a Fill factor (linear) slider to the left of the Distortion slider specifies the relative size of the squares. Its default value is 0.5. Smaller values can be used where lens distortion or pattern alignment makes alignment difficult.
  • The Distortion slider allows highly distorted images (barrel or pincushion) to be analyzed.

When you have completed the fine adjustment, click one of the buttons on the bottom of the window. If you click Yes, Continue, the Multitest settings window, shown below, will open. If you click Yes, Continue in Express mode Multitest will run, display selected results Figures, and save selected results files.

multitest_settings

Multitest settings window

Chart type Selects the Chart type. Applied to the next run.
Color Matrix, ISO speed,
mxn chart settings
Opens the Color matrix (etc.) window, shown below. Color Matrix settings are described in the Color Correction Matrix web page. Other settings are described below.
Run Color Matrix Calculates and applies a color correction matrix if checked.
Reference Reference file. Can be standard or read in (typically from a CSV L*a*b* or density file). Varies somewhat with the chart type.
Color space Color space
Settings for Special grayscale charts Opens a window that lets you select density increment for linear grayscale stepcharts.
Display thumbnails & EXIF on the right Display thumbnails of the entire image and crop as well as EXIF results to the right of the plot. Examples will be shown below.
Plot settings (on right) Selects output figures. Described below.

color_matrix_settings
Color matrix, ISO speed, and mxn chart settings window.

Color correction matrix settings are described on the Color Correction Matrix page.

ISO speed is calculated if Indicent Lux, Aperture, and Exposure (time) are entered. Aperture and Exposure are obtained from the EXIF data if available, but Incident Lux must be measured and entered manually.

If an m (row) x n (column) chart is to be measured the correct values of m and n must be entered here. (The settings shown fit the X-Rite 24-patch Colorchecker). There charts require a L*a*b* reference file (in CSV format), which can be entered in the Multitest settings window.

Reference files

Most charts allow reference files to be entered using the Reference dropdown menu in the Settings area. (IT8.7 and CMP DT003 charts require them.) The available settings depends on the chart type. The previous file is generally available in the dropdown list. Here is a summary of the available reference sources.

Chart Default values Options (data from files)
X-Rite ColorChecker
(24-patch)
Default values from GMB, BabelColor, or Danes Picta (a chart with the same geometry but different colors) L*a*b*, xyY D50, or xyY D65 files in CSV format.
IT8.7 Requires a reference file in IT8.7 format, available from the chart manufacturer, generally available on their websites.
CMP DigitaL TargeT 003 Individually measured reference files are supplied with each target.
X-Rite ColorChecker SG Default values from GMB (L*a*b* D50); See Note below L*a*b*, xyY D50, or xyY D65 files in CSV format.
Stepchart (linear) Even density steps of 0.1, 0.15, 0.20, or 0.30. For the UTT only, L* = 95 to 5 in steps of -5 for the first 19 steps and 1 for step 20. (20-step UTT) should be selected. an ASCII density reference file with one density value per line can be entered.
Note: Several of the default reference values are derived from L*a*b* D50 values, i.e., L*a*b* values acquired under D50 illumination. When the color space has a different reference temperature (sRGB and Adobe RGB both have D65) a Bradford transformation is applied, resulting in a small change in the L*a*b* values. This is rarely a significant concern if you’re making images under a different illuminant: if the white balance algorithm is working properly, it should transform image colors to the appropriate values for the color space.

Except for the IT8.7 and CMP DT 003, reference files must be in CSV (comma-separated file) or CGATS format CSV files can be opened and edited in Excel. Color files (L*a*b*, xyY D50, and xyY D65) have three entries per line; density files have have one entry per line.

Here is an example of an L*a*b* file.

The format is CSV (comma-separated variables).
The format is the same for xyY files.

Added spaces are for visual clarity (readability) only.

55.261, -38.342, 31.37
28.778, 14.179, -50.297
65.711, 18.13, 17.81
51.038, -28.631, -28.638
51.935, 49.986, -14.574
81.733, 4.039, 79.819
42.101, 53.378, 28.19

If you have a spectrophotometer you can create a custom reference file (CGATS format) using techniques described in Measuring test chart patches.

Results figures

1. 3D Color Difference (with split)

In this plot, the height of the region is the selected color difference, in this case ΔE*ab (plain Delta-E). The tops of the patch platforms are shown as split colors with the reference color on the upper-left and the actual input color on the lower-right. This plot enables you to compare color measured with perceptual color differences. (They may not match as well as the theories claim.)

Panasonic_DMC-GF1_hVFAL_20MM_3D_diff

This plot is shown with the thumbnails and EXIF data on the right (set in a chckbox in the Multitest Settings window. Other plots will be shown without these results, i.e., with the box unchecked (so they take up less space).

2. Split Color (Reference/Input) plot

The Split Color plot shows the reference (ideal) color on the upper-left and the input color on the lower right.

If a color correction matrix has been calculated (if the checkbox has been checked), this plot and also the 3D Color Difference plot (above) will display the reference color on the upper-left, the input color on the upper-right, and the corrected color on the bottom.

ColorTest_split_3D_plot_corrected3D Color difference detail,
Color Correction Matrix calculated

ColorTest_split_plot_corrected

Split color detail, Color Correction Matrix calculated


Panasonic_DMC-GF1_hVFAL_20MM_split_diff

3. 2D a*b* plot

Displays the a*b* plane of L*a*b* color space. Chroma differences are (very) approximately the distance between the Reference and Camera values. (This difference is ΔCab. The more complex ΔC94 and ΔC00 (CIEDE 2000) equations (described here) are more accurate.

If a color correction matrix has been calculated, the corrected values will be displayed.

 

  Panasonic_DMC-GF1_hVFAL_20MM_2D_ab

4. xy Chromaticity

This plot is familiar, but less perceptually uniform than the a*b* plot, above. Green gets too much area in this plot.

If a color correction matrix has been calculated, the corrected values will be displayed.

  Panasonic_DMC-GF1_hVFAL_20MM_xy_Chrom

5. B&W Density & White Balance

The Upper plot displays Log(Pixel level/255) (maximum of 1) vs. input density (-log10(exposure)), i.e., uses density units.

The lower plot displays several types of ΔC as a function of patch density. See Colorcheck appendix.

If a color correction matrix has been calculated, the corrected values will be displayed.

  Panasonic_DMC-GF1_hVFAL_20MM_BW_density

6. EXIF data and Color Matrix

This figure shows selected EXIF data and the results of the Color matrix correction, including

  • the color matrix
  • a description of the settings used to calculate the matrix (Color difference used for optimization, weighting, linearization, …)
  • Mean and maximum olor differences between the input and reference image and between the corrected and reference image.
  Panasonic_DMC-GF1_hVFAL_20MM_EXIF

7. Additional 2D color plot

The vectorscope display is shown. One of several plots can be selected here:

  1. u’v’ Chromaticity
  2. Vectorscope
  3. 2D CbCr (from YCbCr)
  4. HS (Hue-sat) from HSV
  5. HS (Hue-Sat) from HSL

These displays are described in Using Multicharts – Displays and Options.

  Panasonic_DMC-GF1_hVFAL_20MM_2D_color

Noise/SNR analysis (not for all charts)

The Noise/SNR analysis is only available on charts that have patches large enough to provide good noise statistics. Noise analysis includes standard pixel noise, chroma noise, scene-referenced noise, sensor (raw) noise, and ISO-15739 visual noise. SNR (Signal-to-Noise Ratio) can also be displayed as a ratio or in dB.

The patches on the ColorChecker SG shown on this page are too small, but fortunately the Imaging-Resource image has a standard 24-patch ColorChecker to the left of the SG that we’ve analyzed to provide the results shown on the right.

Noise analysis is described in Multicharts/Multitest and eSFR ISO Noise. ISO 15739 visual noise was added in Imatest 4.0.

The following selections are available.

  1. Noise vs. input density (RGB)
    Noise in pixels or % of maximum pixel level
  2. Signal/Noise (S/N) vs. input density (RGBY) The luminance (Y-channel) is used for S.
  3. SNR (dB) vs. input density (RGBY) SNR (dB) = 20 log10(S/N).
    Shown on the right, above.
  4. Chroma noise vs. input density
  5. Chroma S/N vs. input density
  6. Chroma SNR (dB) vs. input density
  7. CIELAB (L*a*b*) noise
  8. CIELAB (L*a*b*) S/N
  9. CIELAB (L*a*b*) SNR (dB)
  10. Noise vs. pixel (all patches) These measurements are of primary interest in characterizing sensors from RAW (unprocessed or minimally processed) images
  11. S/N vs. pixel (all patches)
  12. SNR (dB) vs. pixel (all patches)
  13. F-stop (scene-ref) noise & Dyn Rng
  14. F-stop (scene-ref) S/N & Dyn Rng. Levels for Dynamic Range are shown as thick horizontal lines (yellow, orange, etc.).
    Shown on the right.
  15. F-stop (scene-ref) SNR (dB) & Dyn Rng
  16. ISO 15739 Visual Noise (L*u*v*)
  17. ISO 15739 Visual S/N (L*u*v*) SNR, as a fraction
  18. ISO 15739 Visual SNR dB (L*u*v*) in decibels (dB) = 20*log10(fraction)
  Panasonic_DMC-GF1_hVFAL_20MM_noise
GF3_DR36_narrow_i160_1010500_noise