Documentation

Contrast Resolution chart and analysis

Introduction

IntroductionBackgroundThe Contrast Resolution chartPhotographing & analyzing
Tonal response and SNRVisualizationAnalysisTone mapped resultsSummaryVerification

This page describes the Imatest Contrast Resolution chart (available May 2017; supported by Imatest 5.0+), which is designed for the visualization and measurement of contrast separation (the visibility of low contrast objects in larger fields ranging over a wide range of brightness). This chart should be of particular interest to the automotive and security industries. The design was inspired by a talk by Ulrich Seger of Robert Bosch at AutoSens 2016.

Prior to the release of Imatest 5.0 you’ll need to write us at support@imatest.com to obtain a download link.

We plan to introduce the chart to relevant standards committees, such as the IEEE P2020 Automotive System Image Quality group, which was initiated at the AutoSens 2016 conference. We will describe the chart at AutoSens 2017.

Background

Camera Dynamic Range (the range of tones a camera responds to with good contrast and Signal-to-Noise Ratio (SNR) is primarily a function of the sensor and lens (and to some extent, the signal processing). Modern High Dynamic Range (HDR) sensors boast impressive dynamic ranges of 120 dB (a 1,000,000:1 brightness ratio) or more, but when a lens is placed in front of a sensor, flare light (stray light reflected between lens elements and off the lens barrel) limits the system (camera) dynamic range to 100 dB or less. (If you think your system can do better and you have data to back it up, please write us at support@imatest.com. We’ll be delighted to engage with you.  :-))

Flare light can be measured from small pure black fields (“black holes” or light traps in reflective charts) in white fields, which usually extend beyond the image boundaries. Imatest’s older measurement of veiling glare (which indicates susceptibility to flare) is from the Stepchart module; Imatest 5.0+ has added the ISO 18844 flare measurement to the Uniformity module. After the image has been linearized and black level offsets (if any) have been removed, Veiling glare = dark level/white level.

The problem with flare/veiling glare measurements is (1) they cannot be directly related to the visibility of low contrast features in dark areas of the image, and (2) they tend to exaggerate the damage done by flare (i.e., a camera with a 120 dB HDR sensor may show only 50 dB of apparent dynamic range with this measurement).

Standard dynamic range measurements are typically made with transmissive grayscale step charts that have a density range greater than the dynamic range of the camera. While they show overall density response, they don’t give good results for tone mapped images, which widely used for HDR images processed for human viewing. Tone mapping selectively adjusts the tones in the image, lightening dark areas to make them visible in monitors with limited dynamic range. The tone mapped image on the right below was obtained by applying the Matlab tone mapping function with 8 tiles (the default) in the Imatest Image Processing module to the image on the left.

Crop of an image of the Imatest High Dynamic Range test chart, showing effects of tone mapping
Original Tone mapped
Density response— Original Density response— Tone mapped

Plots of log pixel level vs. exposure (-chart density) for the tone mapped images show low and inconsistent differences between patches, which is not at all indicative of the actual visibility of low contrast features within the patches. Tone mapping generally ruins tonal response measurements from conventional grayscale charts.

The Contrast Resolution chart was developed to overcome shortcomings of both flare/veiling glare and traditional dynamic range measurements.

The Contrast Resolution chart

Here is a concept image (different from the actual chart, which has a range of tones too large to be reproduced on a web page).

Contrast Resolution chart concept image

The physical Contrast Resolution Chart is made from two layers of 8×10 inch photographic film. The chart contains 20 large patches with densities ranging from OD (Optical Density) = base + 0.15 to base + 4.90 in steps of 0.25 OD, equivalent to 95 dB. (If the lightest and darkest small patches are included, the total density range is 5.05 OD = 101 dB.) The large patches are used for noise measurements because the small patches are usually too small for good noise statistics.

Inside each large square (illustrated on the right) there are four small patches: light and dark gray, red and blue. The gray patches are 0.15 OD above and below the surrounding square. The difference, 0.30 OD (6 dB), is a 2:1 contrast ratio. The mean density of the light and dark gray patches is identical to the density of the surrounding gray patch to minimize effects on tone mapping. (The same holds true for the blue/red patches.) The small light and gray patches are used for a detailed quantitative analysis; the blue/red patches are used for a visual estimate-only).

(Note that many recent HDR image sensors claim a dynamic range of 120 to 150 dB. This may be the case for the sensor in isolation, but the dynamic range of real cameras with real optical systems isn’t even close. The practical limit is 100 dB at most; worse when bright areas inside or near the image field cause flare light.)

Here is an image of the chart taken by a Canon EOS-6D (a full-frame DSLR with a high quality 100mm macro lens set to ISO 100), originally captured in raw format, converted to a DNG file (because dcraw may not properly remove black level offsets (pedestals) from Canon CR2 files), converted to a 48-bit Adobe RGB TIFF file for analysis (with no sharpening or noise reduction), then cropped. The image below is an 8-bit JPEG derived from the very large 48-bit TIFF file.

Contrast Resolution chart image, originally captured in RAW format.
Click on image to bring up fill-sized image, which can be downloaded for analysis.

Photographing and analyzing the chart

The Contrast Resolution chart can be purchased from the Imatest store. It is a two-layer 8×10 inch (20x25cm) transparency that needs to be back illuminated. Although any relatively uniform, non-flickering lightbox will work, we recommend the ITI Lightbox, which is extremely uniform (97%), can be dimmed over a wide range, and has two color temperatures (normally 3100 and 5100K, but other combinations can be custom-ordered).

The chart should be photographed in a completely darkened room. Care must be taken to avoid any light from reflecting back to the front surface of the chart. We ordinarily surround the light box with a box made from black foam board and corner molding to prevent local reflections. We are thinking of lining the box with blackout material to improve the light absorption. The area behind the camera should be as dark as possible. Light-colored clothing— which can reflect light back to the chart— should be avoided.

Test setup showing black box around the lightbox and chart
Pay no attention to that man behind the curtain.

Frame the chart so that the chart height in the image is at least 400 pixels— more is desirable for high resolution systems. In typical high resolution systems the chart should fill approximately 40% of the vertical or horizontal field. It is generally a good idea to avoid filling the image with the chart because light falloff from the lens could reduce the measurement accuracy.

Analysis of the Contrast Resolution chart is included in the interactive Multicharts and batch-capable Multitest modules in Imatest 5.0+. Please contact us for a download link if this version has not yet been released.

To run the image in Multicharts, click the Multicharts button in the Imatest main window and select 7. Multi-row Grayscale & Color charts from the dropdown menu just below the Read image file button. Then select the image. This opens the Multicharts Special Chart selection window. Select 13. Contrast Resolution. Reference source should be either Default values or a file name we may supply in the future. Then click OK. For succeeding runs, you can either click 3. Contrast resolution (last grayscale) in the Multicharts window (this eliminates the Multicharts Special selection) or click Read image file if 3.Contrast Resolution… is displayed in the dropdown menu. You only need to press 7. Multi-row… if you are changing the chart type.

Chart detection is fully automatic when Auto detection for the 36-patch Dynamic Range and Contrast Resolution charts has been selected in ROI settings. Auto detection has recently become very robust (manual selection is also available). Here is the fine adjust window following automatic detection (or manual selection). If the adjustment is correct (distortion can be added if needed) click Yes, Continue in Express mode (or Yes, Continue). In Multicharts, which is highly interactive, settings can always be changed later. (For Multitest, settings need to be made prior to the run.)

To run Multitest (the non-interactive, batch-capable version of Multicharts), press the Setup button just to the right of Multitest… (middle-left of the Imatest main window). Chart type should be set to either 7. Multi-Row Grayscale... or 3. Contrast Resolution… (if the Contrast Resolution chart was used for the last run). Special (multi-row) grayscale chart type should be set to 13. Contrast Resolution. The only plots of relevance are B&W Density & White Bal., Noise/SNR (two plots can be selected), and Image plot. Then click OK. Multitest: Contrast-Resolution will appear in the Imatest main window. Press this button to run Multitest. More details in the full Multitest instructions.

Contrast Resolution fine adjustment window

Five regions are selected for each of the twenty large patches. The large rectangle is for the uniform gray area: if you’ve framed the image properly it should be large enough for good noise statistics. The two small rectangles to the right are for the dark and light gray areas. The two small squares inside the large rectangle are used to subtract the effects of vertical nonuniformity (due to flare light) from the light-dark gray patch measurement.

The Display settings of interest for the Contrast Resolution analysis are

7.   B&W density & White Balance
11. Noise analysis
12. Display image

Results and Visualization

Tonal response and Signal-to-Noise Ratio (SNR)

Here is a key result for tonal response and Signal-to-noise Ratio (SNR).

Tonal Response and SNR for the 48-bit TIFF file derived from the RAW Canon EOS-6D image

The relatively straight (thin) Red, Green, Blue, and Gray (for the Luminance (Y) channel) curves are from the large gray patches. They correspond to standard results for Dynamic Range charts. The spikes in these curves (which aren’t very prominent) are for the light and dark gray patches within the larger gray patches. The Magenta curves contain the significant results, derived from the difference between the inner gray patches.

In the upper (tonal response) plot the strong Magenta curve ( —-◊—- ) represents the difference signal ΔS for the lighter-darker inner gray patches. This is a very good representation for the signal for determining the visibility of these patches. The weaker Magenta curve ( ——- ) is the difference signal ΔS normalized to the regular signal (the level of the large gray patch). The normalized difference signal stays relatively constant until patch 16, when is starts to drop abruptly.

In the lower (SNR) plot the strong Magenta curve ( ——- with square markers) is the Signal-to-Noise Ratio derived from the light-dark gray difference signal ΔS. This is probably the most important result.

To make sense of these curves we need to look carefully at the image, and we need to deal with the little problem that the contents are of the very dark patches are not visible in normal monitors in normal viewing conditions..

Visualization

The key attribute of the Contrast Resolution analysis is that it enables clear visualization of the visibility of low-contrast objects within larger fields. This cannot be easily done with standard grayscale charts with sequential patches.

Patch contents can be visualized by selecting Display image then selecting Tone Map (which applies the Matlab tonemap function) or one of the Constant patches options.

Tone map (shown on the right) does not produce a reliable quantitative result— it is affected by noise and minimum pixel level— but it’s of interest because the output of automotive cameras designed for human vision (Camera Monitor Systems) will almost certainly be tone mapped (though likely with more sophisticated routines). Tone mapping lightens the dark patches so that interior features can be visualized. 

Tone mapped image (display-only):
NOT used for quantitative analysis

Constant patches (available only in Multicharts) provides more reliable displays. There are three constant patches display options (in the small dropdown window in the Settings region). Their results are similar.

  • Constant patches (xyY)  The image is converted in to xyY color representation, where Y is the (linear) luminance. This calculation uses the image Color space (typically sRGB, though Adobe RGB is occasionally used). The mean value of Y is calculated for each patch, then the entire parch is divided by mean(Y) and scaled for display. This is typically the best calculation for color space images, but results are quite similar to Constant patches (simple).
  • Constant patches (L*)  The image is converted into CIELAB L*a*b* (using the image Color space). The mean value for L* is calculated for each patch and used to scale the image for constant mean L*. Patches in dark areas are typically undersaturated.
  • Constant patches (simple)  The mean value of each patch is calculated, then the entire patch is divided by the mean and scaled for display. No assumptions regarding color space or linearity are applied.

Constant patch image (from Y in xyY): a good indicator of feature visibility.

Several numbers are shown in each patch this display. Here they are for a typical patch (#10).

Line Contents (patch #10) Description
1 S 0.108 0.317 S indicates signal. The first entry (S1 = 0.108) is the signal for the patch normalized to 1.0. The second entry (ΔS/S1 = 0.317) is the difference between the small light and dark gray patches inside the larger patch, normalized to the signal (S1) in the first entry. Log10(S1) and Log10(ΔS/S1) are displayed in the upper plot of the Tone and Noise (or SNR) figure.
2 p10 Indicates the patch number (10)
3 S/N Indicates that the following line contains Signal-to-Noise Ratio (simple ratio; not dB)
                                    Signal-to-Noise Ratios. Noise N is calculated in the large patch. The signal for the first SNR (SNR(S)) is the mean pixel level of the large patch (S1 in line 1). The signal for the second SNR (SNR(ΔS)) is the difference between the small light and dark patches, ΔS. This number relates to how well the two gray patches can be distinguished. SNR(S) and SNR(ΔS) Log10(SNR(S)) and Log10(SNR(ΔS)) are displayed in the lower plot of the Tone and Noise (or SNR) figure (——— and ——-, respectively).

Analysis— Contrast Resolution Dynamic Range

By comparing the Constant patch image with the Tonal Response/SNR plots above we can learn quite a lot about Contrast Resolution.

From the above Constant patch image we observe

  • The inner patches are clearly visible to patch 11, where SNR(ΔS) = 4.70.
  • The inner patches are visible but noisy through patch 13, where SNR(ΔS) = 2.42.
  • The inner patches are difficult to see (visible only because they resemble patches in regions with better SNR) in patches 14 and 15.
  • The inner patches are completely invisible in 16 and beyond.

Note that the normalized signal ΔS/S1 stays fairly consistent through patch 16, and only seriously drops for patches 15 and beyond. This is a pretty clear indicator that the normalized signal is not a good indicator of performance. SNR(ΔS) = ΔS/N, which we will call Contrast Resolution SNR is much better.

Contrast Resolution SNR can be used as the basis for defining a Contrast Resolution Dynamic Range (CRDR)— the range of tones over which the SNR stays above a specified value, which would be somewhere in the range of 6 to 14 dB (2:1 to 5:1), based on our observations. The SNR value might be altered for specific applications, for example, if a lower contrast feature were of interest, the SNR for calculating CRDR (based on the chart image) would be increased.

Illustration of proposed
Contrast Resolution Dynamic Range, based on SNR(ΔS)

A paper, “Signal-to-noise optimization of medical imaging systems” by I.A. Cunningham and R. Shaw, describes a measurement called SNRRose similar to this SNR(ΔS) and indicates that a value of 5 or greater is required for reliable detection of an object. From looking at these results it appears that a lower SNR(ΔS) as low as 2 might be sufficient (though marginal), at least for 2:1 contrast patches. More work needs to be done to determine a good lower limit, which is very likely to be close to the value of 5 from the paper.

Tone mapped results

It is instructive to look at results for a tone mapped image. To obtain such an image we applied the Matlab tonemap tone mapping algorithm with the default value of 8 tiles to the above image using the Imatest Image Processing module.

Tone mapped image, derived from above image,
reduced to 1000 pixels wide

The tone curve in the upper plot (below) is very different from the original image. The tonal response has been flattened so that the gamma (the average slope in gray regions) has been reduced from 0.422 to 0.0734— a meaninglessly low value. But the normalized light-dark gray difference signal ΔS/S (the thin dashed Magenta line: ——-) and the light-dark gray difference Signal-to-Noise Ratio ΔS/N (the strong Magenta curve ——- with square markers) in the lower plot are not very different.

These results clearly show that Contrast Resolution measurements— especially the difference-based Signal to Noise Ratio ΔS/N— are affected very little by tone mapping, i.e., good measurements can be made in tone mapped systems.

Tonal Response and SNR for the tone mapped image derived from the RAW Canon EOS-6D image

The reason that noise and SNR are relatively stable when tone mapping is applied is that the Imatest noise calculation is quite robust against nonuniform illumination within regions of interest— which tone mapping certainly creates. Noise is ordinarily equivalent to the standard deviation (σ) of the levels in a smooth (flat) region, but this simple calculation can misinterpret nonuniformity as noise. Imatest subtracts a second-order fit to the mean horizontal and vertical pixel levels from the region prior to calculating noise— effectively removing nonuniformity from the calculation. The difference between the simple and nonuniformity-corrected calculations can be observed by running the Imatest Image Statistics/Viewer program.

Summary

We have shown that the Contrast Resolution chart produces a clear indication of the visibility of low contrast objects (using small squares with a 2:1 transmittance ratio (a 0.3 Optical Density difference) inside larger squares). Unlike standard grayscale step charts, it works well with tone mapped images.

The SNR based on the light-dark patch difference, Contrast Resolution SNR = SNR(ΔS) = ΔS/N, provides an excellent indication of the visibility of the visibility of the light and dark patches. When  SNR(ΔS) is larger than a value to be determined (at least 2 (6dB), but more likely 4 or 5 (12 or 14 dB)) the objects can be effectively distinguished.  SNR(ΔS)

This criterion can be used to establish a Contrast Resolution Dynamic Range, which could be defined as the range of scene brightness for which small objects with a OD difference of 0.3 (SNR(ΔS) > n dB, where n is somewhere around 2 to 5) can be distinguished. When this level is better established it will be clearly indicated in Imatest results.

Contrast Resolution chart on 17×24 inch Artograph Lightpad
for alternative veiling glare measurement

The Contrast Resolution chart could also be mounted (preferably without a frame) on a large lightbox, such as the 17×24 inch Artograph Lightpad, to measure veiling glare, which would be indicated by the measurement difference (the difference in SNR(ΔS)) between images without and with the light surround.

Verification

The darker patches in the images used in this page contain no visible detail. This naturally leads to the question, is detail present in the charts themselves? This took some effort to verify. What we did is to mask off all but the bottom two rows of the chart (to minimize fogging from flare light), then acquire some images. Here are the results showing that the darkest patches do indeed contain detail.

Bottom two rows (with others masked), Normal exposure

Bottom two rows (with others masked), Extreme overexposure
Note some unevenness from flare light (from the thin unmasked strip) is visible in the bottom row.

These images clearly show that information is present in the darkest patches. some discoloration is noted. This comes from our process, which has some issues in very dark patches. This coloration has little effect on the results, and is not visible in patches bright enough to be visible in normal viewing conditions.