Documentation

Using eSFR ISO Part 1

Imatest eSFR ISO performs highly automated measurements of several key image quality factors using versions of the new ISO 12233:2014 E-SFR (Edge SFR) test chart that may be purchased from the Imatest store (recommended) or printed on a high-quality inkjet printer. Region of Interest (ROI) selection is automatic, based on user-entered criteria (similar to  SFRplus, which it closely resembles). Image quality factors include:

Our new photographic eSFR ISO charts have far more detail than similarly-sized inkjet charts, enabling analysis of much higher resolution cameras.

  • Sharpness, expressed as Spatial Frequency Response (SFR), also known as Modulation Transfer Function (MTF)
  • Lateral Chromatic Aberration
  • Distortion (with less detail than the Distortion module). Details in SFRplus Distortion and Field of View measurements.
  • Noise, measured from the grayscale patches surrounding the center of the chart, includes all types of noise calculated by Multicharts and Multitest (standard pixel noise, chroma noise, scene-referenced noise, sensor (raw) noise, and ISO-15739 visual noise and dynamic range). SNR (Signal-to-Noise Ratio) can also be displayed as a ratio or in dB.
  • Tonal response (also from the grayscale patches).
  • Color accuracy, when used with an eSFR ISO chart that contains the optional color pattern (available in all inkjet and color film LVT charts, but not in B&W photographic charts), near the grayscale patches.
  • ISO sensitivity (Saturation-based and Standard Output Sensitivity), when incident lux is entered.
  • Onset of aliasing and Moiré, measured  from the four pairs of wedges in the Enhanced and Extended charts, whose spatial frequency range is the same as the separate ISO 12233:2014 wedge chart. Figures and results are identical to the Wedge module.

All chart features— slanted-edges, grayscale patches, color patches, and wedges— are located automatically. Special versions of eSFR ISO charts are available with extra wedges, pre-distortion (for use with extremely barrel-distorted “fisheye” lenses), or both.

Watch on YouTube.

eSFR ISO operates in two modes.

  • Interactive/setup mode (run by pressing Rescharts or eSFR ISO setup in the Imatest main window), which allows you to select settings and interactively examine results in detail. Saved settings are used for Auto Mode.
  • Auto mode (run by pressing eSFR ISO Auto in the Imatest main window), which runs automatically with no additional user input. ROIs are located automatically based on settings saved from the interactive/setup mode. This allows images of different sizes and framing to be analyzed with no change of settings. Auto mode works with large batches of files, and is especially useful for automated testing, where framing may vary from image to image.

This document introduces eSFR ISO and explains how to obtain and photograph the chart. Part 2 shows how to run eSFR ISO interactively inside Rescharts and how to save settings for automated runs. Part 3 illustrates the results.

Click on any of the thumbnails below (or on the image itself, repeatedly) to explore the many capabilities of eSFR ISO.
Note the description below the image. Results are described in more detail in Part 3.

 
eSFR ISO setup window thumbnail
Edge and SFR plot for a center region thumbnail
LX7_eSFR-ISO_4.7mm_f2.8_crop_ROI_Geometry thumbnail
3D plot of MTF50P (the spatial frequency where MTF is 50% of it's peak value) thumbnail
Lateral Chromatic Aberratiion plot for an upper-right region thumbnail
Acutance thumbnail
Tonal response thumbnail
Multi-region display thumbnail
Image and Geometry display thumbnail
Edge roughness display showing total, periodic (aliasing), and aperiodic roughness thumbnail
Split color display thumbnail
Color difference (a*b* plane of L*a*b* color space) thumbnail
Noise display thumbnail
Wedge MTF and Aliasing display thumbnail
eSFR ISO setup window

eSFR ISO setup window. Layout is similar to SFRplus setup.
Settings on the lower-left select auto detection of the grayscale step chart, color patches, and wedges.

Edge and SFR plot for a center region

Edge and MTF plot for a single region. Any region can be selected.

LX7_eSFR-ISO_4.7mm_f2.8_crop_ROI_Geometry

Large view of edge with Edge and MTF measurements-- for visual comparison.
Available in Edge & MTF and Image & Geometry displays.

3D plot of MTF50P (the spatial frequency where MTF is 50% of it's peak value)

3D plot. Has somewhat less spatial detail than SFRplus.
You can select any of a great many results (including secondary readouts).

Lateral Chromatic Aberratiion plot for an upper-right region

Lateral Chromatic aberration (affected by the lens and demosaicing) is most significant near the edges of the image.

Acutance

CPIQ Acutance or Subjective Quality Factor:
a perceptual measurement that includes viewing conditions and the human visual system.

Tonal response

Tonal response: Log pixel level vs. log exposure. Gamma is the average slope in the highlight to dark gray areas. The lower curve can contain local contrast or white balance.

Multi-region display

Multi-Region display: can become cluttered if too many regions are selected.
Options are available to simplify the display.

Image and Geometry display

Display the image and selected regions and show a number of geometric results, including distortion, tilt, shifts, convergence angles, and field of view. Checking Crop to ROI lets you compare edge appearance to Edge and MTF results.

Edge roughness display showing total, periodic (aliasing), and aperiodic roughness

Edge roughness, which can be decomposed into periodic (aliasing-related) and aperiodic (noise-related) components.

Split color display

Reference colors on the upper-left; input colors on the lower-right of each patch. Probe can be used to examine each color patch in detail for color differences (Delta-E ab, 94, and 2000).

Color difference (a*b* plane of L*a*b* color space)

Color differences on the a*b* plane of CIELAB color space.

Noise display

Noise measurement from grayscale patches, with tonal response on top.
All noise measurements from Multicharts and Multitest are available, including chroma noise, scene-referenced noise, sensor (raw) noise, and ISO 15739 visual noise.

Wedge MTF and Aliasing display

Wedge MTF and Onset of Aliasing (where the number of detected bars drops).
Wedges are automatically detected when the Wedge checkbox in the Setup window is checked.

 

 

E-SFR ISO test charts

Imatest E-SFR ISO test charts come in three variants: Standard, Enhanced and Extended. All conform to the ISO 12233:2014 standard. The standard E-SFR ISO test chart, which has only features called for in the standard, is shown below.

new_ISO-12233_plain_vanillaStandard E-SFR ISO chart: 3:2 aspect ratio, no added features
Click on any of these charts for larger versions.new_ISO-12233_enh_wedges_3_2_sRGBEnhanced E-SFR ISO test chart: 3:2 aspect ratio, 6 added squares on sides,
16 added color patches, four pairs of added wedge patterns. new_ISO-12233_ext_wedges_16_9_sRGBExtended E-SFR ISO test chart: 16:9 aspect ratio, 6 added squares on sides,
16 added color patches, four pairs of added wedge patterns.

esfr_iso_predistortedEnhanced E-SFR ISO test chart: 3:2 aspect ratio, shown with extra wedges
(4 low frequency, 16 high frequency total) and pre-distortion for use with fisheye lenses
.
A great many options are available.

Sharpness is derived from light/dark slanted edges at the boundaries of the squares, as described in Sharpness: What is it and how is it measured? eSFR ISO can handle a wide range of camera aspect ratios; the left and right sides of the chart may extend beyond the frame or be well inside it.

ISO 12233:2014 Compliance:  The Imatest standard E-SFR chart is similar to the chart illustrated in the ISO 12233:2014 standard document, but with a few minor differences that maintain ISO-compliance. (We recommend the Enhanced or Extended charts.)

  • There are four registration (fiduciary) marks instead of three. An additional black dot, just above the lower-left registration mark, is used to detect orientation. This change was made because the projective transformation used for automatic region detection requires four points. Annex C, Table C.1, I of the standard states, “Asymmetric features for automatic target detection. These features should be placed within the 4:3 active area aspect ratio, but may be placed for convenience as dictated by use cases.”
     
  • The focus-aid patterns in the center of the squares are different from the patterns illustrated in the ISO standard document (which are not described in the standard): they are square (which allows more room for the slanted-edge regions used for measuring sharpness), and they have been designed to minimize interference with automatic region detection. They have no effect on measurement results.
     
  • A 20-patch OECF gray scale pattern is used. Annex C states, “The test chart also includes grey scale patches which are used to determine the OECF. While the example chart in Figure C1 uses 16 OECF test patches; a 20 patch version as described in ISO 14524 may also be substituted.”

The Enhanced and Extended charts contain valuable features (additional squares, wedges, and color patches) in regions of the chart that are not specified by the ISO standard. These features do not violate the standard in any way.

eSFR ISO chart types
Chart type Features Notes
Standard Similar to chart illustrated in the standard document. No added features. 3:2 aspect ratio.
Cannot measure sharpness near the sides or edges of the image. Use only if required. The Enhanced or Extended charts are recommended for most purposes.
Enhanced 3:2 aspect ratio. 6 added squares on sides; several added hyperbolic wedges; 16 added color patches in Imatest-produced inkjet and 8x10 inch LVT charts. Best for 4:3 or 3:2 aspect ratios (camera phones, compact cameras, DSLRs)
Includes several hyperbolic wedges with properties similar to the ISO (CIPA) visual resolution test chart which are automatically detected in the eSFR ISO module or manually selected in the Wedge module.
Extended 16:9 aspect ratio. Same added features as enhanced chart. Best for 3:2 or 16:9 aspect ratios (DSLRs, HDTV and cinema formats)

 

eSFR ISO chart print options (can be selected when ordering)

  Options Notes
Media Inkjet, LVT film, Photographic paper*, Chrome on Glass* *Available in the near future: check the Imatest Store. Photographic paper charts are much finer than inkjet charts of the same size.
Chart aspect ratio

3:2 (Standard & Enhanced) or
16:9 (Extended)

3:2 charts (Standard and Enhanced) are well-suited for DSLRs (3:2 aspect ratio) and compact digital cameras and camera phones (4:3 aspect ratio). The 16:9 (Extended) chart is well-suited for HDTV (16:9 aspect ratio) and for DSLRs (3:2 aspect ratio).
Contrast
4:1  only 4:1 contrast is specified in the new ISO 12233:2014 standard.
Grayscale
stepchart
Included (below center) A 20-patch ISO 14524 circular grayscale pattern is included.
Color chart 16 patches
Included in all inkjet-printed Enhanced and Extended inkjet charts (all sizes) and in LVT film charts (up to 8x10 inches) produced by Imatest. (Not be available on reflective photographic charts.)
Pre-distorted chart
(The standard chart is undistorted.) Two standard levels of pincushion distortion distortion are available for use with fisheye (strongly barrel-distorted) lenses. Available in matte surface only.
Surface Matte or semigloss Semigloss is slightly sharper and has a more accurate grayscale in the deep tones (print densities > 1.5), but is more susceptible to glare (specular reflections), especially with wide angle lenses. Matte surface is recommended for wide angle lenses
Extra wedges Available in all surfaces except reflective photographic Includes several added wedges at the periphery of the chart and four near the center (inside the grayscale).

 

Old ISO 12233:2000 chartAdvantages of the ISO 12233:2014 Edge SFR chart
over the old ISO 12233:2000 chart

Note: the ISO 12233:2000 standard and chart, shown on the right, is referenced in the new ISO 122433:2014 standard, but is no longer an official part of the standard. The new standard specifies three charts, one of which is a slanted-edge (Edge SFR or E-SFR) chart with much lower contrast (4:1; shown on the right, below).

Compared to the old ISO 12233:2000 chart,

  • The E-SFR chart has much less wasted area, especially in the Enhanced and Extended versions. About 90% of the old ISO chart is covered with patterns that have little value for computer analysis.

  •  ISO-12233-2014-chart_from_docISO 12233:2014 Edge SFR chart
    from the ISO document

    You can produce a detailed map of sharpness (MTF) over the image surface with the Enhanced and Extended charts. This cannot be done with the old ISO chart because there are not enough suitable edges— and they are not well-located.

  • Automated region detection, based on location criteria you enter. This makes the E-SFR chart and eSFR ISO module well-suited for automated testing: With the old ISO chart, regions of interest (ROIs) had to be selected carefully whenever the image framing changed, even slightly.

 

  • The 4:1 contrast edges are less likely to clip than the edges in the old ISO chart, whose contrast is specified at ≥ 40:1. The camera operates in a more linear region, and hence results are more consistent and accurate— less affected by overexposure, underexposure, incorrect gamma estimate, or excessive software sharpening. 4:1 contrast edges are more representative of real edges that affect perceived image sharpness.

  • eSFR ISO can measure additional image quality factors, including lateral chromatic aberration, distortion, gamma (contrast), tonal response, and color accuracy (in charts that have the color pattern).

4:1 contrast 5X7 SFRplus chart

  • The old ISO 12233:2000 chart contains no information that can be used to measure gamma (the average slope of the Log pixel vs. log exposure curve. i.e., the contrast), which is needed to linearize the image for the MTF calculation. Accurate gamma measurement is particularly important with high contrast targets. In the E-SFR chart, gamma can be derived either from the individual squares (which have a known 4:1 contrast) or from the grayscale step chart.

Compared to the SFRplus chart,

  • The map of MTF over the image surface is less detailed, though it's sufficient for most purposes.
  • Distortion measurements are slightly less detailed (especially for the Standard chart) because there are no distortion bars and there are fewer features for measuring distortion.
  • eSFR ISO contains a highly detailed noise calculation, including all noise measurements from Multicharts and Multitest.

 

Slanted-edge algorithm The algorithms for calculating MTF/SFR were adapted from a Matlab program, sfrmat, written by Peter Burns () to implement the ISO 12233:2000 standard. Imatest SFR, SFRplus, and eSFR ISO incorporates numerous improvements, including improved edge detection, better handling of lens distortion, a nicer interface, and far more detailed output. The original Matlab code is available on http://losburns.com/imaging/software/SFRedge/index.htm. In comparing sfrmat results with Imatest, note that if no OECF (tonal response curve) file is entered into sfrmat, no tonal response curve is assumed, i.e., gamma = 1 (linear response). Since the default value of gamma in Imatest is 0.5, which is typical of digital cameras, you must set gamma to 1 to obtain good agreement with sfrmat.

Obtaining the chart

Summary

isoesfr12233bw

High resolution photographic eSFR ISO chart

The three Edge SFR charts– Standard, Enhanced, and Extended– can be purchased from the Imatest store in a variety of media: inkjet (reflective and transmissive), LVT film (transmissive), and photographic paper (reflective; much higher resolution than inkjet charts). (For Chrome-on-Glass charts, SFRplus is often preferred because Chrome-on-Glass can't support a grayscale.) Inkjet charts can be printed on widebody inkjet printers, but you must have fine materials, skill, and a knowledge of color management. We strongly recommend that you purchase a chart.

Inkjet and photographic charts should be mounted on 32x40 or 40 x60 inch sheets of 1/2 inch (12.5 mm) thick foam board with a spray adhesive (such as 3MTM 77 or Photo Mount) or double-sided tape (such as 3MTM #568 Positionable Mounting Adhesive). 1/2 inch foam board stays flatter than standard 1/4 or 3/8 inch foam board. PVC board is also very promising: it may be more durable and it comes in a variety of sizes.

Photographing the chart.

Framing requirements:

 

Ideal cropping

The cropping on the right strictly conforms to the ISO standard, but is not required for practical measurements. With this cropping the calibration of the wedges will be correct (LW/PH is the printed number X 100). These numbers are not used for Imatest analysis.

eSFRiso_enhanced_ideal_286W
eSFRiso_enhanced_ideal_mediumcrop_286HClose Cropping: The registration marks and the left and right (center row) squares should be within the image. Use SFR if you can't obtain this. eSFRiso_ctr_50pct_ideal_286HFar cropping: the chart should occupy at least half the image height. This cropping is not recommended because regions near the image boundaries are not analyzed and edges and other features may be too small to provide optimum (consistent) results with low resolution images. 
  • The boundaries of the chart may extend beyond the image (as shown below) or be well within the image. The software is designed to accommodate a wide variety of framing and aspect ratios. Edges near the left and right boundaries will be properly located or omitted if necessary. If the sides of the chart are inside the image, you should try to minimize interfering patterns that could be mistaken for registration marks. Chart surroundings inside the image should be light to middle gray if possible. If there are problems due to the surroundings, you can crop the image in the eSFR ISO setup window.
  • If possible, avoid patterns that could be mistaken for registration marks outside the chart area. In practice eSFR ISO is surprisingly robust in such situations, but it's best to keep surroundings simple, and if possible, close to neutral gray (15-25% reflectance).

esfriso_LX7_typicalWell-framed eSFR ISO image
(though not quite perfect— a little closer would have been better)
It's OK (and quite commonplace) for the sides of the chart to run off the image.

  • The chart may be aligned using techniques shown in Building a Low-Cost Test Lab. eSFR ISO tolerates moderate misalignment: a tilt of ~2 degrees, some perspective (keystone) distortion, etc., but every effort should be made to align the chart properly. Moderate barrel or pincushion distortion (<8% SMIA) is also well tolerated, but a pre-distorted chart should be used with extreme fisheye lenses (with high barrel distortion).
  • The image can be cropped (starting with) to remove interfering features near the edges, using the Crop borders button in the eSFR ISO setup window.

If exposure compensation is available, increasing exposure by +1 f-stop often produces a better exposure, especially on the grayscale stepchart pattern.

Lighting

The chart below summarizes lighting considerations. The goal is even, glare-free illumination. Lighting angles between 30 and 45 degrees are ideal in most cases. At least two lights (one on each side) is recommended; four or six is better. Avoid lighting behind the camera, which can cause glare. Check for glare and lighting uniformity before you expose. A description of recommended lighting setups can be found in The Imatest Test Lab. (We formerly recommended 4700K (near-daylight) SoLux quartz-halogen lamps, but we now recommend LED lamps, which are available with adjustable intensity and color temperature.) The BK Precision 615 Light meter (Lux meter) is an outstanding low-cost instrument (about $100 USD) for measuring the intensity and uniformity of illumination.

Simplified lighting diagram

Distance

Distance and field of view
The camera must be far enough from the chart so you are measuring the sharpness of your camera and lens, not the chart. But remember,

It's the field of view, not the chart-to-camera distance, that counts.

A rough rule of thumb: For an inkjet-printed chart the field of view should be at least
24 inches (60 cm) for a 6-megapixel camera;
40 inches (90 cm) for a 16 megapixel camera.
Details below. For a high-quality photographically-printed chart (such as the charts from Applied Imaging) you can get quite a bit closer.

A letter-sized (8.5x11 inch) chart printed on Premium Luster paper on the Epson 2200 (a high quality pigment-based inkjet photo printer) was analyzed for MTF using the 6.3 megapixel Canon EOS-10D. There was no change when the image field was at least 22 inches (56 cm) wide— twice the length of the chart. Performance fell off slowly for smaller fields.

Choose a camera-to-target distance that gives at least this image field width. The actual distance depends on the sensor size and the focal length of the lens. The minimum image field is illustrated on the right.

Cameras with more pixels, and hence higher potential resolution, should have a larger image field width, hence printed chart width.

Distance/field width guidelines for high quality inkjet charts
(You can get closer with photographically-printed charts.)

The camera-to-target distance is not critical as long as it is greater than a reasonable minimum.
 
Sensor sizes
Desig-
nation
Diagonal
mm.
Width
mm.
Height
mm.
1/3.6" 5.0 4.0 3.0
1/3.2" 5.68 4.54 3.42
1/3" 6.0 4.8 3.6
1/2.7" 6.59 5.27 3.96
1/2" 8.0 6.4 4.8
1/1.8" 8.93 7.18 5.32
2/3" 11.0 8.8 6.6
1" 16.0 12.8 9.6
4/3" 22.5 18.0 13.5
35mm 44.3 24.0 36.0
Image field width (in inches) > 8.8 × sqrt(megapixels)
Image field width (in cm) > 22 × sqrt(megapixels) — or —There should be no more than 140 sensor pixels per inch of target or 55 sensor pixels per centimeter of the target. — or —The distance to the target should be at least 40X the focal length of the lens for 6-10 megapixel digital SLRs.(25X is the absolute minimum for 6 megapixel DSLRs; 40X leaves some margin.) For compact digital cameras, which have much smaller sensors, the distance should be at least 100X the actual focal length: the field of view is about the same as an SLR with comparable pixel count. The recommended distance is described in more detail in Chart quality and distance, below.
There is some confusion about lens focal lengths in compact digital cameras. They are often given as the "35mm-equivalent," which many photographers can relate to viewing angle. 35-105mm or 28-140mm are typical "35mm-equivalent" numbers, but they are not the true lens focal length, which is often omitted from the specs. What is given is the sensor size in 1/n inches, a confusing designation based on the outside diameter of long-obsolete vidicon tubes. It The table on the right relates the 1/ndesignation to the diagonal dimension of the sensor. True focal length = "35mm-equivalent" × (diagonal mm.) / 44.3

Exposure

Good exposure is important for accurate eSFR ISO results. Neither the black nor the white regions of the chart should clip— have substantial areas that reach pixel levels 0 or 255. The best way to ensure proper exposure is to use the histogram in your digital camera. Blacks (the peaks on the left) should be above the minimum and whites (the peak(s) on the right) should be below the maximum.

The above image (taken from the Canon File Viewer Utility) is close to a perfect exposure. Some exposure compensation, typically around +1 f-stop, may be helpful.

 

Tips on photographing the chart
Distance doesn't matter as long as the target far enough from the camera so sharpness is limited by the camera and lens, not by the target. A Wide body printer (capable of printing images at least 24 inches high) are required to print the eSFR ISO chart.
The target should be evenly lit and free of glare.
White balance should be approximately neutral.
Use a sturdy tripod and a cable release. For DSLRs, use the mirror lock if possible. You can use Imatest SFR to find the difference made by a good tripod or mirror lock— to sharpen your technique, literally ( pun intended ).
Be sure to expose the image so detail is maintained in both light and dark areas. Neither should be blocked (clipped). Use your camera's histogram. If more than 0.5% of the pixels are at levels 0 or 255, Imatest SFR will assume that clipping has taken place and issue a warning message. This has no effect on the calculations— it's just a warning that accuracy may be compromised.
Be sure the camera is correctly focused on the chart.
Place slanted-edge images near the corners of the field as well as near the center.
You may find it instructive to photograph the slanted edge target along with a target from Lens testing, but there's no need to do so.

Save the image as a RAW file or maximum quality JPEG. If you are using a RAW converter, convert to JPEG (maximum quality), TIFF (without LZW compression, which is not supported), or PNG. If you are using film, develop and scan it.

If the folder contains meaningless camera-generated file names such as IMG_3734.jpg, IMG_3735.jpg, etc., you can change them to meaningful names that include focal length, aperture, etc., with the View/Rename Files utility, which takes advantage of EXIF data stored in each file.

You are now ready to run Imatest eSFR ISO.