Documentation

Using SFRplus Part 1

The SFRplus chart: features and how to photograph it

Imatest SFRplus performs highly automated measurements of several key image quality factors using a specially-designed test chart that may be purchased from the Imatest store (recommended) or printed on a high-quality inkjet printer. Unlike most other modules, the user does not need to manually select Regions of Interest (ROIs). Image quality factors include

SFRplus operates in two modes.

  • Interactive/setup mode (run by pressing Rescharts or SFRplus setup in the Imatest main window), which allows you to select settings and interactively examine results in detail.
  • Auto mode (run by pressing SFRplus auto in the Imatest main window), which runs automatically with no additional user input. ROIs are located automatically based on settings saved from the interactive/setup mode. This allows images of different sizes and framing to be analyzed with no change of settings. Auto mode works with large batches of files, and is especially useful for automated testing, where framing may vary from image to image.

This document introduces SFRplus and explains how to obtain and photograph the chart. Part 2 shows how to run SFRplus inside Rescharts and how to save settings for automated runs. Part 3 illustrates the results.

New in Imatest 3.10 Improved SFRplus distortion calculations.
New in Imatest 3.9
SFRplus can be run without including the top and bottom bars by setting the row detection to Auto – no bars. Pre-distorted SFRplus charts are available for testing fisheye (extremely barrel-distorted) lenses.

Click on any of the thumbnails below (or on the image itself, repeatedly) to explore the many capabilities of SFRplus. Note the description below the image. Results are described in more detail in Part 3.

 
SFRplus_wellframed_640W thumbnail
3D Plot, showing MTF50 and summary results thumbnail
Multi-Region plot, including summary results thumbnail
Edge and MTF plot for center region thumbnail
Edge image with Edge and MTF mesurements for comparision thumbnail
Chromatic Aberration for lower-left region thumbnail
SQF (Subjective Quality Factor) thumbnail
Tonal response, including gamma and White Balance thumbnail
a*b* Color Error plot thumbnail
Split region color difference plot thumbnail
Lens-style MTF plot thumbnail
Lightness profiles thumbnail
Edge roughness plot thumbnail
Noise historgrams for center region thumbnail
SFRplus_wellframed_640W

Example of a well-framed and well-exposed SFRplus image.

3D Plot, showing MTF50 and summary results
Multi-Region plot, including summary results
Edge and MTF plot for center region
Edge image with Edge and MTF mesurements for comparision
Chromatic Aberration for lower-left region
SQF (Subjective Quality Factor)
Tonal response, including gamma and White Balance
a*b* Color Error plot
Split region color difference plot

Split color display (reference on upper-left, input value on lower-right), derived from color pattern below the center of the chart. Patch values are close (but not identical) to the standard 24-patch color chart. Each patch can be examined in detail with the Probe.

Lens-style MTF plot
Lightness profiles
Edge roughness plot
Noise historgrams for center region
 

The SFRplus test chart

A standard SFRplus test chart (one of many configurations) is shown below.


Standard SFRplus test chart: 5×9 grid, 10:1 and 2:1 contrasts,
20-step 4×5 stepchart (0.1 density increment), and focus star.

Sharpness is derived from light/dark slanted edges at the boundaries of the squares, as described in Sharpness: What is it and how is it measured? SFRplus can handle a wide range of camera aspect ratios; the left and right sides of the chart may extend beyond the frame or be well inside it.

The essential features of the SFRplus chart are

  • Bars at the top and bottom used for measuring distortion and facilitating edge detection.
  • A grid of slanted squares with contrasts no greater than 10:1. 4×7, 5×7, 5×9 and 7×11 (row x columns) grids are available. The tilt of the squares facilitates measurement of Lateral Chromatic Aberration using vertical edges (which are near-tangential) near the corners. This orientation also helps with the Lens-style MTF plots, which are similar to MTF displays on the Canon, Nikon, and Zeiss websites.
  • The squares above and below the middle square are reserved for non-spatial measurements. In the standard chart, the square below the middle is a 4×5 grayscale stepchart with density steps of 0.1 used to measure tonal response and gamma, and the square above the middle square contains either a fine star pattern (used as a focusing aid; not for analysis) or a 4×5 color pattern (with 18 of the 20 colors similar to the industry-standard 24-patch color chart) for measuring color accuracy.
  • A small number of low contrast (typically 2:1) squares (an option with 10:1 contrast charts) for measuring the effects of nonlinear/nonuniform signal processing. Charts may optionally have squares of a single contrast, which is helpful for 3D plots that map the performance over the image surface.
SFRplus chart options (can be selected when ordering)
  Standard Options & notes
Grid 5×9 or 5×7 5×9 is best suited for HDTV (16:9 aspect ratio) and DSLRs (3:2 aspect ratio). 5×7 is best suited for compact digital cameras and camera phones (4:3 aspect ratio). 4×7 and 7×11 are available on special request.
Contrast 10:1 – only 40:1 through 1.2:1 are available in the Test Charts module, but contrast > 10:1 is not recommended. 40:1, which was used in the old ISO 12233 chart, produced unreliable results because images were frequently saturated (clipped).
10:1 (primary), 2:1 (secondary) Most squares have 10:1 contrast; a few (typically 4) have 2:1 contrast to show the effects of nonlinear/nonuniform processing.
4:1 – only RECOMMENDED
4:1 contrast is specified in the new ISO 12233:2014 standard, and is generally recommended, especially for new work. May give lower MTF numbers than the 10:1 chart due to nonuniform signal processing.
Grayscale
stepchart
Included (below center) Not available in chrome on opal or glass charts
Focus star
or Color chart
(above center)
The focus star is normally included. The focus star is used as a focus aid (not for Imatest analysis). Omitted if color chart is included.
Optional 20-patch color chart Replaces focus star if selected. L*a*b* values will be sent in a CSV file. 18 of the 20 patches are close to the industry-standard 24-patch test chart.
Pre-distorted chart (The standard chart is undistorted.) Two levels of pincushion distortion distortion are available for use with fisheye (strongly barrel-distorted) lenses. Available in matte surface only.
Surface Matte or semigloss Semigloss is slightly sharper and has a more accurate grayscale in the deep tones (print densities > 1.5), but is more susceptible to glare (specular reflections), especially with wide angle lenses. Matte surface is recommended for wide angle lenses

Old ISO 12233 ChartAdvantages of the SFRplus chart over
the old ISO 12233:2000 chart

Note: the old ISO 12233:2000 standard and chart, shown on the right, was replaced in February, 2014, but will be widely used for some time afterwards. The new standard specifies three charts, one of which is a slanted-edge (Edge SFR; E-SFR) chart with much lower contrast (4:1; shown on the right, below). Automatic region detection with the new E-SFR chart is supported in the Imatest eSFR ISO module.

Compared to the old ISO 12233:2000 chart,

  • SFRplus has much less wasted area. About 90% of the old ISO chart is covered with patterns that have little value for computer analysis.

  • You can produce a detailed map of sharpness (MTF) over the image surface. This cannot be done with the old ISO chart because there are not enough suitable edges— and they are not well-located.

  • New ISO 12233:2014
    Slanted-edge chart

    Automated region detection, based on location criteria you enter. This makes the SFRplus chart and module well-suited for automated testing: SFRplus is now used in numerous production lines. With the ISO chart, regions of interest (ROIs) must be selected carefully whenever the image framing changes.

 

  • 10:1 (or lower) contrast edges are less likely to clip than the edges in the old ISO chart, whose contrast is specified at ≥ 40:1. The camera operates in a more linear region, and hence results are more consistent and accurate— less affected by overexposure, underexposure, incorrect gamma estimate, or excessive software sharpening. 10:1 or lower contrast edges are more representative of real edges that affect perceived image sharpness. The recommended contrast ratio of 4:1 is consistent with the new ISO 12233:2014 standard.


  • 4:1 contrast 5×7
    SFRplus chart

    The low contrast edges (2:1 contrast in a few edges in some SFRplus charts) provide additional information about signal processing in the camera under test. Although Imatest SFR is relatively insensitive to chart contrast (MTF is normalized to 100% at low spatial frequencies), measured SFR is often affected by chart contrast due to nonlinear and nonuniform signal processing in cameras, as described in the box below.

 

  • The measurement is ISO-compliant. ISO-compliant measurements do not require a standard ISO chart. The new ISO-12233:2014 E-SFR chart standard calls for a pattern with 4:1 contrast, similar to the chart shown above, right, which is fully supported by Imatest SFR.

  • SFRplus can measure additional image quality factors, including lateral chromatic aberration, distortion, gamma (contrast), tonal response, and color accuracy (in charts that have the optional color pattern).

  • The ISO 12233:2000 chart contains no information that can be used to measure gamma (the average slope of the Log pixel vs. log exposure curve. i.e., the contrast), which is needed to linearize the image for the MTF calculation. Accurate gamma measurement is particularly important with high contrast targets. In the SFRplus chart, gamma can be derived either from the individual squares (if their contrast ratio is entered) or from the grayscale step chart.

Compared to the new ISO 12233:2014 chart,

  • The map of MTF over the image surface is more detailed.
  • Distortion measurements are slightly more accurate (especially compared with the Standard chart) because the distortion bars are used.

 

Nonlinear and nonuniform signal processing SFR (MTF) measurements are often affected by chart contrast due to nonuniform signal processing, i.e., processing that depends on the contents of neighboring pixels, and hence may vary throughout an image. Nonuniform processing is almost universal in consumer digital cameras. It improves pictorial quality but complicates measurements. You can avoid it for Imatest measurements by decoding RAW images (if available) with dcraw. It takes two primary forms.

  • Sharpening, applied near contrasty features like edges. Boosts response at high spatial frequencies.
  • Noise reduction, applied in the absence of contrasty features. Attenuates response at high spatial frequencies, i.e., removes fine, low contrast detail (texture), which is interpreted as noise. Many cameras increase noise reduction at high ISO speeds.

The signal processing algorithms are proprietary; they are a part of a manufacturer’s “secret sauce” for producing pleasing images. Though they vary a great deal, the following generalizations can be made.

Most cameras do NOT apply noise reduction and sharpening uniformly throughout an image.
Contrasty edges tend to have better MTF than low contrast edges.

For this reason it is sometimes useful to photograph a chart that has both relatively high and low contrast edges: 10:1 and 2:1 in the table above.

Nonuniform signal processing can be analyzed in depth in the Log F-Contrast module.

 

SFRplus slanted-edge algorithm The algorithms for calculating MTF/SFR were adapted from a Matlab program, sfrmat, written by Peter Burns () to implement the ISO 12233 standard. SFRplus incorporates numerous improvements, including improved edge detection, better handling of lens distortion, and far more detailed output. A description can be found here. The original Matlab code is available on the I3A ISO tools download page by clicking on ISO 12233 Slant Edge Analysis Tool sfrmat 2.0. In comparing sfrmat 2.0 results with Imatest, keep the following in mind: If an OECF (tonal response curve) file is not entered into sfrmat, it assumes that there is no tonal response curve, i.e., gamma = 1. In Imatest, the default gamma is 0.5, which is typical of digital cameras. To obtain good agreement with sfrmat, you must set gamma to 1.

Obtain a test chart.

SFRplus standard SVG 5x9 square pattern
Standard 5×9 SFRplus chart, 10:1 & 2:1
contrast, focus star.


Pre-distorted 5×7 chart, 4:1 contrast, color.
(More strongly distorted charts are available.)

The standard SFRplus test chart consists of a 5×7 or 5×9 grid of squares, which may have a 4:1 contrast ratio (the current recommendation, compatible with the ISO 12233:2014 standard), a 10:1 contrast ratio, or a 10:1 contrast ratio for all but four of the squares and a 2:1 contrast ratio for the remaining four (one column off the center). A small 4×5 patch stepchart (densities in steps of 0.1 from 0.05 to 1.95) is located below the central square and a focus star (not analyzed by SFRplus) or a 20-patch color pattern is located above the center. The chart may be pre-distorted for use with fisheye (strongly barrel-distorted) lenses.

Charts can be purchased from the Imatest store in a variety of confgurations. The chart on the left below contains a color pattern (an L*a*b* reference file is included with purchase). The single-toned chart on the right below contains a star pattern, which can be used as a focus aid (not for analysis). It be produced as a chrome-on-glass transmission target in very small sizes Details here. Charts can be printed on widebody inkjet printers, but you must have fine materials, skill, and a knowledge of color management. We strongly recommend that you purchase a chart.

Charts should be mounted on 32×40 or 40 x60 inch sheets of 1/2 inch (12.5 mm) thick foam board with a spray adhesive (such as 3MTM 77 or Photo Mount) or double-sided tape (such as 3MTM #568 Positionable Mounting Adhesive). 1/2 inch foam board stays flatter than standard 1/4 or 3/8 inch foam board. PVC board is also very promising: it may be more durable and it comes in a variety of sizes.

SFRplus 5x9 color chart
5×9 SFRplus chart with color pattern
SFRplus 5x7 chart with focus star
Single-toned 5×7 SFRplus chart
with focus star

Photograph the chart

Framing requirements:

  • There should be white space above and below the bars (used to measure distortion and locate squares) at the top and bottom of the images. The white areas should be at least 0.5% and no more than 25% of the total image height. Ideally the white space should be ~1-6% of the image height. The chart should be vertically centered if possible, but this is not necessary for successful SFRplus runs.
    • Starting with Imatest 3.9 the bars may be omitted if row detection in the SFRplus setup window is set to Auto – no bars. Some care should be taken with framing since region selection acts as if there were bars at the top and bottom of the image.
  • The stepchart pattern should be centered horizontally.
  • The sides of the chart may extend beyond the image (as shown below) or be well within the image. The software is designed to accommodate a wide variety of framing and aspect ratios. Edges near the left and right boundaries will be properly located. If the sides of the chart are inside the image, you should try to minimize interfering patterns that could be mistaken for chart features. Chart surroundings inside the image should be white or light gray if possible; if that’s not possible you can crop the image in the SFRplus setup window.

SFRplus_wellframed_640WWell-framed SFRplus image
It’s OK (and quite commonplace) for the sides of the chart to run off the image.

  • The chart may be aligned using techniques shown in Building a Low-Cost Test Lab. SFRplus tolerates moderate misalignment: a tilt of ~2 degrees, some perspective (keystone) distortion, etc., but every effort should be made to align the chart properly. Moderate barrel or pincushion distortion (<8% SMIA) is also well tolerated, but a pre-distorted chart should be used with extreme fisheye lenses (with high barrel distortion).
  • The image can be cropped (starting with) to remove interfering features near the edges, using the Crop borders button in the SFRplus setup window.

If exposure compensation is available, increasing exposure by +1 f-stop often produces a better exposure, especially on the grayscale stepchart pattern.

Bad framing
Badly cropped image
Interfering patterns near borders
(Can be cropped.)
Good framing
Well cropped image
Same pattern; interfering patterns masked out
(of course corners won’t be measured.)
Badly cropped image
Missing white space above top distortion bar
Good crop
Some tilt, distortion tolerated
OK for sides of chart to run off image

 

Starting with Imatest 3.9, the top and bottom bars may be omitted if Auto – no bars is selected in the row detection menu in the SFRplus setup window. Region selection will behave as if the bars were located at the top and bottom of the image. For this reason the top of the image should correspond to the top of a row of squares and the bottom should correspond to the bottom of a row of squares, as shown in the example on the right (which isn’t perfect but is close enough).

No bars

Lighting

The chart below summarizes lighting considerations. The goal is even, glare-free illumination. Lighting angles between 25 and 45 degrees are ideal in most cases. At least two lights (one on each side) is recommended; four or six is better. Avoid lighting behind the camera, which can cause glare. Check for glare and lighting uniformity before you expose. Matte charts are recommended for wide angle lenses; it’s very difficult to control glare with semigloss surfaces.

A detailed description of the recommended lighting setup, which uses six high quality (CRI > 98) 4700K (near-daylight) 50W SoLux quartz-halogen lamps, can be found in Building a Low-Cost Test Lab. The BK Precision 615 Light meter (Lux meter) is an outstanding low-cost instrument (about $100 USD) for measuring the intensity and uniformity of illumination.

Simplified lighting diagram

Distance

Distance and field of view
The camera must be far enough from the chart so you are measuring the sharpness of your camera and lens, not the chart. But remember,

It’s the field of view, not the chart-to-camera distance, that counts.

A rough rule of thumb: For an inkjet-printed chart the field of view should be at least
24 inches (60 cm) for a 6-megapixel camera;
40 inches (90 cm) for a 16 megapixel camera.
Details below. For a high-quality photographically-printed chart (such as the charts from Applied Imaging) you can get quite a bit closer.

A letter-sized (8.5×11 inch) chart printed on Premium Luster paper on the Epson 2200 (a high quality pigment-based inkjet photo printer) was analyzed for MTF using the 6.3 megapixel Canon EOS-10D. There was no change when the image field was at least 22 inches (56 cm) wide— twice the length of the chart. Performance fell off slowly for smaller fields.

Choose a camera-to-target distance that gives at least this image field width. The actual distance depends on the sensor size and the focal length of the lens. The minimum image field is illustrated on the right.

Cameras with more pixels, and hence higher potential resolution, should have a larger image field width, hence printed chart width.

Distance/field width guidelines for high quality inkjet charts
(You can get closer with photographically-printed charts.)

The camera-to-target distance is not critical as long as it is greater than a reasonable minimum.
 
Sensor sizes
Desig-
nation
Diagonal
mm.
Width
mm.
Height
mm.
1/3.6″ 5.0 4.0 3.0
1/3.2″ 5.68 4.54 3.42
1/3″ 6.0 4.8 3.6
1/2.7″ 6.59 5.27 3.96
1/2″ 8.0 6.4 4.8
1/1.8″ 8.93 7.18 5.32
2/3″ 11.0 8.8 6.6
1″ 16.0 12.8 9.6
4/3″ 22.5 18.0 13.5
35mm 44.3 24.0 36.0
Image field width (in inches) > 8.8 × sqrt(megapixels)
Image field width (in cm) > 22 × sqrt(megapixels) — or —There should be no more than 140 sensor pixels per inch of target or 55 sensor pixels per centimeter of the target. — or —The distance to the target should be at least 40X the focal length of the lens for 6-10 megapixel digital SLRs.(25X is the absolute minimum for 6 megapixel DSLRs; 40X leaves some margin.) For compact digital cameras, which have much smaller sensors, the distance should be at least 100X the actual focal length: the field of view is about the same as an SLR with comparable pixel count. The recommended distance is described in more detail in Chart quality and distance, below.
There is some confusion about lens focal lengths in compact digital cameras. They are often given as the “35mm-equivalent,” which many photographers can relate to viewing angle. 35-105mm or 28-140mm are typical “35mm-equivalent” numbers, but they are not the true lens focal length, which is often omitted from the specs. What is given is the sensor size in 1/n inches, a confusing designation based on the outside diameter of long-obsolete vidicon tubes. It The table on the right relates the 1/ndesignation to the diagonal dimension of the sensor. True focal length = “35mm-equivalent” × (diagonal mm.) / 44.3

Exposure

Good exposure is important for accurate SFRplus results. Neither the black nor the white regions of the chart should clip— have substantial areas that reach pixel levels 0 or 255. The best way to ensure proper exposure is to use the histogram in your digital camera. Blacks (the peaks on the left) should be above the minimum and whites (the peak(s) on the right) should be below the maximum.

The above image (taken from the Canon File Viewer Utility) is close to a perfect exposure. Some exposure compensation, typically around +1 f-stop, may be helpful.

Tips on photographing the chart
Distance doesn’t matter as long as the target far enough from the camera so sharpness is limited by the camera and lens, not by the target. A Wide body printer (capable of printing images at least 24 inches high) are required to print the SFRplus chart.
The target should be evenly lit and free of glare.
White balance should be approximately neutral.
Use a sturdy tripod and a cable release. For DSLRs, use the mirror lock if possible. You can use Imatest SFR to find the difference made by a good tripod or mirror lock— to sharpen your technique, literally ( pun intended ).
Be sure to expose the image so detail is maintained in both light and dark areas. Neither should be blocked (clipped). Use your camera’s histogram. If more than 0.5% of the pixels are at levels 0 or 255, Imatest SFR will assume that clipping has taken place and issue a warning message. This has no effect on the calculations— it’s just a warning that accuracy may be compromised.
Be sure the camera is correctly focused on the chart.
Place slanted-edge images near the corners of the field as well as near the center.
You may find it instructive to photograph the slanted edge target along with a target from Lens testing, but there’s no need to do so.

Save the image as a RAW file or maximum quality JPEG. If you are using a RAW converter, convert to JPEG (maximum quality), TIFF (without LZW compression, which is not supported), or PNG. If you are using film, develop and scan it.

If the folder contains meaningless camera-generated file names such as IMG_3734.jpg, IMG_3735.jpg, etc., you can change them to meaningful names that include focal length, aperture, etc., with the View/Rename Files utility, which takes advantage of EXIF data stored in each file.

You are now ready to run Imatest SFRplus.