This post addresses concerns about the sensitivity of slanted-edge patterns to signal processing, especially sharpening, and corrects the misconception that sinusoidal patterns, such as the Siemens star (included in the ISO 12233:2014 standard), are insensitive to sharpening, and hence provide more robust and stable MTF measurements.
To summarize our results, we found that the Siemens Star (and other sinusoidal patterns) are nearly as sensitive as slanted-edges to sharpening, and that slanted-edges give reliable MTF measurements that correspond to the human eye’s perception of sharpness.
The relatively high contrast of the Siemens Star (specified as >50:1 in the ISO standard) makes measurement results somewhat sensitive to clipping and nonlinearities in tonal response. Low contrast slanted-edges (4:1 in the ISO standard) closely approximate features our eyes use to perceive sharpness, and they rarely saturate images. Measurements based on slanted-edges accurately represent real-world performance and are well suited for testing of cameras in wide range of applications, from product development to manufacturing.
Full details of the comparison are in the Slanted-Edge versus Siemens Star and the Slanted-Edge versus Siemens Star, Part 2 pages, where we compare MTF calculation results from slanted-edge patterns, which are performed by Imatest SFR, SFRplus, and eSFR ISO, to results from sinusoidal patterns, which are performed by the Imatest Star chart (for the Siemens star) and Log F-Contrast modules, paying special attention to the effects of signal processing, especially sharpening and noise reduction (which are performed by software or in-camera firmware).
Most consumer cameras sharpen images in the presence of contrasty features (like edges) and reduce noise (lowpass filtering; the opposite of sharpening) in their absence. The details of this processing are more complex, or course, and vary greatly for different cameras. For this reason we tested six cameras. The primary example is a camera whose image sensor and pixel size is between that of camera phones and DSLRs: the Panasonic Lumix LX7 (a high-end point-and-shoot with an excellent Leica zoom lens and optional raw output). We also analyze a camera phone with excessive sharpening and four additional cameras.
Quick summary: Raw MTF results are nearly independent of chart type, contrast level, and ISO speed.
Raw images were converted to demosaiced TIFF files using dcraw, which applies no sharpening or noise reduction. Output gamma was set to 1.0 (Linear; i.e., no encoding gamma). No signal processing was applied apart from demosaicing. The converted raw images can be considered to be “original” unprocessed images.
In comparing raw results from slanted edges of varying contrast (2:1, 4:1, and 10:1) with images of the Log F-Contrast and Siemens Star charts, we found that raw results are nearly independent of chart type, contrast level, and ISO speed.
|Raw image @ ISO 80; >10:1 edge
The monotonically-decreasing shape of the MTF curve (clearer on the less noisy image on the left) is characteristic of unsharpened images. Sharpened images have a characteristic boost in the middle frequencies (around half the Nyquist frequency) described in more detail here. We will use this curve for comparison with sharpened images, below.
This is typical of consumer cameras, which tend to have more sharpening near contrasty features and less sharpening (often none at all, or noise reduction (lowpass filtering)— which is the inverse of sharpening) in the absence of contrasty features. The sharpening boost is very clear for slanted-edges (and indeed represents what happens to edges in real world images). Next we look at sharpening in sinusoidal images (Log F-Contrast and the Siemens Star).
Both the Log F-Contrast chart and the Siemens Star have sinusoidal patterns, and hence have similar MTF response characteristics (at similar contrast levels). The key differences:
- The Log F-Contrast measures MTF in a single direction (horizontal), but has a range of contrast levels.
- The Siemens Star measures MTF in multiple directions, but has a single (high) contrast level (specified as > 50:1 ISO 12233:2014).
MTF for Log F-Contrast, rows 1-19 in steps of 3
(of 25), Camera JPEG @ ISO 80
MTF contours for Log F-Contrast image,
Camera JPEG @ ISO 80
The two plots (above) display the same MTF results for the Log F-Contrast chart in two different formats: actual MTF curves (above left) and MTF contour plots (above right). The contour plots are clearer for observing the change in response with contrast, but the MTF curves are better for comparing with the Siemens Star chart results on the right. Sharpening is present in rows with modulation levels greater than 0.1. it drops very rapidly for modulation below 0.1: it’s mostly gone by Row 19 (modulation = 0.06).
These results show that a significant amount of sharpening is present for high contrast sinusoidal patterns.
A camera phone with extreme sharpening
In early 2013 we received images from a camera phone that had an extreme— and highly atypical— amount of sharpening. We described the results of testing this image in Dead Leaves measurement issue. We did not receive a Siemens star image, but, as we have shown in Slanted-Edge versus Siemens Star, the response of Row 1 of the sinusoidal Log F-Contrast chart should be very close to the response of the (high contrast) Siemens Star. Here are the Log F-Contrast results.
|MTF for extremely sharpened image:
Log F-Contrast, rows 1-19 in steps of 3 (of 25),
Camera JPEG @ ISO 80
|MTF contours for extremely sharpened image:
Log F-Contrast, Camera JPEG @ ISO 80
For this camera, the MTF of the sinusoidal Log F-Contrast chart (whose response for the high contrast rows near the top should be similar to the Siemens Star pattern) is remarkably sensitive to sharpening— more than expected. The maximum peak response of 2.8 was for modulation = 0.3 (approximately 2:1 contrast). For higher contrast levels (rows 1-10) sharpening causes the Log F-Contrast chart to saturate, i.e., clip (go to pure white). This saturation is visible in the flat tops of the MTF curves (above-left) and in a crop of the image in the Slanted-Edge versus Siemens Star documentation page.
The Siemens Star, which has high contrast (>50:1; comparable to the top row of the Log F-Contrast chart), would have a lower MTF peak and a higher MTF50 for this camera because of strong saturation, not because it is less sensitive to signal processing. Measurements would look unrealistically good for this system, despite the severe (and highly visible) oversharpening.
Our perception of image sharpness is closely correlated to the appearance of edges (contrast boundaries) in the image. This is a key reason for the use of edges in MTF measurement (the slant makes the result insensitive to sampling phase). Saturation was often a problem with the edges in the old ISO 12233:2000 chart, which had a specified minimum contrast of 40:1 (80:1 or more was typical). This problem has been resolved in the new ISO 12233:2014 standard (released in February 2014; implemented in SFRplus and ISO 12233:2014 Edge SFR charts), which specified an edge contrast of 4:1. Not only are these edges unlikely to saturate, their contrast is similar to real-world features that affect our perception of sharpness. Measurements based on low-contrast slanted-edges accurately represent imaging system performance.
|Results from the six cameras we tested (two in Slanted-Edge vs. Siemens Star (Part 1) and four in Part 2) show that MTF results for both slanted-edges and Siemens stars (as well at other sinusoidal patterns) are sensitive to sharpening (and that there tends to be more sharpening for higher contrast features, although some cameras may reduce sharpening at the very highest contrast levels). We also found that the Siemens star was more affected by saturation and non-uniform signal processing in high contrast regions. Saturation tends to flatten the Siemens star MTF response— to reduce peak MTF, making it appear that it is less sensitive to sharpening, when in fact the MTF measurement is questionable.
The sensitivity of slanted edges to sharpening is not a drawback because it is not excessive and it represents the way we perceive image sharpness.
|More efficient use of space (allows a detailed map of sharpness over the image surface).
Robust in the presence of optical distortion.
|Slightly more sensitive to software sharpening, depending on camera firmware.
Measures only near-horizontal and vertical edges.
|Slightly less sensitive to software sharpening.
Measures MTF for a variety of angles (not just near-horizontal and vertical), though these measurements seem to be affected only by demosaicing and sharpening.
|Requires more space.
Saturation in highly oversharpened images is not obvious from measurement results.
|The slanted-edge pattern provides accurate and robust measurements.
Its speed and efficient use of space (which allows detailed sharpness maps) makes it the best choice for measuring the performance of a wide variety of camera systems and applications, from product development to production testing.