Making Dynamic Range Measurements Robust Against Flare Light

Introduction

A camera’s Dynamic Range (DR) is the range of tones in a scene that can be reproduced with adequate contrast and good signal-to-noise ratio (SNR). Camera DR is often limited by flare light, which is stray light in the image, primarily caused by reflections between lens elements. Flare light reduces DR by fogging images; i.e., washing out detail in dark areas. It is the primary reason that the DR of cameras (which include lenses) is poorer than that of image sensors, which can be up to 150dB (30 million:1) for recent HDR (high dynamic range) sensors.

In the past 2 years we’ve learned that flare light in test chart images could, in theory, be mistaken for the image of the test chart, resulting in exaggerated DR measurements. The situation became real in mid-2019, when we started seeing actual images where this error occurred. By studying these images, we have developed techniques to guard against exaggerated Signal-to-Noise Ratio (SNR) measurements caused by flare light. We describe these techniques here. (more…)

Read More

Correcting nonuniformity in slanted-edge MTF measurements

Slanted-edge regions can often have non-uniformity across them. This could be caused by uneven illumination, lens falloff, and photoresponse nonuniformity (PRNU) of the sensor. 

Uncorrected nonuniformity in a slanted-edge region of interest can lead to an irregularity in MTF at low spatial frequencies. This disrupts the low-frequency reference which used to normalize the MTF curve. If the direction of the nonuniformity goes against the slanted edge transition from light to dark, MTF increases. If the nonuniformity goes in the same direction as the transition from light to dark, MTF decreases. 

To demonstrate this effect, we start with a simulated uniform slanted edge with some blur applied.

Then we apply a simulated nonuniformity to the edge at different angles relative to the edge. This is modeled to match a severe case of nonuniformity reported by one of our customers:

 

Here is the MTF obtained from the nonuniform slanted edges:

If the nonuniformity includes an angular component that is parallel to the edge, this adds a sawtooth pattern to the spatial domain, which manifests as high-frequency spikes in the frequency domain. This is caused by the binning algorithm which projects brighter or darker parts of the ROI into alternating bins.

 

Compensating for the effects of nonuniformity

Although every effort should be made to achieve even illumination, it’s not always possible (for example, in medical endoscopes and wide-FoV lenses).

Imatest 4.5+ has an option for dealing with this problem for all slanted-edge modules (SFR and Rescharts/fixed modules SFRplus, eSFR ISO, SFRreg, and Checkerboard). It is applied by checking the “Nonuniformity MTF correction” checkbox in the settings (or “More” settings) window, shown on the right.

When this box is checked, a portion of the spatial curve on the light side of the transition (displayed on the right in Imatest) is used to estimate the nonuniformity. The light side is chosen because it has a much better Signal-to-Noise Ratio than the dark side. In the above image, this would be the portion of the the edge profile more than about 6 pixels from the center. Imatest finds the first-order fit to the curve in this region, limits the fit so it doesn’t drop below zero, then divides the average edge by the first-order fit. 

The applied compensation flattens the response across the edge function and significantly improves the stability of the MTF:

Summary

For this example, Imatest’s nonuniformity correction reduces our example’s -26.0% to +22.8% change in MTF down to a -3.5% to +4.7% change. This is an 83% reduction in the effect of the worst cases of nonuniformity.

MTF50 versus nonuniformity angle without [blue] and with [orange] nonuniformity correction

While this is a large improvement, the residual effects of nonuniformity remain undesirable. Because of this, we recommend turning on your ISP’s nonuniformity correction before performing edge-SFR tests or averaging the MTF obtained from nearby slanted edges with opposite transition directions relative to the nonuniformity to reduce the effects of nonuniformity on your MTF measurements further.

Detailed algorithm

We assume that the illumination of the chart in the Region of Interest (ROI) approximates a first-order function, L(d) = k1 + k2d, where d is the horizontal or vertical distance nearly perpendicular to the (slanted) edge. The procedure consists of estimating k1 and k2, then dividing the linearized average edge by L(d). 

k1 and k2, are estimated using the light side of the transition starting at a sufficient distance dN from the transition center xcenter, so the transition itself does not have much effect on the k1 and k2 estimate. To find dN we first find the 20% width d20 of the line spread function (LSF; the derivative of the edge), i.e., the distance between the points where the LSF falls to 20% of its maximum value. 

dN = xcenter + 2 d20 

If the edge response for x > dN has a sufficient number of points, it is used to calculate k1 and k2 using standard polynomial fitting techniques. The result is a more accurate representation of the edge with the effects of nonuniformity reduced.

Future work

  • Consider the 2D nonuniformity across the ROI before sampling the 1D average edge
  • Use an image of a flat-field to perform nonuniformity correction within Imatest
  • Consider the impact of noise which was not included in this study
  • Incorporate enhancements to the slanted-edge algorithms into future revisions of ISO 12233

 

For any questions on how to do this, or how we can help you with your projects, contact us at support@imatest.com.

 

Read More

Measuring temporal noise

Temporal noise is random noise that varies independently from image to image, in contrast to fixed-pattern noise, which remains consistent (but may be difficult to measure because it is usually much weaker than temporal noise). It can be analyzed by Colorcheck and Stepchart and was added to Multicharts and Multitest in Imatest 5.1.

It can be calculated by two methods.

  1. the difference between two identical test chart images (the Imatest recommended method), and 
     
  2. the ISO 15739-based method, which where it is calculated from the pixel difference between the average of N identical images (N ≥ 8) and each individual image.

In this post we compare the two methods and show why method 1 is preferred.

(1) Two file difference method. In any of the modules, read two images. The window shown on the right appears. Select the Read two files for measuring temporal noise radio button.

The two files will be read and their difference (which cancels fixed pattern noise) is taken. Since these images are independent, noise powers add. For indendent images I1 and I2, temporal noise is

\(\displaystyle \sigma_{temporal} = \frac{\sigma(I_1 – I_2)}{\sqrt{2}}\)

In Multicharts and Multitest temporal noise is displayed as dotted lines in Noise analysis plots 1-3 (simple noise, S/N, and SNR (dB)).

(2) Multiple file method. From ISO 15739, sections 6.2.4, 6.2.5, and Appendix A.1.4. Available in  Multicharts and Multitest. Currently we are using simple noise (not yet scene-referred noise). Select between 4 and 16 files. In the multi-image file list window (shown above) select Read n files for temporal noise. Temporal noise is calculated for each pixel j using

\(\displaystyle \sigma_{diff}(j) = \sqrt{ \frac{1}{N} \sum_{i=1}^N (X_{j,i} – X_{AVG,j})^2} = \sqrt{ \frac{1}{N} \sum_{i=1}^N X_{j,i}^2 – \left(\frac{1}{N} \sum_{i=1}^N X_{j,i}\right)^2 } \) 

The latter expression is used in the actual calculation since only two arrays, \(\sum X_{j,i} \text{ and } \sum X_{j,i}^2 \), need to be saved. Since N is a relatively small number (between 4 and 16, with 8 recommended), it must be corrected using formulas for sample standard deviation from Identities and mathematical properties in the Wikipedia standard deviation page as well as Equation (13) from ISO 15739.  \(s(X) = \sqrt{\frac{N}{N-1}} \sqrt{E[(X – E(X))^2]}\).

\(\sigma_{temporal} = \sigma_{diff} \sqrt{\frac{N}{N-1}} \) 

We currently recommend the difference method (1) because our experience so far has shown no advantage to method (2), which requires many more images (≥ 8 recommended), but allows fixed pattern noise to be calculated at the same time.

To calculate temporal noise with either method, read the appropriate number of files (2 or 4-16) then push the appropriate radio button on the multi-image settings box.

Multi-image settings window, showing setting for method 1.
if 4-16 images are enterred, the setting for method 2 (Read n files…) will be available.

Results for the two methods

The two methods were compared using identical Colorchecker images taken on a Panasonic Lumix LX5 camera (a moderately high quality small-sensor camera now several years old).

Difference method (1) (two files)

Here are the Multicharts results for 2 files.

Multicharts SNR results for temporal noise, shown as thin dotted lines in the lower plot

Multi-file method (2) (4-16 files)

 

Here are results (SNR (dB)) for runs with 4, 8, and 16 files.

For 4 files, temporal SNR (thin dotted lines) is slightly better than standard noise. Temporal SNR is slightly lower for 8 files and very slightly lower for 16 files.

For 8 and 16 files results are closer to the results for 2 files (though differences between 8 and 16 files are very small).

The bottom line: We recommend the two-file (difference) method because it is accurate and relatively fast. The multi file method is slower for acquiring and analyzing images— at least 8 images are recommended, so why bother?

Temporal SNR from 4 images
Temporal SNR from 8 images
Tempoal SNR from 16 images
Read More

Color difference ellipses

Imatest has several two-dimensional displays for comparing test chart reference (ideal) colors with measured (camera) colors, where reference colors are represented by squares and measured values are represented by circles. The two most familiar representations— CIELAB a*b* and CIE 1931 xy chromaticity— are shown below. They are for the Colorchecker image, also shown below, where the upper-left of each patch is the reference color and the lower-right is the camera color. (more…)

Read More

SFRreg: SFR from Registration Marks

Imatest SFRreg performs highly automated measurements of sharpness (expressed as Spatial Frequency Response (SFR), also known as Modulation Transfer Function (MTF)) and Lateral Chromatic Aberration from images that contain registration mark patterns (circles with two light and two dark quadrants). Unlike standard test charts, these patterns do not need to be located on a flat (planar) surface. Depending on the image source, they offer two advantages. You can

  • Test images at infinity distance (or any distance of choice) using a compact projection system such as the Optikos Meridian camera test system.
  • Test the sharpness of extreme fisheye lenses (with angular fields of view over 180 degrees, whose MTF cannot be measured near the image boundaries with a single flat target) using an array test charts, each consisting of an individual registration mark. Registration mark charts, such as the one shown on the right, may be purchased from the Imatest store or printed on a high-quality inkjet printer. Since Region of Interest (ROI) selection is automatic, they may be positioned where needed. They work best when facing the camera.
registration_markRegistration mark

Details of the regions to be analyzed are based on user-entered criteria (similar to SFRplus or eSFR ISO, which it closely resembles).

Sharpness is derived from light/dark slanted edges inside the registration marks, as described in Sharpness: What is it and how is it measured? SFRreg can handle a wide range of camera aspect ratios and chart arrangements.

SFRreg operates in two modes.

  • Interactive/setup mode  allows you to select settings and interactively examine results in detail. Saved settings are used for Auto Mode.
  • Auto mode  runs automatically with no additional user input. ROIs are located automatically based on settings saved from the interactive/setup mode. This allows images of different sizes and framing to be analyzed with no change of settings. Auto mode works with large batches of files, and is especially useful for automated testing, where framing may vary from image to image.

Part 1 of the instructions introduces SFRreg and explains how to obtain and photograph the chart. Part 2 shows how to run SFRreg inside Rescharts and how to save settings for automated runs. Part 3 illustrates the results.

 

SFRreg images from the Optikos Meridian projection system

Imatest SFRreg is originally designed designed to work with images from the Optikos Meridian system, which consists of several projectors that project registration mark patterns towards a camera. These patterns appear at infinity focus at the camera. A typical image is shown below.

MERIDIAN-7-4K-e1423515004804

Optikos_9_proj_640WImage acquired from a 9-projector Optikos Meridian system

SFRreg images from arbitrary arrays of printed registration mark charts

SFRreg also works with printed registration mark patterns, which can be placed anywhere in the image. For extreme wide-angle (fisheye) lenses they should be oriented facing directly towards the camera. Here is a synthesized image (we’ll add a real one soon). You can add other charts— typically color or grayscale— to the image for additional measurements.

reg_marks_fisheye_fakeFisheye lens image with synthesized registration mark charts oriented facing the camera

SFRreg chart print options (can be selected when ordering)

  Options Notes
Media Inkjet (reflective),
LVT film (transmissive)
Inkjet (reflective) is usually the most practical choice.
Contrast
4:1, 10:1
4:1 contrast is specified in the new ISO 12233:2014 standard.
Surface Matte or semigloss Semigloss is slightly sharper, but is more susceptible to glare (specular reflections), especially with wide angle lenses. Matte surface is recommended for wide angle lenses or difficult lighting situations.

 

Slanted-edge algorithm The algorithms for calculating MTF/SFR were adapted from a Matlab program, sfrmat, written by Peter Burns () to implement the ISO 12233:2000 standard. Imatest SFR, SFRplus, SFRreg, and eSFR ISO incorporates numerous improvements, including improved edge detection, better handling of lens distortion, a nicer interface, and far more detailed output. The original Matlab code is available on http://losburns.com/imaging/software/SFRedge/index.htm. In comparing sfrmat results with Imatest, note that if no OECF (tonal response curve) file is entered into sfrmat, no tonal response curve is assumed, i.e., gamma = 1 (linear response). Since the default value of gamma in Imatest is 0.5, which is typical of digital cameras, you must set gamma to 1 to obtain good agreement with sfrmat.

Obtaining and photographing the charts

Registration Mark charts can be purchased from the Imatest store in a variety of inkjet media (reflective and transmissive) (Other media will be available on request.) Although we recommend that you purchase the charts, they can be printed on photographic-quality inkjet printers, but you must have fine materials, skill, and a knowledge of color management.

 

SFRreg results

When calculations are complete, results are displayed in the Rescharts window, which allows a number of displays to be selected. The following table shows where specific results are displayed. Results plots are very similar to SFRplus and eSFR ISO. We show two samples of results below.

sfrreg_display_selectionSFRreg display selections

Measurement Display
MTF (sharpness) for individual regions 1. Edge and MTF
MTF (sharpness) for entire image 4. Multi-ROI summary
12. 3D plot
13. Lens-style MTF plot
Lateral Chromatic Aberration 2. Chromatic Aberration
Original image showing region selection 8. Image & geometry
EXIF data 7. Summary & EXIF data
Acutance/SQF (Subjective Quality Factor) 3. SQF / Acutance
Edge roughness 14. Edge roughness
Chromatic Aberration (radial) 15. Radial (Chr Aber, etc.)

Multi-ROI summary display

 

sfrreg_multi_regionSFRreg results in Rescharts window: Multiple region (ROI) summary
(Only upper Vertical regions have been selected to keep the view uncluttered.)

The multi-ROI (multiple Region of Interest) summary shown in the Rescharts window (above) contains a detailed summary of SFRreg results. (3D plots also contain an excellent summary.) The upper left contains the image in muted gray tones, with the selected regions surrounded by red rectangles and displayed with full contrast. Up to four results boxes are displayed next to each region. The results are selected the the Display options area on the right of the window, below the Display selection.

The Results selection (right) lets you choose which results to display. N is region number. Ctr-corner distance % is the approximate location of the region. CA is Chromatic Aberration in area, as percentage of the center-to-corner distance (a perceptual measurement). A legend below the image shows which results are displayed.

The View selection (far right) lets you select how many results boxes to display, which can be helpful when many regions overlap. From top to bottom the number of boxes is 4, 3, 2, 2, and 1, respectively.


Results selection
View selection

Edge and MTF display

 

sfrreg_edge_MTFEdge and MTF display in Rescharts window
Diffraction-limited MTF and edge response are shown as a pale brown dotted lines
when pixel spacing (5.7um for the EOS-40D) has been entered.

This display is identical to the SFR Edge and MTF display. The edge (or line spread function) is plotted on the top and the MTF is plotted on the bottom. The edge may be displayed linearized and normalized (the default; shown), unlinearized (pixel level) and normalized, or linearized and unnormalized (good for checking for saturation, especially in images with poor white balance). Edge display is selected by pressing More settings.

There are a number of readouts, including 10-90% rise distance, MTF50, MTF50P (the spatial frequency where MTF is 50% of the peak value, differing from MTF50 only for oversharpened pulses), the secondary readouts (MTF @ 0.125 and 0.25 C/P in this case), and the MTF at the Nyquist frequency (0.5 cycles/pixel). The diffraction-limited MTF curve (not shown above) is displayed as a pale brown dotted line when pixel pitch is entered.

MTF is explained in Sharpness: What is it and how is it measured? MTF curves and Image appearance contains several examples illustrating the correlation between MTF curves and perceived sharpness.

 

Read More

Measuring Multiburst pattern MTF with Stepchart

Measuring MTF is not a typical application for Stepchart— certainly not its primary function— but it can be useful with multiburst patterns, which are a legacy from analog imaging that occasionally appear in the digital world. The multiburst pattern is not one of Imatest’s preferred methods for measuring MTF: see the MTF Measurement Matrix for a concise list. But sometimes customers need to analyze them. This feature is available starting with Imatest 4.1.3 (March 2015).

(more…)

Read More

LSF correction factor for slanted-edge MTF measurements

A correction factor for the slanted-edge MTF (Edge SFR; E-SFR) calculations in SFR, SFRplus, eSFR ISO, SFRreg, and Checkerboard was added to Imatest 4.1.2 (March 2015). This correction factor is included in the ISO 12233:2014 standard, but is not in the older ISO 12233:2000 standard. Because it corrects for an MTF loss caused by the numerical calculation of the Line Spread Function (LSF) from the Edge Spread Function (ESF), we call it the LSF correction factor.

The LSF correction factor primarily affects very high spatial frequencies beyond most of the energy for typical high quality cameras. But it does make a difference for practical measurements: MTF50 for a typical high quality camera (shown below) is increased by about 1.5%. (more…)

Read More

Slanted-Edge versus Siemens Star: A comparison of sensitivity to signal processing

This post addresses concerns about the sensitivity of slanted-edge patterns to signal processing, especially sharpening, and corrects the misconception that sinusoidal patterns, such as the Siemens star (included in the ISO 12233:2014 standard), are insensitive to sharpening, and hence provide more robust and stable MTF measurements.

To summarize our results, we found that the Siemens Star (and other sinusoidal patterns) are nearly as sensitive as slanted-edges to sharpening, and that slanted-edges give reliable MTF measurements that correspond to the human eye’s perception of sharpness. (more…)

Read More

Sharpness and Texture Analysis using Log F‑Contrast from Imaging-Resource

Imaging-resource.com publishes images of the Imatest Log F-Contrast* chart in its excellent camera reviews. These images contain valuable information about camera quality— how sharpness and texture response are affected by image processing— but they need to be processed by Imatest to reveal the important information they contain.

*F is an abbreviation for Frequency in Log F-Contrast.

(more…)

Read More

Measuring Test Chart Patches with a Spectrophotometer

 Using Babelcolor Patch Tool or SpectraShop 4

This post describes how to measure color and grayscale patches on a variety of test charts, including Imatest SFRplus and eSFR ISO charts, the X-Rite Colorchecker, ISO-15729, ISO-14524, ChromaDuMonde CDM-28R, and many more, using a spectrophotometer and one of two software packages.

Measurement results are stored in CGATS files, which can be used as reference files for grayscale and color chart analysis in Multicharts, Multitest, Colorcheck, Stepchart, and SFRplus. In many cases, custom reference files provide more accurate results than the default values.

In addition, Argyll CMS is a free, Open Source, command line-based package that can be used for a number of measurements, including illumination intensity and spectrum. See the Argyll CMS documentation for more details. (more…)

Read More

Region Selection bug workaround

Symptoms of problem:

Upon selection of a region of interest, program stops working, either not responding or crashing.

DOS window error messages: 

  • Operation terminated by user during pause (line 21) In newROI_quest
  • Undefined function or method ‘figure1_KeyPressFcn’ for input arguments of type ‘struct’.
  • Error while evaluating figure KeyPressFcn – Segmentation violation detected

Source of problem:  

automatic translation software such as 有道首页 (Youdao Dictionary) and 金山词霸 (PowerWord)

Solution to problem: 

Temporarily disable the translation software while performing a ROI selection

We will be working to make our software compatible with these sorts of translation programs in the future, as well as improving our own internationalization.  Sorry for the inconvenience.

 

Read More

Microsoft Lync Logo selects Imatest

Imatest is the standard test software used in the recently published Microsoft USB peripheral requirements specification entitled “Optimized for Microsoft Lync Logo”, which can be downloaded here.

Read More

Slanted Edge Noise reduction (Modified Apodization Technique)

For measurement of sharpness, the main driver of variation is noise. A powerful noise reduction technique called modified apodization is available for slanted-edge measurements (SFR, SFRplus, eSFR ISO and SFRreg). This technique makes virtually no difference in low-noise images, but it can significantly improve measurement accuracy for noisy images, especially at high spatial frequencies (f > Nyquist/2). It is applied when the MTF noise reduction (modified apodization) checkbox is checked in the SFR input dialog box or the SFRplus or eSFR ISO More settings window.

Note that we recommend keeping it enabled even though it is NOT a part of the ISO 12233 standard. If the ISO standard checkbox is checked (at the bottom-left of the dialog boxes), noise reduction is not applied.

The strange word apodization* comes from “Comparison of Fourier transform methods for calculating MTF” by Joseph D. LaVigne, Stephen D. Burks, and Brian Nehring of Santa Barbara Infrared. The fundamental assumption is that all-important detail (at least for high spatial frequencies) is close to the edge. The original technique involves setting the Line Spread Function (LSF) to zero beyond a specified distance from the edge. The modified technique strongly smooths (lowpass filters) the LSF instead. This has much less effect on low-frequency response than the original technique and allows tighter boundaries to be set for better noise reduction.

*Pedicure would be a better name for the new technique, but it might confuse the uninitiated.

Modified apodization noise reduction- explanation
Modified apodization: original noisy averaged Line Spread Function (bottom; green),
smoothed (middle; blue), LSF used for MTF (top; red)

 

As of version 5.0.17, additional filtering to increase smoothness of MTF curve in noisy images has been applied.

Read More

Infrared SFRplus test charts available now

The Infrared SFRplus test chart enables accurate, objective testing of infrared camera systems and supports wavelengths from visible light through MWIR.

Details here .

Purchase here .

Read More

Toshiba Collaborates with Imatest on IS Edition

Toshiba America Electronic Components, Inc. (TAEC), a committed leader that collaborates with technology companies to create breakthrough designs, and Imatest LLC, maker of the world’s most popular image quality testing software, have teamed up to enhance Imatest’s Image Sensor (IS) Edition, a highly anticipated and powerful tool to help engineers responsible for configuring Toshiba image sensors.

Details here

Read More

Imatest Users Group on LinkedIn

Join the new Imatest users group to share questions and answers with other image quality testing professionals. Join here

Read More

Blemish Detection Now Available

Version 3.6 of Imatest’s Master Edition delivers sophisticated blemish detection capabilities based on the human visual system’s ability to detect blemishes. Soon this feature will also be added to the Industrial Testing Edition to enable blemish detection in high-volume camera-system production. Details here

Read More

Image Sensor Edition Available Now

Imatest’s much anticipated Image Sensor Edition is now available. Details here

Read More

Imatest expands consulting services

Imatest has expanded its capabilities and staffing to better meet customer demand for consulting services. Now offering skills and experience in imaging science, optics, image quality testing, camera system design, software development, and project management, Imatest looks to play an increasing role in helping customers deliver leading-edge imaging systems. Details here

Read More

Release 3.5 includes ISO sensitivity analysis

ISO sensitivity (also called ISO speed), is included in Release 3.5 of both Imatest Master and Industrial Testing (IT) packages. ISO sensitivity is a measure of how strongly an image sensor and/or camera responds to light. Both Saturation-based and Standard Output Sensitivity, as described in ISO 12232, are calculated, but the measurement procedure has been considerably streamlined as compared to the ISO specification. The calculation is included in four Imatest modules– Stepchart, Colorcheck, Multicharts, and SFRplus– when the incident light level in Lux is entered.



Now imatest users can quickly and easily analyze ISO sensitivity, as shown in the sample results below for a stepchart image captured with a Canon EOS 20D camera. “ISO Sensitivity was the only major image quality factor not supported by Imatest software, and we are pleased to offer our customers this important new feature” said Norman Koren, Imatest CTO.

Read More