Test Setup – imatest https://www.imatest.com Image Quality Testing Software & Test Charts Wed, 03 Jun 2020 16:38:48 -0600 en-US hourly 1 https://wordpress.org/?v=5.3 Which Imatest Charts Support Automatic Detection? https://www.imatest.com/2019/08/what-imatest-charts-support-automatic-detection/ https://www.imatest.com/2019/08/what-imatest-charts-support-automatic-detection/#respond Wed, 21 Aug 2019 19:55:24 +0000 http://www.imatest.com/?p=27780 Using testing charts (also known as targets) improves the efficiency and repeatability of your testing workflow. Imatest offers several charts that support automatic detection as well as charts with registration marks.

Imatest Charts Supporting Auto-Detection

  • SFRplus – Using this target requires the bars on top and bottom of the chart to be present in the image.
  • 24-patch X-Rite Colorchecker – This chart works best without extreme distortion.
  • Checkerboard – Imatest can detect square areas of a checkerboard target that are not excessively bloomed or blurred.
  • Rezchecker – Imatest can detect the version of this chart with four dots.
  • Dot pattern – This chart is used for testing chromatic aberration and SMIA TV Distortion.

Imatest Charts with Registration Marks

Note: Imatest does not perform autodetection for the Obsolete ISO 12233:2000 chart because it can easily produce invalid results. 

If you have a target you would like to be automatically detected, please contact support@imatest.com.

]]>
https://www.imatest.com/2019/08/what-imatest-charts-support-automatic-detection/feed/ 0
Test Chart Substrate and Sizing Guide https://www.imatest.com/2019/04/test-chart-substrate-sizing-guide/ https://www.imatest.com/2019/04/test-chart-substrate-sizing-guide/#respond Tue, 23 Apr 2019 21:30:45 +0000 http://www.imatest.com/?p=18389 This guide enumerates the possibilities and constraints of our test charts and provides resolution (MTF) information for the different options. Imatest has introduced Chart MTF Compensation, which can as much as double the resolution suitability of a given test chart.

Inkjet

Inkjet Substrate - Close up at 1mm

Close up of inkjet print quality

Inkjet is the lowest resolution process we offer, but also the least expensive and most versatile. The minimum print size with inkjet is approximately 8” x 10”, and the maximum print size for a single sheet is 44” x 80”.

Inkjet charts are available in three different substrates: Matte and Semigloss for reflective testing and DisplayTrans for transmissive tests using a backlight. All inkjet options are roughly the same resolution.

Inkjet is best suited for cameras tested at a moderate distance in order to compensate for the low resolution of the print. It is also suitable for low-resolution or wide-angle systems. As the resolution of the target is relatively low, the size of the chart needs to be larger to accommodate a higher resolution sensor.

We can produce custom charts, and custom sizes of most of our standard charts. Contact charts@imatest.com with your requested size and information about your lens (field of view and testing distance) and sensor (pixel count and aspect ratio.) We need the pixel count to confirm the size you are requesting will be adequate for your imaging system. Our Chart Finder Tool can help you determine the size of your imaging plane at a specific testing distance.

Photographic Paper (Silver Halide)

Photographic Paper Print Quality

Close up of photographic paper print quality

Photographic media offers high-resolution reflective prints when a transmissive target is not an option. These prints are over twice the resolution of inkjet media.

The minimum size of a photographic print is approximately 8” x 10” and the maximum size is approximately 40” x 50”.

Our standard photographic targets include our eSFR ISO chart (3 sizes) and a multi-size option of our SFRplus chart. This multi-size is best suited for testing low-resolution systems at close distances. We can offer custom charts on this media for an additional fee. Contact charts@imatest.com for information. 

Transmissive Photographic Film

Our Color and Black & White film targets are printed using high-precision LVT Technology. These transmissive targets require a lightbox in order to be used properly. 

Black and White

Black and White Film Print Quality

Close up of Black and White Film print quality

Black and White film is over twice the resolution of our inkjet targets, and can optionally include a color patch for color measurements. The size of the film is 12” x 20” or 16” x 20” (with ½” on each side as unprintable area). The minimum target size on this media is roughly 2” x 3”. We can print multiple targets on a single sheet of film depending on the size and resolution requirements to help make this option more cost-effective.

This media is suited for customers testing in a small area with low to medium resolution cameras.

Color

Color Film print quality

Close up of Color Film print quality

Our high-precision color film targets are printed on 8” x 10” film with a maximum printed area of 9.25” x 7.75”. The minimum target size on this media is approximately 0.5” x 1”. As with the Black and White film, we can print multiple targets to a single sheet of film to make this a more cost-effective option.

This media is suited for customers requiring a very compact testing system or those using conversion optics to simulate infinity within a very small space. The fine detail of the film allows us to test mid to high-resolution cameras at close distances and with a small field of view. Color film has the highest dynamic range of all of the targets we offer. 

Chrome on Glass Photomask

Chrome on Glass Print Quality

Close up of Chrome on Glass print quality

Chrome Photomasks are our sharpest option for resolution testing. We use a lithographic photomasking process to print chrome directly onto glass. This process is single tone, so there is no gray-scale or tonal variance.

Chrome on Glass offers superior resolution when color and tonal measurements are not needed. We can make targets as small as 2mm x 3mm for measuring devices such as endoscopes or microscopy equipment. When printing to glass, the chrome has an optical density of 1, which gives a contrast ratio of approximately 10:1, suitable for resolution measurement.

Our standard glass plates are 4” x 4” with a non-printable border of 1 cm. We can source plates up to 20” x 20” and as small as 1” x 1”. Any plates larger than 4” x 4” require an extended lead time of 6-8 weeks. 

We can customize the size of various targets such as our SFRplus or ISO-eSFR targets on request. We can also print most custom design requests to glass, though some may require an engineering fee depending on the complexity of the design. Contact charts@imatest.com.

]]>
https://www.imatest.com/2019/04/test-chart-substrate-sizing-guide/feed/ 0
Compensating MTF Measurements for Chart Quality Limitations https://www.imatest.com/2019/02/compensating-mtf-measurements-for-chart-quality-limitations/ https://www.imatest.com/2019/02/compensating-mtf-measurements-for-chart-quality-limitations/#respond Wed, 27 Feb 2019 15:50:29 +0000 http://www.imatest.com/?p=25134 Camera MTF (sharpness) measurements are subject to a number of variations, some of which, like noise, are random and difficult to control, and some of which are systematic and can be corrected. Variations caused by limitations in chart sharpness are in the latter category. These variations are also affected by the Field of View (FoV) of the image used to test the camera, which is closely related to chart size for charts designed to fill the camera frame. For a given print technology, increasing the FoV, which typically means increasing the spacing between the chart and camera, will increase the measured MTF— making it more accurate. However, there are many practical situations where space is limited and small FoVs are called for, and chart sharpness can significantly affect the measurements. This post describes a method for quantifying and correcting such measurement variations. The method has the following steps.

  1. Measure the test chart MTF and fit it to an equation.
  2. Compensate camera MTF measurements for the test chart sharpness.
  3. Based on measurements with and without compensation for a variety of charts and test magnifications mtest, determine the conditions where (a) chart quality is good enough so no compensation is needed, and (b) chart quality is too low to be reliably compensated.

There is little mention of test chart sharpness in the literature, perhaps because when the ISO 12233:2000 standard was created, high-resolution cameras had only about one megapixel, and hence it was easy to print adequate test charts.

View the video below for Norman Koren’s talk on Compensating MTF Measurements for Chart Quality Limitations. This article describes the concept in detail. 

Measuring test chart MTF

The first step in compensating camera MTF measurements is to measure the test chart MTF. This should be done by photographing the same features used to measure camera MTF. We describe the process for slanted-edges, but other patterns, such as Siemens stars, could also be used. Since we have found that edge sharpness in inkjet charts can depend on the edge orientation, we measure up to four edges (the left, right, top, and bottom of a dark square on a light background).

We performed our chart measurements with a 24 Megapixel APS-C camera that has a 23.5×15.6mm sensor with a 3.88-micron pixel pitch. We used a mechanically-focused 60mm prime macro lens that had a scale that displayed magnification. This enabled us to maintain constant chart magnification mchart for our tests, which is difficult to accomplish with lenses that have electronic focusing. We used mchart = 1:2 (0.5×) for inkjet charts and mchart = 1:1 (1×) for photographic paper and film charts, which are sharper. Chrome-onGlass charts are too sharp to be measured with this setup.

FIGURE 1. CHART MTF MEASUREMENT SETUP

The setup (Fig. 1) consists of a custom-machined aluminum base and a sturdy aluminum extrusion column. Fine focus is controlled by a micrometric positioning sliding plate. The lens is fixed at mchart = 1:2 or 1:1. Simple adjustments are used to keep the sensor plane is parallel to the test chart. A non-flickering LED ring light is used for reflective charts; a light box is used for transmissive charts.

When calculating chart MTF it is important to select large enough Regions of Interest (ROIs) to obtain consistent results. For inkjet charts, which can have rough edges, especially when photographed at chart magnification mchart = 1:1 (1×), MTF50 can vary by as much as ±10% for small ROIs. To obtain good measurement consistency, we photographed inkjet charts at mchart = 1:2 (0.5×), using region sizes ≥ 900×1300 pixels. For chart media other than inkjet, all of which are finer, we used mchart = 1:1.

Fitting measured MTF to an equation

The measured chart MTF must be fit to an equation in order to reliably perform MTF compensation, where the measured camera MTF is divided by the model of the chart MTF projected on the sensor. We have chosen a function that (a) is simple– only two parameters, (b) is a good match to our chart MTF measurements, and (c) is guaranteed to decrease at high spatial frequencies.

Equation (1)

MTFchart(fchart) = exp (-afchart – (bfchart)2)

Where fchart is the spatial frequency in Cycles/Object mm on the chart. a and b are found using an optimizer to match Equation (1) with MTF measurements for f ≤ f30, the first frequency where MTF drops below 0.3 (30%). This is done because noise can dominate MTF measurements for f > f30, as illustrated below (Fig. 3).

Parameters a and b are stored in a file along with metadata (the name of the test chart image file, chart magnification mchart, date, etc.) This file is read into the analysis program when chart compensation is to be applied. We have found that, apart from old charts made with unknown printers and settings, charts don’t need to be measured individually. A compensation file based on media, printer type, and print settings should be sufficient.

Chart MTF measurement examples

Fig. 2 illustrates an MTF measurement for a horizontal slanted-edge on an inkjet chart photographed with mchart = 0.5×. The upper curve shows the average edge profile. The rise distance (14 pixels) is large enough so that camera software sharpening will not affect the results. The lower plot shows the measured MTF (bold black curve) and the fit to the MTF (bold cyan curve) from Equation (1) for f < f30. The parameters for this fit are a = 0.10507 and b = 0.09407.

FIGURE 2. INKJET CHART MTF MEASUREMENT

Curious artifacts sometimes appear in chart MTF measurements. Figure 3 is the MTF of an edge from a chart printed with the edges along the directions of paper and print head motion (not slanted). Test charts printed this way are cut slanted. A strong MTF response peak, visible around 13.5 Cycles/Object mm, appears to be caused by periodicity in the inkjet dots. (It’s not present when edges are printed slanted.) Fortunately, it’s well outside the analysis passband as well as the frequencies used to calculate a and b in Equation (1).

FIGURE 3. INKJET CHART MTF SHOWING RESPONSE PEAK

There are sometimes surprises in the MTF measurements. Chart MTF for photographic film printed on an LVT (Light Valve Technology) printer have a response indicative of sharpening (Fig. 4), apparently caused by uneven depletion of the film developer near edges—familiar to the author from his wet darkroom days. Note that the curve from Equation (1) (cyan) is a good match to the MTF curve, which has a sharpening bump.

FIGURE 4. COLOR LVT FILM MTF SHOWING CHEMICAL SHARPENING

MTF compensation

To calculate MTF compensation, the chart spatial frequency in Cycles/Object mm, fchart, must be transformed into Cycles/Pixel (C/P) on the image sensor. For test magnification mtest,

Equation (2)

f(C/Obj mm) = f(C/P) × mtest × pixels/mm

The MTF of the chart projected on the image sensor is:

Equation (3)

MTFprojected(f) = MTFchart(f(C/P) × mtest × pixels/mm) 

Finally, the chart-compensated MTF is:

Equation (4)

MTFchart−comp(f) = MTFmeasured(f)/MTFdiv(f)

Where:

Equation (5)

MTFdiv(f) = max(MTFprojected(f), 0.3)

Limiting the minimum value of MTFdiv to 0.3 prevents excessive high-frequency noise boost. And as we have shown in Fig. 3, chart MTF measurements at frequencies where MTF drops below 0.3 (f > f30) can be strongly affected by chart noise (especially for inkjet charts), and hence are not reliable.

Camera testing and verification

The effects of chart compensation were tested using a 10-megapixel digital camera (a Panasonic Lumix LX5 from 2010) that had the following:

  • A small 5.4×8.1mm sensor with 2.14-pixel size to ensure low test magnification mtest for most of the charts, intended to keep lens performance relatively consistent throughout the tests.
  • RAW output to minimize nonlinear signal processing commonly found in JPEG files.
  • A high-quality zoom lens set to 50mm (35mm-equivalent) at f/4.

With this camera, we expected MTF measurements to be affected only slightly for the largest charts, which are designed to fill the image frame, and hence have the largest Fields of View (FoVs). We expected MTF measurements to be degraded significantly for the smallest charts.

FIGURE 5. SOME OF THE TEST CHARTS USED TO VERIFY MTF COMPENSATION

Fig. 5 illustrates the two types of slanted-edge chart used in our testing: Imatest SFRplus (a grid of slanted squares with bars at the top and bottom) and eSFR ISO (an enhanced version of the ISO 12233:2014 edge SFR chart). Both charts have 4:1 contrast slantededges, as recommended in ISO 12233:2014. Sizes varied over a range greater than 10:1. These charts were printed over several years on a variety of media— inkjet and photographic paper (reflective), and color photographic film (transmissive).

The reported Fields of View (FoVs) are typically slightly larger than the active area of these charts. Both chart types have geometrical features that facilitate the calculation of test magnification mtest. For each image, four edges — Left (L), Right (R), Top (T), and bottom (B) from the square closest to the chart center — were analyzed for compensated and uncompensated MTF.

Since we didn’t know the history of the charts, MTF for sample edges was measured individually for each chart.

 

Compensated and uncompensated results

MTF measurements in Figures 6-9, each from an edge near the center of four very different test charts, illustrate the effects of chart MTF on results. Uncompensated MTF is shown as a magenta line. Compensated MTF is a bold black line. MTFdiv is a cyan line. MTF50 (the spatial frequency where MTF drops to 50% of its low-frequency level) is the key summary metric for comparing results.

FIGURE 6

The large inkjet chart in Fig. 6 (147×97cm Field of View FoV) has MTF50 = 1166 LW/PH (uncompensated) and 1345 LW/PH (compensated). Correction makes only a small difference in the MTF measurement, as expected. The MTF50 difference is largely caused by a small noise-related response bump.

FIGURE 7

The small inkjet chart in Fig. 7 (32×21cm FoV) has MTF50 = 922 LW/PH (uncompensated) and 1302 LW/PH (compensated). Correction makes a significant difference. Results would be inaccurate without it. f30 is far enough above the Nyquist frequency to ensure good measurement results.

FIGURE 8

The small, low-quality inkjet chart in Fig. 8 (25×17cm FoV) has f30 well below the Nyquist frequency. Results are not reliable. This chart is inadequate for measuring the quality of this camera system.

FIGURE 9

The small but extremely high-quality LVT film chart in Fig. 9 (26×17cm FoV) has MTF50 = 1466 LW/PH (uncompensated) and 1445 LW/PH (compensated). The chart MTF response bump causes a slight decrease in the corrected MTF response.

Results summary

Figures 10 and 11 contain detailed results for five SFRplus and five eSFR ISO charts of various sizes and media. This somewhat arbitrary grouping was chosen because the results would not all fit on a single figure. The four groups of five bars on the left (light magenta background) represent uncompensated MTF50. The four groups of five bars on the right (light yellow background) represent compensated MTF50. Compensated MTF50 is generally larger and much more consistent, as indicated in Table 1, below.

Each group of five bars represents MTF50 results for slanted edges from different charts. Groups are labeled by the edge near the image center used for analysis. The four groups on the left (L, R, T, B) contain uncompensated MTF50. The four groups on the right (L comp, R comp, T comp, B comp) contain the compensated MTF50 of the corresponding edges.

FIGURE 10. SUMMARY RESULTS FOR FIVE SFRPLUS CHARTS UNCOMPENSATED MTF50 ON LEFT; COMPENSATED ON RIGHT.

In Fig. 10, the five bars in each group are for (1) a large inkjet chart (124×82 cm FoV), (2) a medium inkjet chart (49×32 cm FoV), (3) a small inkjet chart (25×17 cm FoV), (4) 8×10-inch color LVT film (26×27 cm FoV), and (5) small color LVT film (14×9 cm FoV).

The medium and small inkjet charts showed the greatest improvement. The small LVT chart may have had consistently lower corrected MTF50 because it had a larger magnification than the other charts, which might have affected lens performance.

FIGURE 11. SUMMARY RESULTS FOR FIVE ESFR ISO CHARTS UNCOMPENSATED MTF50 ON LEFT; COMPENSATED ON RIGHT.

In Fig. 11, the five bars in each group are for(1) a large inkjet chart (147×97 cm FoV),(2) a medium-large photographic paper chart (126×84 cm FoV),(3) a medium-large inkjet chart (129×83 cm FoV), (4) a medium inkjet chart (65×42 cm FoV), and (5) a small photographic paper chart (32×21 cm FoV). The small photographic paper chart showed the greatest improvement.

The key takeaway from Figures 10 and 11, summarized in Table 1, is that compensated results are larger and have a lower standard deviation σ, i.e., they are more consistent and hence more accurate.

TABLE 1. SUMMARY RESULTS FOR FIGURES 10 & 11

Predicting test chart suitability

Equations (1) through (5) can be used to predict the suitability of a test chart for a specific application.

We should note that our guidelines for chart suitability assume that the camera is relatively sharp, i.e., not out of focus or blurred for another reason, such as poor lens quality. A reasonable criterion for a “sharp camera” is that it makes good use of available pixels, which would be the case when the unsharpened MTF50 > 0.1 Cycle/Pixel (C/P). Typical values are around 0.15 – 0.3 C/P for high-quality cameras.

After running numerous images, we developed the following guidelines for chart suitability, based on the projected chart MTF at the Nyquist frequency (0.5 C/P), MTF@ fNyq, on the image sensor.

TABLE 2. GUIDELINES FOR TEST CHART SUITABILITY

Fig. 12 shows results of a test chart suitability calculation for the chart/camera combination of Fig. 7. The compensation file had a = 0.1138 and b = 0.07027. Three of the following four parameters are manually entered.

  • Vertical field = 212 mm
  • µm per pixel = 2.14
  • Sensor height = 5.431 mm
  • Magnification = 0.02562

In typical operation the three geometrical parameters (vertical field, µm per pixel, and sensor height) are entered, and magnification mtest = sensor height / vertical field. From Table 2, we see that the key result, MTF@ fNyq = 0.41, indicates that the chart is being used close to its operating limit (MTF@ fNyq = 0.3), and MTF compensation is definitely required.

Limitations

Analysis is not reliable at spatial frequencies where the test chart MTF projected on the image sensor, MTF@ fNyq, drops below 0.3. This is well beyond the normal recommended limits.

Greater care is required when analyzing chart measurements. The correct compensation file should be specified and the correct test magnification mtest (or geometric parameters for calculating mtest) must be entered.

Chart compensation does not (yet) work well for strongly barrel-distorted (fisheye) images, where radial magnification is a function of distance from the image sensor, and hence radial and tangential magnification may differ.

Noise at high frequencies– especially above fNyq – may be strongly boosted. The response above fNyq should usually be ignored.

Conclusions

MTF compensation improves the consistency and accuracy of camera MTF measurements made from different test charts (often at different locations), especially when charts are used near their megapixel limits. Some MTF variation, primarily due to noise, remains after compensation. In our verification tests we may have observed some MTF variation caused by differences in lens performance at different test magnifications.

MTF compensation files should be created for each printer/media/setting combination. Except for old charts where these details are not known, charts don’t need to be measured individually. We have determined two limits related to MTF compensation:

  1. An upper limit, MTF50 @ Nyquist ≥ 0.9, beyond which chart compensation has little effect,
  2. A lower limit, MTF50 @ Nyquist < 0.3, beyond which compensated results are unreliable,

MTF compensation effectively doubles the megapixel usability limits for most test charts.

We have been using the chart MTF measurement techniques described here to improve the quality control of our printed test charts.

eSFR ISO Slanted Edge Chart - Select a Chart Landing Page

 Shop ISO Standard Test Charts

Standards

ISO 12233:2017 – Electronic still picture imaging – Resolution and spatial frequency response.

Download as PDF

]]>
https://www.imatest.com/2019/02/compensating-mtf-measurements-for-chart-quality-limitations/feed/ 0
Flare: ISO-18844 in the Visible and NIR https://www.imatest.com/2018/11/flare-iso-18844-in-the-visible-and-nir/ https://www.imatest.com/2018/11/flare-iso-18844-in-the-visible-and-nir/#respond Mon, 12 Nov 2018 19:54:33 +0000 http://www.imatest.com/?p=24368 Last month, we posted Considerations when evaluating a Near Infrared camera. We reviewed important considerations for a NIR camera test setup and demonstrated how sharpness can vary between visible and NIR wavelengths. This post continues on that theme by testing Veiling Glare (Flare) using the ISO 18844 method.

What is Veiling Glare or Flare?

Flare occurs to some degree in all lenses and optical systems. Stray light that enters the lens is reflected among the optical components and fogs the image. Below is an example of very bad lens flare.

What is veiling flare or glare in photography?

An example of bad flare (right).

In these images, we can see how Flare is showing light in the image where there should be none. Flare in optical components is a major limiting factor for a systems overall dynamic range. Sensors often can boast dynamic range >120 dB. In practice, the system cannot get close to that range because of the effects of flare. For more information, please read our documentation on Veiling Glare (Flare).

Setting up the NIR camera test

For this test, we will be using the same Raspberry Pi camera and Imatest light source from our previous post. The major differences are the chart and image processing. The ISO 18844 chart features numerous black dots arranged in an X pattern according to the standard. The chart is made out of clear polyester film and the dots are made from Acktar black to prevent reflections that would ruin the test.

The other major difference for this test from our previous one is the change in processing. The image needs to be linear. Nonlinear processes such as auto white balance and jpeg compression can influence the results if enabled. Fortunately, it is possible to get raw sensor data from the Raspberry Pi Camera V2. This recipe was used to capture a raw image from the Pi.

Analyzing the NIR image

Images of the flare chart were captured under two lighting conditions; visible and NIR (850 nm).

Visible wavelength raw image

Visible wavelength raw image

NIR wavelength raw image

NIR wavelength raw image

Note: These images look different than normal images captured on the Pi because this is raw data. There is no auto white balance, gamma, or other processing done. For more information on reading raw images with Imatest see our documentation.

We then analyze each image with the Uniformity module in Imatest Master 5.1.9. In the Uniformity Settings dialog we want to select “ISO 18844 flare method C: Linearize with Color Space”. For this analysis, we can ignore the rest of the options.

Imatest will display several plots related to uniformity but we are only interested in the ISO 18844 flare plot for this test. The plots for each of our measurements can be seen below.

ISO 18844 Flare Visible light plot

ISO 18844 Flare Visible light plot

ISO 18844 Flare NIR light plot

ISO 18844 Flare NIR light plot

The plots show the % flare for each dot according to the 18844 calculation as well as the Mean Flare at the bottom. From the plots, we see that Mean Flare for the visible light sources is 5.2% and 13.6% for the NIR.

Conclusions

Similar to our previous test on sharpness, there is a drastic change in quality when the wavelength of light is varied. The ISO 18844 Flare increases more than 2.5 times in the NIR versus the visible. The expectation then is the NIR camera has a diminished dynamic range due to the increased flare. In applications such as security or automotive cameras, the diminished dynamic range could significantly decrease the effectiveness of the system to detect obstacles or intruders. The large change in quality shows how important it is to construct a test environment that matches the application of the imaging system. This is especially true when the system is multi-purpose, such as a security camera with a day (visible) and night (NIR) modes. A good follow up to this test would be to measure the dynamic range of the system in either lighting condition.

Need more help with Imatest?

Request a SolutionContact Imatest

]]>
https://www.imatest.com/2018/11/flare-iso-18844-in-the-visible-and-nir/feed/ 0
High-contrast edge-SFR test targets produce invalid MTF results https://www.imatest.com/2018/06/high-contrast-edge-sfr-test-targets-produce-invalid-mtf-results/ https://www.imatest.com/2018/06/high-contrast-edge-sfr-test-targets-produce-invalid-mtf-results/#respond Tue, 12 Jun 2018 20:45:38 +0000 http://www.imatest.com/?p=22621 The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio. These are typically produced at the maximum dynamic range of a printer, which can be anywhere from 40:1 to 80:1.  The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values. Some camera manufacturers who want better MTF results may take advantage of this anomaly to overstate the quality of the cameras they produce. This is why it is critical to validate cameras with a proper measurement system that includes a low-contrast target.

4:1 contrast is the international standard

ISO Standard Chart

Enhanced eSFR ISO chart

The international standards committee (ISO) has addressed this issue by reducing the contrast ratio to a lower 4:1 level. See the following excerpt from ISO 12233:2014  (Page 7, section 0.3):

All other imaging conditions being equal, camera SFRs using different target contrast edge features can be significantly different, especially with respect to their morphology. This is largely due to non-linear image processing operators and would not occur for strictly linear imaging systems. To moderate this behaviour, a lower contrast slanted edge feature (Figure C1) was chosen to replace the higher contrast version of the first edition. This feature choice still allows for acuity amenable SFR results beyond the half sampling frequency and helps prevent non-linear data clipping that can occur with high contrast target features. It is also a more reliable rendering of visually important contrast levels in naturally occurring scenes.

The standard defines the edge-SFR chart with modulation contrast shall be between 0.55 and 0.65. You can raise 10 the power of the contrast to determine the contrast ratio or in other words: The contrast ratio shall be between 3.54: 1 and 4.47:1.

Clipping warnings

Edge Profile_Horizontal

Notice the cusp in the average edge function as it clipped because of the saturation of the sensor.

A Clipping warning is issued if more than 0.5% of the pixels are clipped (saturated), i.e., if dark pixels reach level 0 or light pixels reach the maximum level (255 for bit depth = 8). This warning is emphasized if more than 5% of the pixels are clipped. Clipping reduces the accuracy of SFR results. It makes measured sharpness better than reality.

The percentage of clipped pixels is not a reliable index of the severity of clipping or of the measurement error. For example, it is possible to just barely clip a large portion of the image with little loss of accuracy. The plot on the right illustrates relatively severe clipping, indicated by the sharp “shoulder” on the black line (the edge without standardized sharpening). The sharp corner makes the MTF look better than reality. The absence of a sharp corner may indicate that there is little MTF error.

Clipping can usually be avoided with a correct exposure– neither too dark nor light. A low contrast target is recommended for reducing the likelihood of clipping: it increases exposure latitude and reduces the sensitivity of the MTF results to errors in estimating gamma.

Advantages of the ISO 12233:2014/2017 Edge SFR chart over the old ISO 12233:2000 chart

Old ISO 12233:2000 chartThe text below is from Using eSFR ISO Part 1.

The ISO 12233:2000 standard and chart, shown on the right, is referenced in the new ISO 122433:2014 standard, but is no longer an official part of the standard. The new standard specifies three charts, one of which is a slanted-edge (Edge SFR or E-SFR) chart with much lower contrast (4:1; shown on the right, below).

Compared to the old ISO 12233:2000 chart,

  • The E-SFR chart has much less wasted area, especially in the Enhanced and Extended versions (the Enhanced chart is shown on the right, above).

  • About 90% of the old ISO chart is covered with patterns that have little value for computer analysis.

  •  ISO-12233-2014-chart_from_docISO 12233:2014 Edge SFR chart
    from the ISO document

    You can produce a detailed map of sharpness (MTF) over the image surface with the Enhanced and Extended charts. This cannot be done with the old ISO chart because there are not enough suitable edges— and they are not well-located.

  • Automated region detection, based on location criteria you enter. This makes the E-SFR chart and eSFR ISO module well-suited for automated testing: With the old ISO chart, regions of interest (ROIs) had to be selected carefully whenever the image framing changed, even slightly.

 

  • The 4:1 contrast edges are less likely to clip than the edges in the old ISO chart, whose contrast is specified at ≥ 40:1. The camera operates in a more linear region, and hence results are more consistent and accurate— less affected by overexposure, underexposure, incorrect gamma estimate, or excessive software sharpening. 4:1 contrast edges are more representative of real edges that affect perceived image sharpness.

  • eSFR ISO can measure additional image quality factors, including lateral chromatic aberration, distortion, gamma (contrast), tonal response, and color accuracy (in charts that have the color pattern).

4:1 contrast 5X7 SFRplus chart

  • The old ISO 12233:2000 chart contains no information that can be used to measure gamma (the average slope of the Log pixel vs. log exposure curve. i.e., the contrast), which is needed to linearize the image for the MTF calculation. Accurate gamma measurement is particularly important with high contrast targets. In the E-SFR chart, gamma can be derived either from the individual squares (which have a known 4:1 contrast) or from the grayscale step chart.

Compared to the SFRplus chart,

  • The map of MTF over the image surface is less detailed, though it’s sufficient for most purposes.
  • Distortion measurements are slightly less detailed (especially for the Standard (minimal) chart) because there are no distortion bars and there are fewer features for measuring distortion.
  • eSFR ISO contains a highly detailed noise calculation, including all noise measurements from Multicharts and Multitest.
  • Framing is more flexible since you don’t need white space above and below top and bottom bars. As long as the registration marks are reasonably well inside the image, region detection should work.

Low contrast targets

These test targets have contrast ratios that comply with the ISO standard:

Inkjet ISO 12233:2014
Photographic ISO 12233:2014
Film ISO 12233:2014
SFRplus inkjet 4:1 contrast ratio
SFRreg Test Chart
Checkerboard Test Chart

See also

Wikipedia: Clipping (Signal processing)
Slanted-edge SFR Saturation
Implementing Pass/Fail in Imatest

]]>
https://www.imatest.com/2018/06/high-contrast-edge-sfr-test-targets-produce-invalid-mtf-results/feed/ 0
Megapixel suitability for test charts https://www.imatest.com/2018/03/megapixel-suitability/ https://www.imatest.com/2018/03/megapixel-suitability/#respond Fri, 02 Mar 2018 00:14:38 +0000 http://www.imatest.com/?p=21420 Megapixel suitability is based on the analysis of the Modulation Transfer Function (MTF) that is obtained from slanted-edge chart images captured at magnifications of around 0.5× (for inkjet charts) or 1× (for photographic charts) using a high quality DSLR or mirrorless camera and macro lens.

The calculations are described in the following links:

Note: the old Chart Quality Index (CQI) calculation has been deprecated.

Megapixel suitability is based on the spatial frequencies where the projected chart MTF on sensor is 0.9 at the Nyquist frequency (0.5 cycles/pixels). We’ll omit the details of the calculations here. The beauty of the new approach is that megapixel suitability can be determined from just three items:

  1. File (the chart MTF compensation file) that has a model of the chart MTF (measured in cycles per object mm).
  2. Pixel height of the image sensor.
  3. Vertical Field of View of the camera, typically somewhat larger than the height of charts designed to fill the frame (SFRplus, eSFR ISO, etc.).

We use the Chart suitability display for Black and White LVT film (which is very much sharper than inkjet prints) as an example.

The standard Black & White LVT chart, printed on 12 inch × 20 inch film, is designed to have a 27cm vertical field of view; 27cm (on the x-axis) corresponds to MTF@Nyquist = 0.9 (the green diagonal line, below) for sensor height = 5500 pixels (left y-axis) or 45 megapixels at 3:2 aspect ratio (right y-axis). At this high quality level, MTF compensation is not required. If we push the chart to MTF@Nyquist = 0.7 (the olive diagonal line), which is still reasonably good, but requires MTF compensation, we reach sensor height = 8500 pixels (about 100 Megapixels at 3:2 aspect ratio).

Chart suitability display for B&W LVT film. Click image for full-size view.

The Megapixel suitability calculation shown here assumes that:

  1. The lens is of high quality.
  2. The sensor aspect ratio is 3:2, which can be changed in the box on the right. The change only affects the Megapixel numbers; i.e., Megapixel suitability, on the right y-axis.

For 16:9 aspect ratio sensors, multiply the megapixel suitability by 1.185.
For 4:3 aspect ratio sensors (with left/right sides of chart cropped), multiply megapixel suitability by 0.889.

Additions to Imatest 5.1

Imatest 5.1, released in April 2018, has an important enhancement that increases the megapixel suitability of most Imatest charts by up to a factor of 2. The MTF for most charts, which is a function of the chart media and printing technique, has been measured and fit to a simple two-parameter function that can be used to correct MTF measurements by deconvolution (by dividing the measured camera+chart MTF by the chart MTF function projected on the image sensor). The correction can be applied by entering an MTF correction file into the settings windows for any MTF calculations. For more details, see Compensating camera MTF measurements for chart and sensor MTF.

Chart Quality Calculator that uses the new MTF functions is also available. It provides a clearer and more accurate estimate of MTF suitability (including the expected MTF loss from the media with and without the correction) than the older Chart Quality Index.

Chart suitability displays for several media types are found in Test chart suitability for MTF measurements.

See Also

]]>
https://www.imatest.com/2018/03/megapixel-suitability/feed/ 0
Lightbox Uniformity Comparison https://www.imatest.com/2018/01/lightbox-uniformity-comparison/ https://www.imatest.com/2018/01/lightbox-uniformity-comparison/#respond Fri, 26 Jan 2018 17:24:39 +0000 http://www.imatest.com/?p=21185  

Lightbox Brightness Uniformity* CRI (spec.)
Viewing Area Dimensions Controls
Imatest LED Lightbox 1 to 100,000 lux ** 90 to 95% Over 97

260 x 220 mm to 1440 x 1100 mm (9 Sizes)

400 (W) x 380 (H) x 200mm (D) to 1655 x 1296 x 200 mm (9 Sizes) WiFi, USB, Manual
Imatest LED Light Panel 30 to 1,000 lux 90 to 95%   229 x 152mm to 907 x 680mm (5 Sizes)
289 x 212 x 40mm to 967 x 740 x 55mm (5 Sizes)
WiFi, USB, Manual
IQL LED Lightbox 10 to 40,000 lux     254 x 279.4mm 472.4 x 383.5 x 129.5mm Wireless via Android
GL-16e Lightbox Viewer 5750 lux 63.6% 96-98 10 x 18″ (25 x 46 cm) 15x25x5″ (38x64x13cm) Manual
GL-20e Lightbox Viewer            
GL-30e Lightbox Viewer            
GL-44e Lightbox Viewer            
GLX-3044 Lightbox Viewer N/A N/A 96-98 30×42″ (76x107cm) 35x49x5″ (89x124x13cm) Manual
GLX-3856 LIghtbox Viewer N/A N/A 96-98 38×56″ (96x142cm) 43x63x5″ (109x160x13cm) Manual
GLE-10 Lightbox Viewer N/A TBC 96-98 8×10″ (20x25cm) 15.5×12.25×3.25″ (39x31x8cm) Manual
GLE GLX-30 Lightbox Viewer 5000 lux 70.1% 96-98 16″ x 30″ (41 x 76cm) 21 x 37.5 x 5″  Manual
Artograph LightPad® 930 2820 lux  77%   12×9″   On/Off
Artograph LightPad® 950 2740 lux 78.5%   24×17″   On/Off

*measured using 9 rectangular regions, as described below.
** measured using 30-10,000 lux model, ultra bright version has 90% uniformity

 

A better color quality measurement?  The color quality of light sources is traditionally measured by CRI (Color Rendering Index), which has a maximum value of 100 (%). Recently, doubt has been cast on the accuracy of CRI, and a new Color Rendition measurement has been proposed: IES TM-30-15. It’s unfamiliar and the linked document doesn’t have an equation or algorithm for calculating it from the light spectrum. We’ll wait and see…

Lightbox Uniformity- Detailed Measurements

For the key measurement, the definition of uniformity is

Uniformity = 100%*(1 – (maximum of 9 regions- minimum of 9 regions) / maximum of 9 regions)

where the 9 rectangular regions (shown in the figures below) include the top, bottom, left, right, 4 corners, and center. The rectangular region dimensions are 10% of the crop width and height and (except for the center region) they are located 5% of the width and height from the boundaries, as described below.

 

Click on any of the images below to view full-sized.

ITI LED Lightbox

Uniformity = 95.2%

Response is very even, but unusual in that the center is slightly dimmer than the top and bottom.

Note that the contour line increments are 0.01 (1%), lower than the other lightboxes because the ITI is much more uniform. (0.02 contour increments wouldn’t show very much.)

The lightbox spectra for the standard 3100K and 5100K settings, provided by ITI, are shown below.

ITI_uniformity_contours
ITILED-3100k ITILED-5100k

GTI GL-16e Lightbox

Uniformity = 63.6%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox. Both the GL-16e and GLX-30 have very wide aspect ratios. Their uniformity would be much better if less of the sides were included in the measurement.

 

GTI_GLX16e_uniformity_contours

   

Artograph 930 12×9 inch Light Pad

We use this inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 77%

Artograph_12x9_uniformity_contours

Artograph 950 24×17 inch Light Pad

We use this large, relatively inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 78.5%

(Figure is darker because image was less exposed.) 

Arto_950_uniformity_contours

The control

The control for these measurements was made by capturing images immediately in front of the ITI lightbox (no more than 1cm distant). Results were repeatable when the camera was moved around the lightbox. Contour increments are very small: only 0.002 (0.5%).

Uniformity = 98.75%

EOS-6D_closeup_uniformity

GTI GLX-30 Lightbox

(No longer available in the Imatest Store)

Uniformity = 70.1%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX30_uniformity_contours

GTI GLE 12e Lightbox

(No longer available in the Imatest Store)

Uniformity = 65.8%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX12e_uniformity_contours

How we made the measurements

We developed a methodology for measuring lightbox uniformity because we were not aware of any relevant standards.

  • Photograph the lightbox using a camera with a long focal length marco lens. Such lenses tend to be highly uniform, i.e., have very low vignetting. We used the Canon EOS-6D with the highly-regarded 100mm f/2.8 macro lens set at f/8. Be sure to capture raw images. We used manual focus because the EOS-6D didn’t focus well on this image.
  • Frame the lightbox so it occupies about the central 30% of the image (10% by area). This makes the already low vignetting insignificant. Here is the framing (and region selection) for two lightboxes.
Click on the images to display them larger.
ITI_uniformity_cropITI Lightbox GTI_uniformity_cropGTI Lightbox
  • Open Uniformity (or Uniformity Interactive) and read in the raw file, converting it to a gamma = 1 (linear) file. (This means it’s not a standard file, but pixel level will be proportional to illumination.) Here are the recommended settings from the Imatest dcraw GUI window. The key settings are Output gamma = 1.0 (Linear), Auto white level checked, and Normalize by 1.0.

lightbox_dcraw_settings

  • Crop the images just inside the bright areas of the lightbox, as shown above. If there are areas of rapid illumination falloff close to the edges of the lightbox image, it’s OK to omit them.
  • Click Yes to open the Uniformity settings box. The key settings are shown inside the red rectangles. The corner and side regions (the rectangles) are 10% of the ROI (linearly), and the location of the regions is 5% (of the ROI size) from the ROI boundaries. We feel this is a reasonable summary metric since most tone and color measurements are made in the central two-thirds of the image. Uniformity is more important when measuring tone and color than it is for MTF, even though Imatest corrects for patch nonuniformity due to vignetting and uneven illumination.

lightbox_uniformity_settings

  • After you click OK (not shown) Uniformity runs and the results figures appear. The key nonuniformity summary metric does not appear in the figures it’s in the CSV and JSON file output. Here is the relevant CSV output.

    Uniformity = 100%*(1 – (maximum of 9 squares – minimum of 9 squares) / maximum of 9 squares)

Nonuniformity LRTB sides ctr (%) 36.44
Uniformity LRTB sides ctr (%) 63.56
  • and here is the JSON output:

         “nonuniformity_LRTB_sides_ctr_pct”: [36.4],
         “uniformity_LRTB_sides_ctr_pct”: [63.56],

 

]]>
https://www.imatest.com/2018/01/lightbox-uniformity-comparison/feed/ 0
Avoiding reflections on transmissive charts for dynamic range and flare testing https://www.imatest.com/2017/12/avoiding-reflections-transmissive-charts-dynamic-range-flare-testing/ https://www.imatest.com/2017/12/avoiding-reflections-transmissive-charts-dynamic-range-flare-testing/#respond Mon, 18 Dec 2017 18:13:25 +0000 http://www.imatest.com/?p=20708

The testing of dynamic range and flare requires a proper test chart and environmental setup to get accurate results.High precision and high-density chart technologies such as photographic film or chrome on glass come with the disadvantage of glossiness which makes them susceptible to specular reflections on the target. This affects the use of dynamic range or contrast resolution and backlit flare targets that lack anti-reflective patches. 

The goal of the backlit test setup is to maximize the fraction of light that is coming directly from your light source, through your test target, and into your lens, Any additional light that is present in your system can disrupt your measurements either by causing reflections on the analysis patch of your target, or otherwise introducing additional flare (veiling glare) into the image.

As the optimal lens and camera body housing will be manufactured with anti-reflective coating, it is good to take a similar approach with your test environment.  Nothing is actually entirely black, but dark, opaque materials come in many different forms. These can be characterized by their total hemispherical reflectance (THR).

Here are how some dark materials compare:

Product THR (vis+NIR) Notes
Black Paint 5-6% Not dark enough
Black Felt 2.5% Flexible and low-cost
Acktar black 1.5% Comes on adhesive foil
Vantablack 0.17% Costly, not durable, dangerous to humans

For general testing purposes, we use black felt for covering our darkened test box

Block all light from entering or reflecting inside the testing environment

For accurate testing, you should create a dark space where no outside light is able to enter, and as much internally generated light is absorbed. Here is a diagram of a test environment:

Minimize internal direct reflections

The more that the surfaces in your test environment are lit up, the more they will become reflections in the glossy chart.

The body of the camera under test or even just glass of a camera lens can reflect back onto a test chart.This is especially important incident towards dark analysis regions.

If using Imatest charts that are properly centered, the reflection might not fall on to a dark analysis region for dynamic range measurement. Here is an example of a reflection of a dev kit PCB that would disrupt dynamic range or contrast resolution measurement:

Reflective camera parts should be blocked by black masks. The entire area behind your camera will directly be reflected back, so this environment should be made as dark as possible.

Minimize other internal light emissions

Sensor development kits often have blinking LED’s that reflect directly, Front facing cameras with displays that are not disabled or blocked off can be particularly problematic.

]]>
https://www.imatest.com/2017/12/avoiding-reflections-transmissive-charts-dynamic-range-flare-testing/feed/ 0
DMX Lighting Control Software https://www.imatest.com/2017/12/dmx-lighting-control-software/ https://www.imatest.com/2017/12/dmx-lighting-control-software/#respond Fri, 15 Dec 2017 22:38:23 +0000 http://www.imatest.com/?p=20879 Lights such as the Kino Flo Select 31 LED use a DMX control interface to signal changes of light level and color temperature.

DMX is an open protocol and there are a large number of possibilities for hardware and software control. I’ve tried out several of these are more oriented to stage control
 
For DMX controller hardware I have personally tried and will recommend the Enttec DMX USB Pro controller worked well for us.
For computer control software with a user Interface, the one that has worked best for me so far is the Q Light Controller Plus (QLC+) which has nice cross-platform support.
 
For light control software that provides an automated API, the best one I have found, and the one we use to automatically control lights in our lab is the Open Lighting Architecture (OLA). The downside of this is that it only has good support for Linux. I have attached a python script that includes an example OLA calls, along with calls to an isolight puck, which you can overlook.  Here is a basic call to set the lights to middle color temp and maximum intensity:
 
from ola.ClientWrapper import ClientWrapper
intensity = 255
cct       = 127
universe = 1
data      = array.array('B') 
data.append(intensity)
data.append(cct)
data.append(intensity)
data.append(cct)
wrapper = ClientWrapper()
client = wrapper.Client()
client.SendDmx(universe, data, DmxSent)
 
Our goal for Imatest 5.3 is to have an instrument control interface built into Imatest Master that can directly communicate with DMX hardware.

 

]]>
https://www.imatest.com/2017/12/dmx-lighting-control-software/feed/ 0
Using the DarkWorld chart mask https://www.imatest.com/2017/08/using-darkworld-chart-mask/ https://www.imatest.com/2017/08/using-darkworld-chart-mask/#respond Tue, 08 Aug 2017 23:20:31 +0000 http://www.imatest.com/?p=19681 This post is meant to help with the correct setup and operation of the new DarkWorld Chart Mask

The DarkWorld mask pairs with the Imatest 36 patch Dynamic Range Test Chart and frame to block additional light from entering the camera. This allows for more accurate measurements by reducing the flare light coming from the target. Tests with transmissive targets should be performed in a completely dark environment with no ambient lighting apart from the light box.

DarkWorld Chart Mask Setup

If you have previously ordered a Dynamic Range Chart, setting up the new DarkWorld chart mask is as easy as 123! Each DarkWorld chart mask comes with six strips of Velcro. If you order the frame and mask together, you will not have to worry about applying these strips yourself.

Step 1: Verify that you have a new DarkWorld chart mask and a 36 patch dynamic range chart with a frame, as well as six strips of Velcro: 2 soft, non adhesive-backed strips and 4 smaller, rough adhesive-backed strips.

Step 2: Place the chart mask in the frame, and apply the 4 rough velcro strips by peeling off their adhesive backing. The Velcro is best placed in the vertical center, on both the left and right side. They should be placed end to end with a small gap to prevent interference when applying and removing the mask. 

Step 3: Firmly apply the soft, non adhesive-backed strip to the velcro on the chart mask. That’s it! Now you can ahdere the hanging tabs to the frame or peel them off to remove the Chart mask. 

The assembly is now ready for testing with an ITI Lightbox! Simply slide the frame into the rails on the front of the lightbox, and follow the intructions for using Multicharts to measure the dynamic range of your camera system with a single image. 

]]>
https://www.imatest.com/2017/08/using-darkworld-chart-mask/feed/ 0