P2020 Automotive Engineering Technology May 2019

Imatest attended the P2020 meeting on May 13 and 14, 2019 in Ann Arbor, Michigan. Paul Romancyzk, PhD., Senior Imaging Scientist, and Rob Sumner, Lead Engineer, represented Imatest. Paul co-led the discussion on Color Separation within the Image Quality for Machine Vision subgroup.

ieee standards association, ieee standards association board, electronic engineering technology, automotive engineering technology

P2020 May 2019

(more…)

Read More

Imatest at IEEE P2020

Imatest sent two engineers to the IEEE P2020 Automotive Imaging Quality face-to-face meeting in Dusseldorf, Germany this past February in 2019. The IEEE P2020 standard, which is still in development, aims to define KPIs and test procedures to address the many challenges relevant (and often unique) to automotive imaging.

The group hopes to publish the standard in the year 2020 (no relation to the standard’s identifying number) as a unifying guideline for developers of autonomous driving systems. Most driver-assist and autonomous-driving systems rely heavily on cameras despite the significant press other imaging modalities, such as LIDAR, have received. As Level 3, 4, and 5 autonomous vehicles start to come on the road in 2020 and beyond, it is vital that these systems have an agreed-upon set of rigorous performance metrics. (more…)

Read More

Compensating MTF Measurements for Chart Quality Limitations

Camera MTF (sharpness) measurements are subject to a number of variations, some of which, like noise, are random and difficult to control, and some of which are systematic and can be corrected. Variations caused by limitations in chart sharpness are in the latter category. These variations are also affected by the Field of View (FoV) of the image used to test the camera, which is closely related to chart size for charts designed to fill the camera frame. For a given print technology, increasing the FoV, which typically means increasing the spacing between the chart and camera, will increase the measured MTF— making it more accurate. However, there are many practical situations where space is limited and small FoVs are called for, and chart sharpness can significantly affect the measurements. This post describes a method for quantifying and correcting such measurement variations. The method has the following steps.

  1. Measure the test chart MTF and fit it to an equation.
  2. Compensate camera MTF measurements for the test chart sharpness.
  3. Based on measurements with and without compensation for a variety of charts and test magnifications mtest, determine the conditions where (a) chart quality is good enough so no compensation is needed, and (b) chart quality is too low to be reliably compensated.

There is little mention of test chart sharpness in the literature, perhaps because when the ISO 12233:2000 standard was created, high-resolution cameras had only about one megapixel, and hence it was easy to print adequate test charts.

View the video below for Norman Koren’s talk on Compensating MTF Measurements for Chart Quality Limitations. This article describes the concept in detail. 

Measuring test chart MTF

The first step in compensating camera MTF measurements is to measure the test chart MTF. This should be done by photographing the same features used to measure camera MTF. We describe the process for slanted-edges, but other patterns, such as Siemens stars, could also be used. Since we have found that edge sharpness in inkjet charts can depend on the edge orientation, we measure up to four edges (the left, right, top, and bottom of a dark square on a light background).

We performed our chart measurements with a 24 Megapixel APS-C camera that has a 23.5×15.6mm sensor with a 3.88-micron pixel pitch. We used a mechanically-focused 60mm prime macro lens that had a scale that displayed magnification. This enabled us to maintain constant chart magnification mchart for our tests, which is difficult to accomplish with lenses that have electronic focusing. We used mchart = 1:2 (0.5×) for inkjet charts and mchart = 1:1 (1×) for photographic paper and film charts, which are sharper. Chrome-onGlass charts are too sharp to be measured with this setup.

FIGURE 1. CHART MTF MEASUREMENT SETUP

The setup (Fig. 1) consists of a custom-machined aluminum base and a sturdy aluminum extrusion column. Fine focus is controlled by a micrometric positioning sliding plate. The lens is fixed at mchart = 1:2 or 1:1. Simple adjustments are used to keep the sensor plane is parallel to the test chart. A non-flickering LED ring light is used for reflective charts; a light box is used for transmissive charts.

When calculating chart MTF it is important to select large enough Regions of Interest (ROIs) to obtain consistent results. For inkjet charts, which can have rough edges, especially when photographed at chart magnification mchart = 1:1 (1×), MTF50 can vary by as much as ±10% for small ROIs. To obtain good measurement consistency, we photographed inkjet charts at mchart = 1:2 (0.5×), using region sizes ≥ 900×1300 pixels. For chart media other than inkjet, all of which are finer, we used mchart = 1:1.

Fitting measured MTF to an equation

The measured chart MTF must be fit to an equation in order to reliably perform MTF compensation, where the measured camera MTF is divided by the model of the chart MTF projected on the sensor. We have chosen a function that (a) is simple– only two parameters, (b) is a good match to our chart MTF measurements, and (c) is guaranteed to decrease at high spatial frequencies.

Equation (1)

MTFchart(fchart) = exp (-afchart – (bfchart)2)

Where fchart is the spatial frequency in Cycles/Object mm on the chart. a and b are found using an optimizer to match Equation (1) with MTF measurements for f ≤ f30, the first frequency where MTF drops below 0.3 (30%). This is done because noise can dominate MTF measurements for f > f30, as illustrated below (Fig. 3).

Parameters a and b are stored in a file along with metadata (the name of the test chart image file, chart magnification mchart, date, etc.) This file is read into the analysis program when chart compensation is to be applied. We have found that, apart from old charts made with unknown printers and settings, charts don’t need to be measured individually. A compensation file based on media, printer type, and print settings should be sufficient.

Chart MTF measurement examples

Fig. 2 illustrates an MTF measurement for a horizontal slanted-edge on an inkjet chart photographed with mchart = 0.5×. The upper curve shows the average edge profile. The rise distance (14 pixels) is large enough so that camera software sharpening will not affect the results. The lower plot shows the measured MTF (bold black curve) and the fit to the MTF (bold cyan curve) from Equation (1) for f < f30. The parameters for this fit are a = 0.10507 and b = 0.09407.

FIGURE 2. INKJET CHART MTF MEASUREMENT

Curious artifacts sometimes appear in chart MTF measurements. Figure 3 is the MTF of an edge from a chart printed with the edges along the directions of paper and print head motion (not slanted). Test charts printed this way are cut slanted. A strong MTF response peak, visible around 13.5 Cycles/Object mm, appears to be caused by periodicity in the inkjet dots. (It’s not present when edges are printed slanted.) Fortunately, it’s well outside the analysis passband as well as the frequencies used to calculate a and b in Equation (1).

FIGURE 3. INKJET CHART MTF SHOWING RESPONSE PEAK

There are sometimes surprises in the MTF measurements. Chart MTF for photographic film printed on an LVT (Light Valve Technology) printer have a response indicative of sharpening (Fig. 4), apparently caused by uneven depletion of the film developer near edges—familiar to the author from his wet darkroom days. Note that the curve from Equation (1) (cyan) is a good match to the MTF curve, which has a sharpening bump.

FIGURE 4. COLOR LVT FILM MTF SHOWING CHEMICAL SHARPENING

MTF compensation

To calculate MTF compensation, the chart spatial frequency in Cycles/Object mm, fchart, must be transformed into Cycles/Pixel (C/P) on the image sensor. For test magnification mtest,

Equation (2)

f(C/Obj mm) = f(C/P) × mtest × pixels/mm

The MTF of the chart projected on the image sensor is:

Equation (3)

MTFprojected(f) = MTFchart(f(C/P) × mtest × pixels/mm) 

Finally, the chart-compensated MTF is:

Equation (4)

MTFchart−comp(f) = MTFmeasured(f)/MTFdiv(f)

Where:

Equation (5)

MTFdiv(f) = max(MTFprojected(f), 0.3)

Limiting the minimum value of MTFdiv to 0.3 prevents excessive high-frequency noise boost. And as we have shown in Fig. 3, chart MTF measurements at frequencies where MTF drops below 0.3 (f > f30) can be strongly affected by chart noise (especially for inkjet charts), and hence are not reliable.

Camera testing and verification

The effects of chart compensation were tested using a 10-megapixel digital camera (a Panasonic Lumix LX5 from 2010) that had the following:

  • A small 5.4×8.1mm sensor with 2.14-pixel size to ensure low test magnification mtest for most of the charts, intended to keep lens performance relatively consistent throughout the tests.
  • RAW output to minimize nonlinear signal processing commonly found in JPEG files.
  • A high-quality zoom lens set to 50mm (35mm-equivalent) at f/4.

With this camera, we expected MTF measurements to be affected only slightly for the largest charts, which are designed to fill the image frame, and hence have the largest Fields of View (FoVs). We expected MTF measurements to be degraded significantly for the smallest charts.

FIGURE 5. SOME OF THE TEST CHARTS USED TO VERIFY MTF COMPENSATION

Fig. 5 illustrates the two types of slanted-edge chart used in our testing: Imatest SFRplus (a grid of slanted squares with bars at the top and bottom) and eSFR ISO (an enhanced version of the ISO 12233:2014 edge SFR chart). Both charts have 4:1 contrast slantededges, as recommended in ISO 12233:2014. Sizes varied over a range greater than 10:1. These charts were printed over several years on a variety of media— inkjet and photographic paper (reflective), and color photographic film (transmissive).

The reported Fields of View (FoVs) are typically slightly larger than the active area of these charts. Both chart types have geometrical features that facilitate the calculation of test magnification mtest. For each image, four edges — Left (L), Right (R), Top (T), and bottom (B) from the square closest to the chart center — were analyzed for compensated and uncompensated MTF.

Since we didn’t know the history of the charts, MTF for sample edges was measured individually for each chart.

 

Compensated and uncompensated results

MTF measurements in Figures 6-9, each from an edge near the center of four very different test charts, illustrate the effects of chart MTF on results. Uncompensated MTF is shown as a magenta line. Compensated MTF is a bold black line. MTFdiv is a cyan line. MTF50 (the spatial frequency where MTF drops to 50% of its low-frequency level) is the key summary metric for comparing results.

FIGURE 6

The large inkjet chart in Fig. 6 (147×97cm Field of View FoV) has MTF50 = 1166 LW/PH (uncompensated) and 1345 LW/PH (compensated). Correction makes only a small difference in the MTF measurement, as expected. The MTF50 difference is largely caused by a small noise-related response bump.

FIGURE 7

The small inkjet chart in Fig. 7 (32×21cm FoV) has MTF50 = 922 LW/PH (uncompensated) and 1302 LW/PH (compensated). Correction makes a significant difference. Results would be inaccurate without it. f30 is far enough above the Nyquist frequency to ensure good measurement results.

FIGURE 8

The small, low-quality inkjet chart in Fig. 8 (25×17cm FoV) has f30 well below the Nyquist frequency. Results are not reliable. This chart is inadequate for measuring the quality of this camera system.

FIGURE 9

The small but extremely high-quality LVT film chart in Fig. 9 (26×17cm FoV) has MTF50 = 1466 LW/PH (uncompensated) and 1445 LW/PH (compensated). The chart MTF response bump causes a slight decrease in the corrected MTF response.

Results summary

Figures 10 and 11 contain detailed results for five SFRplus and five eSFR ISO charts of various sizes and media. This somewhat arbitrary grouping was chosen because the results would not all fit on a single figure. The four groups of five bars on the left (light magenta background) represent uncompensated MTF50. The four groups of five bars on the right (light yellow background) represent compensated MTF50. Compensated MTF50 is generally larger and much more consistent, as indicated in Table 1, below.

Each group of five bars represents MTF50 results for slanted edges from different charts. Groups are labeled by the edge near the image center used for analysis. The four groups on the left (L, R, T, B) contain uncompensated MTF50. The four groups on the right (L comp, R comp, T comp, B comp) contain the compensated MTF50 of the corresponding edges.

FIGURE 10. SUMMARY RESULTS FOR FIVE SFRPLUS CHARTS UNCOMPENSATED MTF50 ON LEFT; COMPENSATED ON RIGHT.

In Fig. 10, the five bars in each group are for (1) a large inkjet chart (124×82 cm FoV), (2) a medium inkjet chart (49×32 cm FoV), (3) a small inkjet chart (25×17 cm FoV), (4) 8×10-inch color LVT film (26×27 cm FoV), and (5) small color LVT film (14×9 cm FoV).

The medium and small inkjet charts showed the greatest improvement. The small LVT chart may have had consistently lower corrected MTF50 because it had a larger magnification than the other charts, which might have affected lens performance.

FIGURE 11. SUMMARY RESULTS FOR FIVE ESFR ISO CHARTS UNCOMPENSATED MTF50 ON LEFT; COMPENSATED ON RIGHT.

In Fig. 11, the five bars in each group are for(1) a large inkjet chart (147×97 cm FoV),(2) a medium-large photographic paper chart (126×84 cm FoV),(3) a medium-large inkjet chart (129×83 cm FoV), (4) a medium inkjet chart (65×42 cm FoV), and (5) a small photographic paper chart (32×21 cm FoV). The small photographic paper chart showed the greatest improvement.

The key takeaway from Figures 10 and 11, summarized in Table 1, is that compensated results are larger and have a lower standard deviation σ, i.e., they are more consistent and hence more accurate.

TABLE 1. SUMMARY RESULTS FOR FIGURES 10 & 11

Predicting test chart suitability

Equations (1) through (5) can be used to predict the suitability of a test chart for a specific application.

We should note that our guidelines for chart suitability assume that the camera is relatively sharp, i.e., not out of focus or blurred for another reason, such as poor lens quality. A reasonable criterion for a “sharp camera” is that it makes good use of available pixels, which would be the case when the unsharpened MTF50 > 0.1 Cycle/Pixel (C/P). Typical values are around 0.15 – 0.3 C/P for high-quality cameras.

After running numerous images, we developed the following guidelines for chart suitability, based on the projected chart MTF at the Nyquist frequency (0.5 C/P), MTF@ fNyq, on the image sensor.

TABLE 2. GUIDELINES FOR TEST CHART SUITABILITY

Fig. 12 shows results of a test chart suitability calculation for the chart/camera combination of Fig. 7. The compensation file had a = 0.1138 and b = 0.07027. Three of the following four parameters are manually entered.

  • Vertical field = 212 mm
  • µm per pixel = 2.14
  • Sensor height = 5.431 mm
  • Magnification = 0.02562

In typical operation the three geometrical parameters (vertical field, µm per pixel, and sensor height) are entered, and magnification mtest = sensor height / vertical field. From Table 2, we see that the key result, MTF@ fNyq = 0.41, indicates that the chart is being used close to its operating limit (MTF@ fNyq = 0.3), and MTF compensation is definitely required.

Limitations

Analysis is not reliable at spatial frequencies where the test chart MTF projected on the image sensor, MTF@ fNyq, drops below 0.3. This is well beyond the normal recommended limits.

Greater care is required when analyzing chart measurements. The correct compensation file should be specified and the correct test magnification mtest (or geometric parameters for calculating mtest) must be entered.

Chart compensation does not (yet) work well for strongly barrel-distorted (fisheye) images, where radial magnification is a function of distance from the image sensor, and hence radial and tangential magnification may differ.

Noise at high frequencies– especially above fNyq – may be strongly boosted. The response above fNyq should usually be ignored.

Conclusions

MTF compensation improves the consistency and accuracy of camera MTF measurements made from different test charts (often at different locations), especially when charts are used near their megapixel limits. Some MTF variation, primarily due to noise, remains after compensation. In our verification tests we may have observed some MTF variation caused by differences in lens performance at different test magnifications.

MTF compensation files should be created for each printer/media/setting combination. Except for old charts where these details are not known, charts don’t need to be measured individually. We have determined two limits related to MTF compensation:

  1. An upper limit, MTF50 @ Nyquist ≥ 0.9, beyond which chart compensation has little effect,
  2. A lower limit, MTF50 @ Nyquist < 0.3, beyond which compensated results are unreliable,

MTF compensation effectively doubles the megapixel usability limits for most test charts.

We have been using the chart MTF measurement techniques described here to improve the quality control of our printed test charts.

eSFR ISO Slanted Edge Chart - Select a Chart Landing Page

 Shop ISO Standard Test Charts

Standards

ISO 12233:2017 – Electronic still picture imaging – Resolution and spatial frequency response.

Download as PDF

Read More

Imatest Branded Lightboxes and Light Panels

We are pleased to announce that beginning January 2019, we are offering Imatest branded light panels and boxes. Imatest has responded to feedback from our customers by creating these two new uniform light sources. Our new LED Lightbox and Light Panel offer a wide variety of light intensities and color temperatures including near-infrared channels. We designed them to integrate easily with test fixtures, allow for precise chart alignment with chart rails, and to serve a variety of testing scenarios.

“We wanted to offer our own Imatest branded hardware that has been vetted and tested by our engineers which enable reliable results when used with our software and test charts. Our goal is to provide solutions for image quality testing that produces the results our customers need to exceed their expectations. The Imatest Light Panels and Lightboxes are made with the same high standards for quality that are found in our charts and software,” says Christian Taylor, Imatest’s Hardware Engineering Manager.

(more…)

Read More

Partnership with Shenzhen Polytechnic University

Imatest partnered with Shenzhen Polytechnic University to support the digital image quality testing curriculum at the university. With the expansion of their image quality program, Imatest sponsored and authorized the development of a state of the art research and education lab.

 

The lab and ceremony

On October 31st, 2018, Henry Koren, Director of Engineering, and Ian Longton, Imaging Science Engineer joined Shenzhen Polytechnic Executive Vice Presidents of Communication Engineering, Dean He Yuhua and Yang Yunyan, Vice President Zhang Xuilang, and Colorspace General Manager Nick Liu and colleagues for the official unveiling ceremony.

(more…)

Read More

Electronic Imaging Symposium 2019

Join Imatest at the Electronic Imaging Symposium from Jan. 13 – 17 in Burlingame, California USA. Explore the entire imaging science ecosystem, from capture through image processing to how we and our surrogate machines see and interpret images. Henry Koren and Norman Koren will be presenting on the following.

Reducing the cross-lab variation of image quality metrics

Henry Koren
 
As imaging test labs seek to obtain objective performance scores of camera systems, many factors can skew the results. IEEE Camera Phone Image Quality (CPIQ) Conformity Assessment Steering Committee (CASC) working group members performed round-robin studies where an assortment of mobile devices was tested within heterogeneous imaging labs. This paper investigates how the existence of near-infrared energy in light sources that attempt to simulate CIE illuminants can influence test results. Numerous other impacts, including the influence of opal diffusers used for uniformity testing, how test scene framing can alter white balance and exposure, and how chart quality and texture frequency distribution can skew results. We introduce a test procedure which is intended to reduce intra-lab variability and a method for assessing an independent lab’s competence in conforming with the IEEE testing standards.
 
Date/Time: Tuesday January 15, 2019, 3:50 – 4:10
Location: Grand Peninsula Ballroom E
 
 

Compensating MTF measurements for Chart Quality limitations

Norman Koren
 
Objective measurements of imaging system sharpness are typically derived from test chart images. It is generally assumed that if testing instructions are followed—if the chart print quality is fine enough and the chart magnification (on the sensor) is low enough—  test chart quality will have little impact on the overall measurement.
 
This assumption may not be valid when extremely high-resolution cameras are tested with standard charts or when smaller than optimum test charts are used because of laboratory or production line space limitations.
 
MTF compensation is applied by dividing the measured system MTF by chart-projected MTF as a function of the sensor.
  1. Measurements made under different conditions— with different test charts (transmissive as well as reflective) by different people in different labs— are more consistent.
  2. The megapixel suitability of test charts is increased by a factor of approximately 2 (1.4x linearly).
  3. The key advantage of MTF compensation is that measurements are more accurate (in an absolute sense).
 
Date/Time: Tuesday January 15, 2019, 9:10am
Location: Grand Peninsula Ballroom E
Read More

December 2018 Reading List

Our team has outlined a list of our favorite imaging resources that we use on the job here at Imatest. Take a look at our top picks for the new year.

Articles

Determining MTF With A Slanted Edge Target 

This article by Douglas Kerr is a very digestible explanation (with diagrams) of the principles of slanted edge MTF testing.

Blogs

Image Sensors World

Author Vladimir Koifman shares the latest news and discussions on image sensors and image sciencing technology. 

Books

Digital Image Processing

This book has been the foundational text for the study of digital image processing for over 40 years. It is suited for those with prior background in mathematical analysis, vectors, matrices, probability, statistics, linear systems, and computer programming.

 

Camera Image Quality Benchmarking 

Part of The Wiley-IS&T Series in Imaging Science and Technology, this book contains basic information and approaches for the use of subjectively correlated image quality metrics and outlines a framework for camera benchmarking.  The authors show how to quantitatively compare the image quality of cameras used for consumer photography. 

 

Handbook of Camera Monitor Systems

This handbook offers a comprehensive overview of Camera Monitor Systems (CMS), ranging from the ISO 16505-based development aspects to practical realization concepts. 

 

 

The Manual of Photography, Tenth Edition

This book is good for someone who has some foundation in imaging but would like to round out their knowledge. It is comprehensive of many key concepts, but not too detailed that is hard to comprehend.

 

Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light

Principles of Optics is one of the classic science books of the twentieth century, and probably the most influential book in optics published in the past forty years.

 

Videos

Marc Levoy’s Lectures on Digital Photography

These 18 videos represent a sequence of lectures on digital photography from a Stanford photography course. For those who don’t have access to YouTube, you may view the lectures here.

Websites

ClarkVision.com

Astrophotography site rich in information about how digital imaging sensors work, pixel size and photon collection, lenses, noise, etc. – and how to use this information as a photographer. 

Photography Fundamentals & Tutorial Sites

Luminous-Landscape.com

CambridgeinColour.com

Read More

Flare: ISO-18844 in the Visible and NIR

Last month, we posted Considerations when evaluating a Near Infrared camera. We reviewed important considerations for a NIR camera test setup and demonstrated how sharpness can vary between visible and NIR wavelengths. This post continues on that theme by testing Veiling Glare (Flare) using the ISO 18844 method.

What is Veiling Glare or Flare?

Flare occurs to some degree in all lenses and optical systems. Stray light that enters the lens is reflected among the optical components and fogs the image. Below is an example of very bad lens flare.

What is veiling flare or glare in photography?

An example of bad flare (right).

In these images, we can see how Flare is showing light in the image where there should be none. Flare in optical components is a major limiting factor for a systems overall dynamic range. Sensors often can boast dynamic range >120 dB. In practice, the system cannot get close to that range because of the effects of flare. For more information, please read our documentation on Veiling Glare (Flare).

Setting up the NIR camera test

For this test, we will be using the same Raspberry Pi camera and Imatest light source from our previous post. The major differences are the chart and image processing. The ISO 18844 chart features numerous black dots arranged in an X pattern according to the standard. The chart is made out of clear polyester film and the dots are made from Acktar black to prevent reflections that would ruin the test.

The other major difference for this test from our previous one is the change in processing. The image needs to be linear. Nonlinear processes such as auto white balance and jpeg compression can influence the results if enabled. Fortunately, it is possible to get raw sensor data from the Raspberry Pi Camera V2. This recipe was used to capture a raw image from the Pi.

Analyzing the NIR image

Images of the flare chart were captured under two lighting conditions; visible and NIR (850 nm).

Visible wavelength raw image

Visible wavelength raw image

NIR wavelength raw image

NIR wavelength raw image

Note: These images look different than normal images captured on the Pi because this is raw data. There is no auto white balance, gamma, or other processing done. For more information on reading raw images with Imatest see our documentation.

We then analyze each image with the Uniformity module in Imatest Master 5.1.9. In the Uniformity Settings dialog we want to select “ISO 18844 flare method C: Linearize with Color Space”. For this analysis, we can ignore the rest of the options.

Imatest will display several plots related to uniformity but we are only interested in the ISO 18844 flare plot for this test. The plots for each of our measurements can be seen below.

ISO 18844 Flare Visible light plot

ISO 18844 Flare Visible light plot

ISO 18844 Flare NIR light plot

ISO 18844 Flare NIR light plot

The plots show the % flare for each dot according to the 18844 calculation as well as the Mean Flare at the bottom. From the plots, we see that Mean Flare for the visible light sources is 5.2% and 13.6% for the NIR.

Conclusions

Similar to our previous test on sharpness, there is a drastic change in quality when the wavelength of light is varied. The ISO 18844 Flare increases more than 2.5 times in the NIR versus the visible. The expectation then is the NIR camera has a diminished dynamic range due to the increased flare. In applications such as security or automotive cameras, the diminished dynamic range could significantly decrease the effectiveness of the system to detect obstacles or intruders. The large change in quality shows how important it is to construct a test environment that matches the application of the imaging system. This is especially true when the system is multi-purpose, such as a security camera with a day (visible) and night (NIR) modes. A good follow up to this test would be to measure the dynamic range of the system in either lighting condition.

Need more help with Imatest?

Request a SolutionContact Imatest

Read More

5 Common Pitfalls in Image Quality Testing

Having trouble with your Imatest test results? This article explains the five most common pitfalls in image quality testing, and how to resolve them.

You’re using a chart with overly high contrast, like the obsolete ISO 12233 test chart.

Your chart may have too high contrast edges. The most common example we see of this is the obsolete ISO 12233 test chart, although this can be the case with any test chart. If you are using the ISO 12233 2000 test chart, you should not be. 

The obsolete ISO 12233 2000 test chart.

ISO 12233 test chart

The ISO 12233 2000 standard test chart has a very high contrast ratio (the white is very white and the black is very black) that can cause clipping in signal and lead to invalid MTF values. This is because the contrast of the chart often saturates the sensor, reporting higher MTF results than there might actually be. Therefore, we recommend using a chart with a 4:1 contrast ratio as revised in the ISO 12233 2014 standard, such as the ISO 12233 2014 chart or SFRplus chart. The lower contrast ratio produces more accurate measurements by providing MTF measurements that do not clip.

If you need help selecting the right contrast in a chart, email us at charts@imatest.com.

You’re using the wrong dynamic range chart.

When you’re testing dynamic range, you need to make sure you’re using a dynamic range chart which has a higher dynamic range than your camera to ensure you’re measuring the camera and not the chart. The common pitfall here is that people tend to use tonal step charts which have lower dynamic ranges than their sensors, such as the old Kodak Q14 or X-rite colorchecker. For example, if you are using a typical matte reflective chart which has a dynamic range of 48dB, and you’re testing a camera which has a dynamic range of 80dB, then the highest result you could get is 48dB. This is inaccurate.

However, if you’re using a chart with 120dB, you’ll get a much more accurate reading because it will measure the camera and not the chart. Before you test, you should check to see that your camera is not exceeding the dynamic range of what your chart is rated for. Once you have verified this, you will know which dynamic range chart to select. Our 36-Patch Dynamic Range chart is the most popular chart we sell with densities exceeding, 50db, 100db, and 150db.

Verifying the quality of your dynamic range chart.

The usage of linear dynamic range charts such as the Stouffer T4410 or Xyla target causes many problems including difficulty selecting dark regions that do not have visible contrast (especially using distorted lenses) as well as the linear density slope which interferes with the radial non-uniformity of the imaging system. The ISO standard dictates that each patch should be the same distance from the center and charts that have radial layouts of patches are much less impacted by light falloff. Also, targets which are very dark are not representative of more luminous scenes and overlook the impact of flare on dynamic range which leads to higher dynamic range performance, which will not hold up when the camera is exposed to a scene which is not mostly blackened. Imatest’s 36-patch targets include an option for a “DarkWorld” mask which can produce both dark and gray scenes using a single target.

Determining dynamic range chart quality.

Statistics from an image of XYLA-21: dark regions
are difficult to select.

If you need help determining which chart is best suited for you, please email us at charts@imatest.com.

Your chart quality is poor.

Similar to the dynamic range chart pitfall, when testing the sharpness of your camera system, you need to ensure the resolution of your chart is high enough for the device you’re testing. If your reflective or transmissive chart’s MTF is too low the accuracy of your camera resolution measurements will be limited by the resolution of the chart.The quality of your chart is poor.

The culprit is print quality. The print process can affect the results of your test, especially at close distances or with high-megapixel cameras. For example, using a chart printed on a basic laser printer to test a high-resolution mobile phone camera, could reveal inaccurate results, shown by the steps in the MTF graph.

Example comparing poor chart quality and good chart quality for image quality testing.

Example of poor chart quality (left) and good chart quality (right).

Avoid this pitfall by using a high-quality chart on higher resolution substrates such as inkjet or film, or by applying MTF Compensation to compensate for the chart quality. To ensure the highest quality, we encourage you to get charts printed from Imatest. If you don’t know what quality chart you need, contact charts@imatest.com.

You’re not filling the camera field of view.

When testing, it’s important to ensure you’re completely filling the camera field of view with the test chart. If the chart you are using does not include measurement regions near the corners of your field of view, your analysis will not be comprehensive and you may overlook problems with MTF or signal loss at the extents of the image plane. When measuring distortion, the full field of view can be extrapolated from points available within the center, but this is less accurate than having real points to measure. The following video helps illustrate how the field of view affects your setup.

Watch Here

This is particularly challenging for cameras with wide or ultra-wide fields of view where planar targets can’t fill the FoV. We offer specialized test equipment to measure resolution and uniformity of ultra-wide devices.

You’re not testing at the right focus distance.

Test distance is calculated by focal distance; in other words, the target needs to be set at the distance(s) in which it is in focus. The common pitfall we see is not testing at the right focal distance. A comprehensive test regimen will involve testing between the focal distance and the hyperfocal distance. You can’t always move closer to a target in order to fill the field of view, as this can fall below the minimum focal distance of your lens – or below the distance which a chart of a particular does not negatively impact measurements.

Not filling the field of view and testing within the camera's focus distance.

This chart is not at a distance which fills the lens field of view while in focus.

The chart needs to be within a distance to fill the lens field of view in focus. You may need to test at more than one focus distance if you have a lens with variable or automatic focus so that you can validate optimal focus across the range of focus of your device.

To find the minimum focus distance, check your lens specifications or contact the lens manufacturer. To find the maximum focus distance, or hyperfocal distance, see this hyperfocal distance calculator.

Still need help with Imatest?

Contact Imatest

Read More

Shanghai Imatest Training Course and Seminar – May 2019

Imatest in Shanghai

Imatest representatives will visit Shanghai, China, May 14-16, 2019 to offer a free information seminar and a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.

 

Imatest Training Course - Shanghai - May 2019

 

Two-Day Training Course

The training course on May 15 & 16 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments. 

Training starts at 9:00 and will end at 17:00-18:00 depending on questions.

Location:
Holiday Inn Shanghai Pudong
No. 899 Dongfang Road,Pudong New Area, Shanghai
+86 21 5830 6666
Ming Hall


View the detailed training schedule here.

Sign up online or contact a reseller in your area.

Register Online

 

Free Information Seminar

If you are interested in finding out more about how Imatest software can improve your image quality testing, we encourage you to come to our free information seminar before the two-day Training Course:

Time: May 14 from 13:30 – 18:00

Location:
Holiday Inn Shanghai Pudong
No. 899 Dongfang Road,Pudong New Area, Shanghai
+86 21 5830 6666
Han Hall

Sponsored by ColorSpace.

Please register here if you are interested in attending. 

 

We look forward to meeting you in person.

 

 

Read More

Seoul, Korea Imatest Training Course and Seminars – April 2019

 

Imatest in Seoul, Korea

Imatest engineers will visit Seoul, April 17-19, 2019, to offer a free information seminar and a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.

 

 

Two-Day Training Course

image (2)

The training course on April 18 & 19 offers attendees insight on the full capabilities of Imatest software in both research and development and manufacturing environments. Training starts at 09:00 and will end between 17:00 – 18:00, depending on questions.

 

Location Details:

Best Western Gangnam Hotel
139 Bongeunsa-ro, Nonhyeon 1(il)-dong, Gangnam-gu, Seoul, South Korea
Tel: 82-2-6474-2000 / Fax: 82-2-6474-2002
https://www.bestwesterngangnam.com

 

Instructor:

Rob Sumner
Imaging Science Engineer at Imatest

 

View the detailed Training schedule.

Sign up online or contact a reseller in your area.

 

Register Online

 

Free Information Seminar

If you are interested in finding out more about how Imatest software can improve your image quality testing, we encourage you to come to our free information seminar.

Time:

April 17th, 2019
1:30 – 4:30 pm

Location:

Best Western Gangnam Hotel
139 Bongeunsa-ro, Nonhyeon 1(il)-dong, Gangnam-gu, Seoul, South Korea
Tel: 82-2-6474-2000 / Fax: 82-2-6474-2002
https://www.bestwesterngangnam.com

 

Please register if you are interested in attending.

We look forward to meeting you in person.

 

Seoul Skyline photo credit: Jimmy McIntyre

Read More

Boulder, Colorado Imatest Training Course – May 2019

Imatest Training Course in Boulder, Colorado

Imatest representatives will host a two-day training course at our headquarters in Boulder, Colorado May 8 & 9, 2019 for professionals using or considering Imatest software to improve their image quality testing processes.

 Boulder, CO - Imatest Training Course - May 2019

 

Two-Day Training Course

image (2)

The training course on May 8 & 9 2019 offers attendees insight on the full capabilities of Imatest software in both research development and manufacturing environments. Training starts at 8:30 am and will end at 5:30 pm, depending on questions. Please contact us about nearby lodging suggestions.

 

Where:

Imatest Headquarters

4775 Walnut Street Suite 200 Boulder, CO 80301

 

View the detailed Training schedule.

 

Sign up online to reserve your seat.

 

Register Online

 

We look forward to meeting you in person.

 

 

Read More

San Jose, CA Training Course

Imatest in San Jose

 

This event has ended. Please see our schedule for future training.

 

Imatest representatives will visit San Jose August 1-3, 2018 to offer a free information seminar and a paid 2-day Training Course to professionals using or considering Imatest software to improve their image quality testing processes.

2-day Training Course

The Training Course on August 2 & 3, 2018 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments.

Training will run from 9:00 am to 5:30 pm each day, depending on questions. The event will be located in the Larkspur Landing Hotel in Campbell.

Visit our training page to view a detailed training schedule.

 

Group Discounts

Enroll 2-4 attendees to receive 20% off per attendee.

Enroll 5 or more attendees to receive 30% off per attendee.

 

 

Skyline photo by Timothy L. Preen

Read More

Electronic Imaging 2018

We are pleased to announce that we will return to the Bay Bridge, San Francisco at dusk. Shot from Yerba Buena Island.IS&T Electronic Imaging conference in 2018, to take place from January 28 – February 1 in Burlingame, CA. The conference will be held at the Hyatt Regency San Francisco Airport and will feature 20 technical conferences covering a variety of electronic imaging topics, ranging “from image sensing to display and hardcopy to machine vision in use in transportation systems” (Electronic Imaging).

We will have a booth in the exhibition hall and will be demonstrating our software and test charts. Please contact us to meet with our image quality experts. 

Additionally, Norman Koren will be presenting his paper, “Measuring the impact of flare lighting on dynamic range” on Monday, January 29, 2018 at 3:30 pm. 

The dynamic range of recent HDR image sensors, defined as the range of exposure between saturation and 0 dB SNR, can be extremely high: 120 dB or more. But the dynamic range of real imaging systems is limited by veiling glare (flare light), arising from reflections inside the lens, and hence rarely approaches this level. Veiling glare measurements, such as ISO 18844, made with black cavities on white fields, result in large numbers that are difficult to relate to dynamic range. Camera dynamic range is typically measured from grayscale charts, where veiling glare depends on the design and layout of the chart, leading to inconsistent results. We discuss the interaction between veiling glare and dynamic range measurements and the need for standardization of test conditions. We also propose a new dynamic range definition and a new test chart design for directly measuring the visibility of low contrast features over a wide range of scene brightness, which is important for real-world applications, especially in the automotive and security industries where the visibility of people and obstacles in shadow regions is critical. Unlike standard grayscale charts, the new test chart provides meaningful results when tone mapping is applied.

 

 

Read More

Cologne, Germany Training Course

Imatest in Cologne, Germany

Imatest representatives will visit Cologne, Germany September 24-25, 2018 to offer a paid 2-day Training Course to professionals using or considering Imatest software to improve their image quality testing processes.

 

2-day Training Course

image (2)

The training course offers attendees insight on the full capabilities of Imatest software in both research development and manufacturing environments. Training starts at 09:00 and will end at 17:00 – 18:00, depending on questions.

 

When: September 24-25th, 2018 

Location: TBA. 

 

View the detailed Training schedule.

 

Sign up online or contact a reseller in your area.

 

This class has limited availability. Please contact us at training@imatest.com if you are interested in attending. Or you may view our current training schedule here

Read More

Using Arbitrary Charts for Development of the P1858 Variation Combo Chart

The release of Imatest 5.0 introduced a number of powerful new features, including the Arbitrary Charts module which enables Imatest analysis of test chart designs which would be otherwise unsupported by the software. This new module allows user-defined chart layouts for any situation which requires one.

The primary concept of Arbitrary Charts is that the user supplies a chart-definition text file which declares the location and properties of features on a test chart. The features can be placed in essentially any configuration and Imatest will still be able to automatically analyze the chart.

Here at Imatest, we believe we’ve designed some very comprehensive test charts, such as our flagship SFRplus and eSFR-ISO charts. These charts are designed to provide a number of image quality factor measurements, work with an extremely wide range of cameras, are available in pre-distorted versions for wide-angle lenses, and more. They each have their own dedicated modules in Imatest which recognize these chart designs.

While we highly recommend the use of these and our other standard charts, we also recognize that there are situations which may require a more custom target. For example:

  • Working with a legacy database of test images of an older chart

  • Meeting a test spec which requires specific, non-standard targets

  • custom reticles for collimator systems

  • Special combination charts for accelerated testing

 

P1858 Combination Chart Conception

As mentioned above, some situations require a custom target, including the need to combine charts for accelerated testing. Early in 2017, a customer and frequent collaborator approached us about adding support for a new chart. The IEEE p1858-2016 Cell Phone Image Quality standard requires images of at least five different standard test charts for the complete set of seven metrics. The customer wanted to significantly reduce the need for several charts by developing a test chart that provides a “good enough” measurement of all of the metrics from a single image. We worked together with them, jointly developing what would become the P1858 Variation Combo Chart and the Arbitrary Charts Module. 

The immediate goal in this case was to reduce testing time of a device by at least a factor of five. (This is likely a low bound on the time savings, as switching between test targets is typically the most time-intensive part of a well-tuned automatic test bench.) The customer wanted results that, even if they weren’t technically in spec, would provide an indication of how a test device would score on a full-fledged CPIQ test, and wanted them to be produced quickly.

This would, of course, require a set of compromises on the CPIQ metrics:

  • SFR measurements from just outside of the center of field, instead of dead center

  • 24 color patches based on the X-Rite ColorChecker instead of the 140 of the ColorChecker SG

  • Color Uniformity results measured from the chart background areas between the other features, rather than an entire flat-field image

  • Chromatic Aberration and Geometric Distortion measurements from slanted edges around the field, instead of the dot pattern

These changes from the CPIQ spec were deemed acceptable for this purpose, and the P1858 Combination Chart was born. This design is now publically available, with the approval of the original customer.

 

Using the Arbitrary Charts Module

We have been very happy to collaborate with Imatest on their new 5.0 release to allow us to design a combination chart that reduces our objective metric capture needs, improving our productivity and eliminating the challenge of correlating data from multiple capture conditions.

– Lead Imaging Scientist, Imatest Customer

At first glance, the P1858 Combination Chart looks very similar to Imatest’s standard eSFR-ISO chart.

Imatest eSFR-ISO Chart

P1858 Variant Combination Chart

The most obvious difference here is that there is the dead leaves texture pattern in the center of the chart instead of a slanted square. However, there are many other subtle differences which would make this completely unusable by our standard eSFR-ISO module:

  • The grayscale OECF patches have been enlarged and moved outward

  • Extra color patches have been added to approximate all of the patches on the X-Rite ColorChecker

  • Four individual slanted edges have been added on the inside of the OECF patches, so SFR measurements are still possible near center of field

  • In general, the positioning of wedges and slanted squares is different

These differences preclude any standard module in Imatest from performing an automatic analysis of the chart (though selections of slanted edges and wedges could be made manually in individual images using the SFR and Wedge modules, respectively – not a great solution).

Fortunately, this chart was designed in concert with the development of the Arbitrary Charts module, with the express purpose of making analysis of this sort of chart automatic and systematic. Moreover, not only can the combined results be produced from a single image but they are produced from a single module run in Imatest. Instead of running a module to analyze the color, a module for sharpness, a module for uniformity, etc., the Arbitrary Charts module simplifies the testing procedure and output parsing.

The ability to configure our own charts is revolutionizing our approach to quantitative image quality testing and test development.

-Lead Imaging Scientist, Imatest Customer

As an added bonus, the customer is able to make slight changes to this design at will and simply update the chart definition file supplied to the module, without having to wait for new builds of Imatest to catch up.

 

Learning to Use Arbitrary Charts

The capability to analyze user-defined charts is new in Imatest 5.0, and is under continual development. New capabilities are being added to the Arbitrary Charts module with each release.

Further information about the Arbitrary Charts module and its current capabilities can be found on its documentation page.

The most important novel aspect of working with this module is the introduction of the chart definition file which enables the software to understand images of the test chart. To help communicate the role and construction of this file, we are producing a series of short videos that describes the process.

 

Read More

Using Arbitrary Charts Video Series

This series of short videos provides an introduction to the Arbitrary Charts module. Learn how to develop Chart Definition Files

Oct. 25, 2017
Watch on YouTube or download
 

Arbitrary Chart Definition Part 1: In this video, we introduce chart definition JSON files, their structure, and purpose in the Arbitrary Charts module.

 
Read More

Five reasons to use Imatest 5.0

This week, we launched Imatest 5.0. We built Imatest 5.0 to provide users with a streamlined workflow for more efficient image quality analysis. Here are five reasons to start using Imatest 5.0 today.

Refined User Experience

Imatest 5.0 features several new tools and enhancements across our products to help you improve your workflow. Whether you want to reduce the number of images needed for complete system qualification with Arbitrary Charts; process a collection of images with automated analysis routines within the brand new Test Manager, or process batches of images with our enhanced image processing module, Imatest 5.0 is an improved experience for users needing an expedited testing process.

Innovative Image Quality Measurements

The newest release provides several new and enhanced measurements to help your focus on the image quality factors that matter to your business’ camera system.

  • Contrast Resolution was designed for the visualization and measurement of contrast separation. This is particularly important for the automotive industry where camera systems need to distinguish between low contrast objects in a larger field with varying ranges of brightness.
  • The ISO 18844 flare measurement added to the Uniformity module was created to address the need to accurately measure veiling glare.
  • The enhancements to the SFRreg Center Chart provides geometric utilities for measuring tilt and rotational measurements, an especially important factor for users needing to test wide-angled measurements.

Measuring Contrast Resolution using Multicharts.

Focus on the image quality factors that matter most

Imatest tools serve many industries with widely varying requirements, for example testing an automotive camera system is different from testing a consumer device. Even within one industry, companies need to test several image quality factors on several test charts, complicating the capture analysis workflow. Imatest 5.0 lets you zero in on the image quality factors and results that matter most to your business.

  • Customizable reports allows users to more efficiently filter and present select image quality data from a variety of image quality factors.
  • The new arbitrary charts functionality serves a similar purpose by letting users define efficient layouts of analysis features within one chart.

Enhanced support for industry standards

Industry standards provide a foundation for the testing and analysis of imaging systems and clearly define testing methods for reliable measurements of system performance. Imatest 5.0 goes along way to support and supplement industry testing standards.

Run predefined test plans for standard testing procedures such as CPIQ within the new Test Manager.

The new P1858 Variant Combo chart, supported by Arbitrary Charts, combines several analysis features  required by the standard into one chart. This enables users to minimize the number of images needed in the testing process and increases testing efficiency and accuracy.

Imatest 5.0 also features an enhanced Uniformity module to support the ISO 18844 standard for image flare measurements on digital cameras. The standard was created to address some of the difficulties of making ISO 9358 measurements with cameras.

P1858 Variant Combo chart

Streamlined Product Line

The Imatest 5.0 release includes the streamlining of the Imatest product line. As of Imatest 5.0, Imatest Master will now includes image acquisition capabilities that were previously available in Imatest IS, which is now discontinued. Imatest users will now have access to the acquisition library. The library supports direct acquisition from a wide range of frame grabbers and cameras, as well as industry standard interfaces like CameraLink, GigE Vision, and USB through DirectShow (Windows) or QuickTime (OS X). Direct image acquisition cuts out several steps in the image quality testing process and allows for in-the-loop testing with Imatest.

In order to maximize performance in the new Test Manager, we utilized parallel processing. Previously, parallel processing was only available in Imatest IT-P. With the release of Imatest 5.0, Imatest IT-P functionality is now included in Imatest IT . This enables all Imatest users to rapidly process large volumes of images. By utilizing high-performance processors with 6+ cores, parallel processing can improve throughput by over 2.5x.

Learn MoreStart Free Trial

 

Read More

Imatest IS Discontinued with Image Acquisition now Available in Imatest Master

As of Imatest 5.0, Imatest Master now features image acquisition capabilities. Previously, image acquisition capabilities were supported by Imatest IS, which has been discontinued as a separate product. This provides all of our customers with access to the acquisition library. The library supports direct acquisition from a wide range of frame grabbers and cameras, as well as industry standard interfaces. Direct image acquisition cuts out several steps in the image quality testing process and allows for in-the-loop testing with Imatest.

(more…)

Read More

Imatest featured as one of three companies changing the autonomous driving landscape

Imatest was recently featured in Automoblog.net’s article, Three Companies Changing the Autonomous Driving Landscape. Carl Anthony writes:

“With driverless cars, the implication is huge because cameras will play a vital role in the forthcoming autonomous world. In order for autonomy to deliver on its promises of reducing collisions and traffic fatalities, image quality is essential. Imatest takes this into consideration as today’s automotive trends usher us further into autonomy.”

Read the full article at www.automoblog.net/2017/06/30/three-companies-changing-the-autonomous-driving-landscape/. 

Read More