Join Imatest at the Electronic Imaging Symposium from Jan. 13 – 17 in Burlingame, California USA. Explore the entire imaging science ecosystem, from capture through image processing to how we and our surrogate machines see and interpret images. Henry Koren and Norman Koren will be presenting on the following.
Reducing the cross-lab variation of image quality metrics
Henry Koren
As imaging test labs seek to obtain objective performance scores of camera systems, many factors can skew the results. IEEE Camera Phone Image Quality (CPIQ) Conformity Assessment Steering Committee (CASC) working group members performed round-robin studies where an assortment of mobile devices was tested within heterogeneous imaging labs. This paper investigates how the existence of near-infrared energy in light sources that attempt to simulate CIE illuminants can influence test results. Numerous other impacts, including the influence of opal diffusers used for uniformity testing, how test scene framing can alter white balance and exposure, and how chart quality and texture frequency distribution can skew results. We introduce a test procedure which is intended to reduce intra-lab variability and a method for assessing an independent lab’s competence in conforming with the IEEE testing standards.
Date/Time: Tuesday January 15, 2019, 3:50 – 4:10
Location: Grand Peninsula Ballroom E
Compensating MTF measurements for Chart Quality limitations
Norman Koren
Objective measurements of imaging system sharpness are typically derived from test chart images. It is generally assumed that if testing instructions are followed—if the chart print quality is fine enough and the chart magnification (on the sensor) is low enough— test chart quality will have little impact on the overall measurement.
This assumption may not be valid when extremely high-resolution cameras are tested with standard charts or when smaller than optimum test charts are used because of laboratory or production line space limitations.
MTF compensation is applied by dividing the measured system MTF by chart-projected MTF as a function of the sensor.
Measurements made under different conditions— with different test charts (transmissive as well as reflective) by different people in different labs— are more consistent.
The megapixel suitability of test charts is increased by a factor of approximately 2 (1.4x linearly).
The key advantage of MTF compensation is that measurements are more accurate (in an absolute sense).
We are migrating to a better, faster licensing platform for our software on January 1, 2019. You may need to update your software to activate it on a new device, depending on your software version. Updating your software is free of charge.
Who is affected by this?
If your license code starts with 2848, you will need to update your software after January 1, 2019 if you need to install it on other devices or reactivate it on a current device. Your software will continue to work as normal until you need to reinstall or reactivate.
What actions do you need to take?
You will need to update your software in order install it on other devices or reactivate it on your current device. Check below for your software build download link. If you’re not sure which version you need, contact us.
What happens if you don’t update?
You can continue to use your software as it is without updating it. However, if you attempt to reactivate it or install your software on another device after January 1, 2019, you won’t be able to do so. You’ll need to update the software to continue using it.
What are the benefits of this?
Our licenses will now run properly on OSX versions High Sierra and higher.
Our licenses will have fewer issues activating on machines with heavy security and those that run in offline environments.
What version do you have?
As a reminder, this applies to licensing codes starting in 2848 only.
On opening Imatest, the version you are running will be displayed at the top of the main screen, in the command window, or in the Help > About window.
Download your software update today.
Version 4.1+ :
Visit our download page for the latest builds of these versions which support the new licensing system.
Versions 4.0 and below:
Select your software version from the appropriate drop-down menus below to access your download link.
This book has been the foundational text for the study of digital image processing for over 40 years. It is suited for those with prior background in mathematical analysis, vectors, matrices, probability, statistics, linear systems, and computer programming.
Part of The Wiley-IS&T Series in Imaging Science and Technology, this book contains basic information and approaches for the use of subjectively correlated image quality metrics and outlines a framework for camera benchmarking. The authors show how to quantitatively compare the image quality of cameras used for consumer photography.
This handbook offers a comprehensive overview of Camera Monitor Systems (CMS), ranging from the ISO 16505-based development aspects to practical realization concepts.
This book is good for someone who has some foundation in imaging but would like to round out their knowledge. It is comprehensive of many key concepts, but not too detailed that is hard to comprehend.
Principles of Optics is one of the classic science books of the twentieth century, and probably the most influential book in optics published in the past forty years.
These 18 videos represent a sequence of lectures on digital photography from a Stanford photography course. For those who don’t have access to YouTube, you may view the lectures here.
Astrophotography site rich in information about how digital imaging sensors work, pixel size and photon collection, lenses, noise, etc. – and how to use this information as a photographer.
Witnessing the Aurora Borealis should be at the top of every photographer’s bucket list. This guide details the most important considerations for capturing images of the Northern lights. To be successful, you’ll need to be in the right place at the right time, equipped with the right camera and a decent amount of luck on your side.
Last month, we posted Considerations when evaluating a Near Infrared camera. We reviewed important considerations for a NIR camera test setup and demonstrated how sharpness can vary between visible and NIR wavelengths. This post continues on that theme by testing Veiling Glare (Flare) using the ISO 18844 method.
What is Veiling Glare or Flare?
Flare occurs to some degree in all lenses and optical systems. Stray light that enters the lens is reflected among the optical components and fogs the image. Below is an example of very bad lens flare.
An example of bad flare (right).
In these images, we can see how Flare is showing light in the image where there should be none. Flare in optical components is a major limiting factor for a systems overall dynamic range. Sensors often can boast dynamic range >120 dB. In practice, the system cannot get close to that range because of the effects of flare. For more information, please read our documentation on Veiling Glare (Flare).
Setting up the NIR camera test
For this test, we will be using the same Raspberry Pi camera and Imatest light source from our previous post. The major differences are the chart and image processing. The ISO 18844 chart features numerous black dots arranged in an X pattern according to the standard. The chart is made out of clear polyester film and the dots are made from Acktar black to prevent reflections that would ruin the test.
The other major difference for this test from our previous one is the change in processing. The image needs to be linear. Nonlinear processes such as auto white balance and jpeg compression can influence the results if enabled. Fortunately, it is possible to get raw sensor data from the Raspberry Pi Camera V2. This recipe was used to capture a raw image from the Pi.
Analyzing the NIR image
Images of the flare chart were captured under two lighting conditions; visible and NIR (850 nm).
Visible wavelength raw image
NIR wavelength raw image
Note: These images look different than normal images captured on the Pi because this is raw data. There is no auto white balance, gamma, or other processing done. For more information on reading raw images with Imatest see our documentation.
We then analyze each image with the Uniformity module in Imatest Master 5.1.9. In the Uniformity Settings dialog we want to select “ISO 18844 flare method C: Linearize with Color Space”. For this analysis, we can ignore the rest of the options.
Imatest will display several plots related to uniformity but we are only interested in the ISO 18844 flare plot for this test. The plots for each of our measurements can be seen below.
ISO 18844 Flare Visible light plot
ISO 18844 Flare NIR light plot
The plots show the % flare for each dot according to the 18844 calculation as well as the Mean Flare at the bottom. From the plots, we see that Mean Flare for the visible light sources is 5.2% and 13.6% for the NIR.
Conclusions
Similar to our previous test on sharpness, there is a drastic change in quality when the wavelength of light is varied. The ISO 18844 Flare increases more than 2.5 times in the NIR versus the visible. The expectation then is the NIR camera has a diminished dynamic range due to the increased flare. In applications such as security or automotive cameras, the diminished dynamic range could significantly decrease the effectiveness of the system to detect obstacles or intruders. The large change in quality shows how important it is to construct a test environment that matches the application of the imaging system. This is especially true when the system is multi-purpose, such as a security camera with a day (visible) and night (NIR) modes. A good follow up to this test would be to measure the dynamic range of the system in either lighting condition.
Having trouble with your Imatest test results? This article explains the five most common pitfalls in image quality testing, and how to resolve them.
You’re using a chart with overly high contrast, like the obsolete ISO 12233 test chart.
Your chart may have too high contrast edges. The most common example we see of this is the obsolete ISO 12233 test chart, although this can be the case with any test chart. If you are using the ISO 12233 2000 test chart, you should not be.
ISO 12233 test chart
The ISO 12233 2000 standard test chart has a very high contrast ratio (the white is very white and the black is very black) that can cause clipping in signal and lead to invalid MTF values. This is because the contrast of the chart often saturates the sensor, reporting higher MTF results than there might actually be. Therefore, we recommend using a chart with a 4:1 contrast ratio as revised in the ISO 12233 2014 standard, such as the ISO 12233 2014 chart or SFRplus chart. The lower contrast ratio produces more accurate measurements by providing MTF measurements that do not clip.
If you need help selecting the right contrast in a chart, email us at charts@imatest.com.
You’re using the wrong dynamic range chart.
When you’re testing dynamic range, you need to make sure you’re using a dynamic range chart which has a higher dynamic range than your camera to ensure you’re measuring the camera and not the chart. The common pitfall here is that people tend to use tonal step charts which have lower dynamic ranges than their sensors, such as the old Kodak Q14 or X-rite colorchecker. For example, if you are using a typical matte reflective chart which has a dynamic range of 48dB, and you’re testing a camera which has a dynamic range of 80dB, then the highest result you could get is 48dB. This is inaccurate.
However, if you’re using a chart with 120dB, you’ll get a much more accurate reading because it will measure the camera and not the chart. Before you test, you should check to see that your camera is not exceeding the dynamic range of what your chart is rated for. Once you have verified this, you will know which dynamic range chart to select. Our 36-Patch Dynamic Range chart is the most popular chart we sell with densities exceeding, 50db, 100db, and 150db.
Verifying the quality of your dynamic range chart.
The usage of linear dynamic range charts such as the Stouffer T4410 or Xyla target causes many problems including difficulty selecting dark regions that do not have visible contrast (especially using distorted lenses) as well as the linear density slope which interferes with the radial non-uniformity of the imaging system. The ISO standard dictates that each patch should be the same distance from the center and charts that have radial layouts of patches are much less impacted by light falloff. Also, targets which are very dark are not representative of more luminous scenes and overlook the impact of flare on dynamic range which leads to higher dynamic range performance, which will not hold up when the camera is exposed to a scene which is not mostly blackened. Imatest’s 36-patch targets include an option for a “DarkWorld” mask which can produce both dark and gray scenes using a single target.
Statistics from an image of XYLA-21: dark regions
are difficult to select.
If you need help determining which chart is best suited for you, please email us at charts@imatest.com.
Your chart quality is poor.
Similar to the dynamic range chart pitfall, when testing the sharpness of your camera system, you need to ensure the resolution of your chart is high enough for the device you’re testing. If your reflective or transmissive chart’s MTF is too low the accuracy of your camera resolution measurements will be limited by the resolution of the chart.
The culprit is print quality. The print process can affect the results of your test, especially at close distances or with high-megapixel cameras. For example, using a chart printed on a basic laser printer to test a high-resolution mobile phone camera, could reveal inaccurate results, shown by the steps in the MTF graph.
Example of poor chart quality (left) and good chart quality (right).
Avoid this pitfall by using a high-quality chart on higher resolution substrates such as inkjet or film, or by applying MTF Compensation to compensate for the chart quality. To ensure the highest quality, we encourage you to get charts printed from Imatest. If you don’t know what quality chart you need, contact charts@imatest.com.
You’re not filling the camera field of view.
When testing, it’s important to ensure you’re completely filling the camera field of view with the test chart. If the chart you are using does not include measurement regions near the corners of your field of view, your analysis will not be comprehensive and you may overlook problems with MTF or signal loss at the extents of the image plane. When measuring distortion, the full field of view can be extrapolated from points available within the center, but this is less accurate than having real points to measure. The following video helps illustrate how the field of view affects your setup.
This is particularly challenging for cameras with wide or ultra-wide fields of view where planar targets can’t fill the FoV. We offer specialized test equipment to measure resolution and uniformity of ultra-wide devices.
You’re not testing at the right focus distance.
Test distance is calculated by focal distance; in other words, the target needs to be set at the distance(s) in which it is in focus. The common pitfall we see is not testing at the right focal distance. A comprehensive test regimen will involve testing between the focal distance and the hyperfocal distance. You can’t always move closer to a target in order to fill the field of view, as this can fall below the minimum focal distance of your lens – or below the distance which a chart of a particular does not negatively impact measurements.
This chart is not at a distance which fills the lens field of view while in focus.
The chart needs to be within a distance to fill the lens field of view in focus. You may need to test at more than one focus distance if you have a lens with variable or automatic focus so that you can validate optimal focus across the range of focus of your device.
To find the minimum focus distance, check your lens specifications or contact the lens manufacturer. To find the maximum focus distance, or hyperfocal distance, see this hyperfocal distance calculator.
Imatest representatives will visit Beijing November 12-14, 2019 to offer a free information seminar and a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The training course on November 13 & 14 offers attendees insight on the full capabilities of Imatest software in both research development and manufacturing environments. Training starts at 9:00 and will end at 17:00 – 18:00, depending on questions. Location is TBD.
If you are interested in finding out more about how Imatest software can improve your image quality testing, we encourage you to come to our free information seminar on November 12th.
Please register if you are interested in attending.
Imatest representatives will visit San Jose, CA, April 22-23, 2019, to offer a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The Training Course on April 22 & 23 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments.
Training will run from 8:30 am to 5:30 pm each day, depending on questions. Visit our training page to view a detailed training schedule.
Imatest representatives will visit Shanghai, China, May 14-16, 2019 to offer a free information seminar and a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The training course on May 15 & 16 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments.
Training starts at 9:00 and will end at 17:00-18:00 depending on questions.
Location: Holiday Inn Shanghai Pudong
No. 899 Dongfang Road,Pudong New Area, Shanghai
+86 21 5830 6666
Ming Hall
If you are interested in finding out more about how Imatest software can improve your image quality testing, we encourage you to come to our free information seminar before the two-day Training Course:
Time: May 14 from 13:30 – 18:00
Location: Holiday Inn Shanghai Pudong
No. 899 Dongfang Road,Pudong New Area, Shanghai
+86 21 5830 6666
Han Hall
Imatest engineers will visit Seoul, April 17-19, 2019, to offer a free information seminar and a paid two-day training course to professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The training course on April 18 & 19 offers attendees insight on the full capabilities of Imatest software in both research and development and manufacturing environments. Training starts at 09:00 and will end between 17:00 – 18:00, depending on questions.
If you are interested in finding out more about how Imatest software can improve your image quality testing, we encourage you to come to our free information seminar.
Imatest representatives will host a two-day training course at our headquarters in Boulder, Colorado May 8 & 9, 2019 for professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The training course on May 8 & 9 2019 offers attendees insight on the full capabilities of Imatest software in both research development and manufacturing environments. Training starts at 8:30 am and will end at 5:30 pm, depending on questions. Please contact us about nearby lodging suggestions.
Recent growth in the automotive and security industries has increased the number of cameras designed for viewing both Near Infrared (NIR) and visible wavelengths of light. NIR illumination is invisible to the human eye and can light a dark scene without being visible or annoying. (more…)
Learn more about the development of standards for automotive camera systems.
The IEEE-SA P2020 is a working group for automotive imaging standards. Its goal is to define a set of standards to resolve the current ambiguity in the measurement of image quality in automotive imaging systems. (more…)
With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren
Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform. (more…)
Slanted-edge regions can often have non-uniformity across them. This could be caused by uneven illumination, lens falloff, and photoresponse nonuniformity (PRNU) of the sensor.
Uncorrected nonuniformity in a slanted-edge region of interest can lead to an irregularity in MTF at low spatial frequencies. This disrupts the low-frequency reference which used to normalize the MTF curve. If the direction of the nonuniformity goes against the slanted edge transition from light to dark, MTF increases. If the nonuniformity goes in the same direction as the transition from light to dark, MTF decreases.
To demonstrate this effect, we start with a simulated uniform slanted edge with some blur applied.
Then we apply a simulated nonuniformity to the edge at different angles relative to the edge. This is modeled to match a severe case of nonuniformity reported by one of our customers:
Here is the MTF obtained from the nonuniform slanted edges:
If the nonuniformity includes an angular component that is parallel to the edge, this adds a sawtooth pattern to the spatial domain, which manifests as high-frequency spikes in the frequency domain. This is caused by the binning algorithm which projects brighter or darker parts of the ROI into alternating bins.
Compensating for the effects of nonuniformity
Although every effort should be made to achieve even illumination, it’s not always possible (for example, in medical endoscopes and wide-FoV lenses).
Imatest 4.5+ has an option for dealing with this problem for all slanted-edge modules (SFR and Rescharts/fixed modules SFRplus, eSFR ISO, SFRreg, and Checkerboard). It is applied by checking the “Nonuniformity MTF correction” checkbox in the settings (or “More” settings) window, shown on the right.
When this box is checked, a portion of the spatial curve on the light side of the transition (displayed on the right in Imatest) is used to estimate the nonuniformity. The light side is chosen because it has a much better Signal-to-Noise Ratio than the dark side. In the above image, this would be the portion of the the edge profile more than about 6 pixels from the center. Imatest finds the first-order fit to the curve in this region, limits the fit so it doesn’t drop below zero, then divides the average edge by the first-order fit.
The applied compensation flattens the response across the edge function and significantly improves the stability of the MTF:
Summary
For this example, Imatest’s nonuniformity correction reduces our example’s -26.0% to +22.8% change in MTF down to a -3.5% to +4.7% change. This is an 83% reduction in the effect of the worst cases of nonuniformity.
MTF50 versus nonuniformity angle without [blue] and with [orange] nonuniformity correction
While this is a large improvement, the residual effects of nonuniformity remain undesirable. Because of this, we recommend turning on your ISP’s nonuniformity correction before performing edge-SFR tests or averaging the MTF obtained from nearby slanted edges with opposite transition directions relative to the nonuniformity to reduce the effects of nonuniformity on your MTF measurements further.
Detailed algorithm
We assume that the illumination of the chart in the Region of Interest (ROI) approximates a first-order function, L(d) = k1 + k2d, where d is the horizontal or vertical distance nearly perpendicular to the (slanted) edge. The procedure consists of estimating k1 and k2, then dividing the linearized average edge by L(d).
k1 and k2, are estimated using the light side of the transition starting at a sufficient distance dN from the transition center xcenter, so the transition itself does not have much effect on the k1 and k2 estimate. To find dN we first find the 20% width d20 of the line spread function (LSF; the derivative of the edge), i.e., the distance between the points where the LSF falls to 20% of its maximum value.
dN = xcenter + 2 d20
If the edge response for x > dN has a sufficient number of points, it is used to calculate k1 and k2 using standard polynomial fitting techniques. The result is a more accurate representation of the edge with the effects of nonuniformity reduced.
Future work
Consider the 2D nonuniformity across the ROI before sampling the 1D average edge
Use an image of a flat-field to perform nonuniformity correction within Imatest
Consider the impact of noise which was not included in this study
Incorporate enhancements to the slanted-edge algorithms into future revisions of ISO 12233
For any questions on how to do this, or how we can help you with your projects, contact us at support@imatest.com.
We now offer a complete, customizable image quality testing solution for security camera systems to provide our customers with an easy, effective way to outfit their labs. While working with some of the top security camera manufacturers, our engineers have compiled all of the necessary lab materials in a convenient package. (more…)
This course has ended. Please see our schedule to see and register for future training.
Imatest in San Jose
Imatest representatives will visit San Jose September 5-6, 2018 to offer a paid two-day Training Course to professionals using or considering Imatest software to improve their image quality testing processes.
Two-Day Training Course
The Training Course on September 5 & 6, 2018 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments.
Training will run from 8:30 am to 5:30 pm each day, depending on questions. The event will be located in the Larkspur Landing Hotel in Campbell.
This event has ended. Please see our schedule for future training.
Imatest representatives will visit San Jose August 1-3, 2018 to offer a free information seminar and a paid 2-day Training Course to professionals using or considering Imatest software to improve their image quality testing processes.
2-day Training Course
The Training Course on August 2 & 3, 2018 offers attendees insight on the full capabilities of Imatest software in both research & development and manufacturing environments.
Training will run from 9:00 am to 5:30 pm each day, depending on questions. The event will be located in the Larkspur Landing Hotel in Campbell.
Our engineers are continually adding new features and updating Imatest software to provide you with the best analysis tool on the market. In our most recent release, our team has added several new modules to provide you better analysis tools. (more…)
It is important to test your camera system in environments which reproduce lighting conditions similar to where you intend to use the camera in the real world. Failure to test a camera under low light conditions may lead to overstating the camera’s performance. (more…)
[contact-form-7 id="38213" title="Master Subscription"] 
Windows 64 bit
Mac OSX
Linux
MTF50 versus nonuniformity angle without 
