News

Staying in touch with imatest

July 27, 2018July 27, 2018 Gage

Introducing Imatest 5.1

Our engineers are continually adding new features and updating Imatest software to provide you with the best analysis tool on the market. In our most recent release, our team has added several new modules to provide you better analysis tools. Here are the top three new features, and why should upgrade to Imatest 5.1:

Compensate for chart MTF to perform accurate high-resolution or close-distance sharpness tests

MTF Compensation reduces the impact of the chart on measured imaging system MTF.

Geometrically calibrate single or multi-camera devices obtaining camera intrinsic and extrinsic matrices for accurate machine vision and ADAS applications

A geometrically calibrated device provides a mapping between pixel coordinates and real-world position.

Direct image acquisition for Sony sensors

Imatest now supports direct image acquisition from Sony sensor development boards through the Sony AYA software tool.

Learn More

July 16, 2018July 17, 2018 Gage

Why is it Important to Test in Low-Light?

It is important to test your camera system in environments which reproduce lighting conditions similar to where you intend to use the camera in the real world. Failure to test a camera under low light conditions may lead to overstating the camera’s performance.

Image sensors collect light (signal) into pixel wells, then convert the resulting analog voltage levels into digital numbers. Dark current is where electrons are released from thermal activity which becomes indistinguishable from electrons released via photoresponse. The dark current that exists in uncooled image sensors leads to dark noise. At low light levels, exposures are longer to gather more light, which gives more time for dark-current electrons to be gathered. This leads to dark noise and read noise representing a larger portion of the overall response, which reduces the signal to noise ratio (SNR). Low-light conditions are most challenging for higher resolution sensors with small pixel pitches where the particle and wave nature of light can seriously impact the performance of your camera.

The definition of “low light” depends on the application. For mobile devices (compact camera modules) low light is defined by the IEEE CPIQ standard as 25 lux, which resembles a dimly lit indoor space representative of the worst cases where people expect their phones to function properly. As mobile devices get better in low light scenarios, customer expectations may shift. For security or automotive industries, the levels are much lower based on how dark the outdoor environment may get^[1]:

Condition	Illumination (Lux)
Sunlight	107527
Full Daylight	10752
Overcast Day	1075
Very Dark Day	107
Low-Light (CPIQ)	25
Twilight	10.8
Deep Twilight	1.08
Full Moon	0.108
Quarter Moon	0.0108
Starlight	0.0011
Overcast Night	0.0001

In order to achieve light levels similar to the darkest scenes, most lab lighting setups that contain fluorescent or LED sources may not be able to be directly dimmed to very low levels, so the dim light levels may be achieved by a variety of methods shown in the following table. Ultra-low-light levels can be achieved by combining these approaches:

Method	Light Reduction	Notes
Increasing distance (d)	Between 1 / d and 1 / d²	Difficult to move large lighting setups, labs have limited available space
Reflecting off Munsel N5 painted walls	0.18	Most labs are painted with these sorts of walls, but rotating lights back and forth may be difficult to perform repeatably without a mechanical motion stage.
Reflecting off black painted walls	0.05
Fresnel reflection from beam splitter	0.04	Apparatus could impinge FoV and would also require repeatable rotation of lights.
Imatest Low-light filter (mask + neutral density filter)	0.0125	Manually attaches to these LED lights

Eliminating additional sources of light within your lab is also important: Putting felt around doors to block light from other rooms, covering exit signs, covering indicator LEDs and power strips and installing baffling and light traps to block unwanted reflections.

Note: Verifying the low light levels can be challenging as many illuminance meters do not go to extreme low light levels.

Once you can reproduce low-light levels in your lab, you can perform tests to answer a number of questions about your system performance:

Does the radiometric calibration for black level successfully reduce the effects of dark current without also eliminating useful signal that might impact the dynamic range of your system?
Does the black level compensation work across the range of nominal operating temperatures that your sensor will experience?
Does your ISP react appropriately to dark scenes by increasing the sensitivity (gain)?
How consistently does your autofocus system (if you have one) perform under low light?
What happens to moving objects captured under low light conditions?
For security cameras with near-infrared ‘night-vision’ support: does the Infrared cutoff filter engage and disengage at the appropriate light level?
Do the applied gamma curve(s) and tone mapping help to boost the perceived quality of the dark scene or assist the observer’s ability to recognize objects in the dark?
Does the noise reduction strike the right balance of reducing visual noise without also blurring the appearance of relevant textures (skin and foliage) within the image?

A comprehensive tuning and testing regimen will involve performing a full battery of objective tests in a wide range of lighting conditions. By increasing the range of light levels you can reproduce in your lab, you can provide the most challenging condition in which you can validate the performance of your camera system.

Photo by Tuur Tisseghem from Pexels

June 20, 2018June 27, 2018 Gage

How to Test Dynamic Range:
A step-by-step use case with the Pixel 2 XL

Outline

Google Pixel 2 XL

In this post, we will be using the Contrast Resolution Chart and Imatest Master to measure the dynamic range of a Google Pixel 2 XL. The dynamic range of a camera is the reproducible tonal range in an imaging system. Put simply, it is the range between the darkest black and the brightest white of an image and is typically measured in decibels (dB). It is an important image quality factor in many applications from machine vision to mobile cameras and more recently, automotive camera systems. In this use case, we will be evaluating both the HDR+ (default) and HDR off modes of the Pixel 2 XL; however the procedure can be used to test any camera system’s dynamic range. You can read more details about Imatest’s dynamic range test solutions.

Setup

Testing a camera’s dynamic range can be difficult and requires careful setup. In this section, we will review all of the relevant components of performing a sufficient dynamic range test.

Chart

How to Test Dynamic Range - Contrast Resolution Chaart

Contrast Resolution Chart

One of the major components to testing dynamic range and the first consideration is the chart. There are a variety of dynamic range test charts available but the one we will be using for this test is the Contrast Resolution chart. There are several unique features to this chart that make it ideal for testing a system like the Pixel 2 XL. The first is the radial arrangement of the patches. This is important because it helps control for lens fall off. Second is the dynamic range of the target itself. The chart must have a higher dynamic range than the system or else you end up measuring the chart and not the camera. The Contrast Resolution chart has a dynamic range of ~95 dB which is sufficient for almost all systems with a lens. The third and most important reason for choosing this chart is the Contrast resolution dynamic range calculation. The inner light and dark gray patches on this target allow for a more robust calculation of the dynamic range of tone mapped images; which can adversely affect the quality of other dynamic range calculations.

Lighting

ITI Lightbox with Contrast Resolution Chart

The next component to consider is the light source. For our test, we will be using the 10,000 lux ITI Lightbox 5100K. This lightbox is a good choice for our purposes because of its uniformity (>95%). The analysis of the chart assumes a constant amount of light being transmitted across and through the chart. The contrast resolution dynamic range calculation works because the chart has known density steps between the patches. If the light source has low uniformity than the assumption is not true and the results can be impacted.

Environment

Test setup

The third consideration is the environment you will test in. It is important to test dynamic range in a sufficiently dark environment limiting the amount of stray light that might enter the camera system as much as possible. The extra light entering the system can increase the amount of flare light (sometimes called veiling glare). Flare is the stray light that fogs images and is caused by reflections between surfaces of lens components and the inside barrel of the lens. Stray light can cause reduced contrast in dark parts of the image which in turn can make for erroneous dynamic range measurements. See our recent study: “Measuring the impact of flare light on dynamic range” for more information. Our testing environment will have only one controlled light source to illuminate the chart and no other sources of light (e.g., overhead lighting, LEDs on electronics, etc.).

A common source of erroneous measurements due to the testing environment are reflections on the chart itself. Because the darkest patches of the target let through such little light, it is easy for a non-negligible amount of stray light to be reflected off the front side of the chart. This extra light will make it seem as though more light is present in the patch area than is actually being transmitted through the chart and will adversely influence the analysis. To mitigate the reflections, we use a custom shroud to block surfaces that can reflect light from the lightbox. Unfortunately, the camera itself is the most likely culprit for unwanted light due to the potential for light to bounce off of it. The Pixel 2 XL in our case, is no exception. The particular model we are using has a white back which reflects lots of light back at the chart’s front surface. Since we are aware of this, we have taken extra care to shield all of the phone except for the lens from the light to stop any unwanted light from bouncing back at the chart.

Example of a development board reflection off the front of the chart

More on the shroud

Depending on the system, and the environment of the test, a shroud may not be necessary. However, it provides a consistent lighting environment for every test and should be considered for benchmarking and comparing different systems. For more information on reflections and shroud materials, see this post. For information on custom testing environments, please contact sales@imatest.com .

Capture Procedure

Contrast Resolution chart viewed through the native camera app

Ten images were captured with the Pixel 2 XL with the above setup. The phone was set to HDR+ mode. For each image, the exposure was adjusted on the device by tapping the area on the screen where the upper gray background of the chart was present (directly to the left of the second row). We did this to ensure similar exposure in all of the images. The acrylic shroud with matte black interior was placed around the lightbox with the phone placed as close as possible to the opening on the shroud. For this test, the lightbox was set to the 5100K color temperature setting and minimum brightness (~30 lux). We chose these settings because they are representative of a realistic low-light environment in which the camera phone may be used. Please note, the light settings could vary greatly depending on the application. Once enough images were taken, we inputted them into our software, Imatest Master, for proper analysis.

Analysis

The following analysis of these images was done in Imatest Master 5.0.

Each image was analyzed in the Multitest module. Opening the Multitest Setup, we made sure to check that “3. Contrast Resolution” was selected for the chart type.

The plot we were most interested in was the Noise/SNR plot. Under this section in the setup dialog, we selected “3. SNR dB vs. Inp Density (RGBY)” and “3. Exposure in DB (-20*target density) DR in dB” from the drop downs. We also checked “Image plot” so we could visualize the results on the images themselves. See here for more information analyzing the Contrast Resolution chart.

The data was saved in JSON format and the relevant metrics were then parsed and averaged. The data we are interested was saved in the field “Contrast_Resolution_Dyn_Rng_dB”. The field contains 4 values, the Contrast Resolution dynamic range in decibels at SNR = 0, 6, 12, 20.

Results

The plot below shows the Contrast Resolution Dynamic Range for the two modes averaged over ten captures. The standard error is included. We aggregated this data from several runs of the Multitest module in Imatest Master. As you can see in the plot below, there was a significant increase in dynamic range when using HDR+ mode versus HDR off.

Chart comparison of the HDR-off vs. HDR+ results

NOTE: If you are using Imatest software, this information can be viewed for individual images using the Noise Plot in the Multicharts module.

Since we checked “Image Plot”, we can also visualize the results on the image of the Contract Resolution chart. The two plots below show an exaggerated image plot so that the noise in each patch can be visually examined.

HDR Off Image Plot (Constant xyY)

HDR+ Image Plot (Constant xyY)

A notable difference in the plots can be seen in patch 11 (isolated images below). We can see that in the HDR+ plot (right), the details of the inner gray patches are still visible where they are not in the HDR off plot (left).

Patch 11 visual comparison (Left: HDR Off, Right: HDR+)

Conclusion

By following this procedure, using the Contrast Resolution chart and Imatest Master, we were able to analyze the dynamic range of our camera effectively. As we can see in the above diagrams, the summary plot results indicate that the HDR+ mode increased the dynamic range in the images by 5 dB on average. We are also able to visually see the increased range by looking at patches in the Image Plots view.

Ideally, we want to tailor our test setup as much as possible to the application being tested. In this particular case, the subject of our testing is the different HDR processing of the Pixel 2 XL. We have limited information on the specific algorithms being used when HDR is turned off versus HDR+ and their impact on image quality under different lighting situations and with different target subjects, so we are limited in the conclusions we can draw. Thus, the results reported above do not represent all use cases. A more exhaustive study would need to be done to produce results that would reflect the broad range of possible use cases.

For more information, or to learn more about how to properly set up your testing environment, talk to a solution expert.

June 12, 2018July 17, 2018 Henry Koren

High-contrast edge-SFR test targets produce invalid MTF results

The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio, These are typically produced at the maximum dynamic range of a printer which can be anywhere from 40:1 to 80:1. The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values.

Some camera manufacturers who want better MTF results may take advantage of this anomaly to overstate the quality of the cameras they produce. This is why it is critical to validate cameras with a proper measurement system that includes a low-contrast target. (more…)

January 25, 2018February 2, 2018 Henry Koren

Measuring the impact of flare light on dynamic range

Abstract

The dynamic range of recent HDR image sensors, defined as the range of exposure between saturation and 0 dB SNR, can be extremely high: 120 dB or more. But the dynamic range of real imaging systems is limited by veiling glare (flare light), arising from reflections inside the lens, and hence rarely approaches this level. Veiling glare measurements, such as ISO 18844, made with black cavities on white fields, result in large numbers that are difficult to relate to dynamic range. Camera dynamic range is typically measured from grayscale charts, where veiling glare depends on the design and layout of the chart, leading to inconsistent results. (more…)

November 29, 2017December 4, 2017 Imatest Admin

Using Arbitrary Charts for Development of the P1858 Variation Combo Chart

The release of Imatest 5.0 introduced a number of powerful new features, including the Arbitrary Charts module which enables Imatest analysis of test chart designs which would be otherwise unsupported by the software. This new module allows user-defined chart layouts for any situation which requires one.

The primary concept of Arbitrary Charts is that the user supplies a chart-definition text file which declares the location and properties of features on a test chart. The features can be placed in essentially any configuration and Imatest will still be able to automatically analyze the chart.

Here at Imatest, we believe we’ve designed some very comprehensive test charts, such as our flagship SFRplus and eSFR-ISO charts. These charts are designed to provide a number of image quality factor measurements, work with an extremely wide range of cameras, are available in pre-distorted versions for wide-angle lenses, and more. They each have their own dedicated modules in Imatest which recognize these chart designs.

While we highly recommend the use of these and our other standard charts, we also recognize that there are situations which may require a more custom target. For example:

Working with a legacy database of test images of an older chart
Meeting a test spec which requires specific, non-standard targets
custom reticles for collimator systems
Special combination charts for accelerated testing

P1858 Combination Chart Conception

As mentioned above, some situations require a custom target, including the need to combine charts for accelerated testing. Early in 2017, a customer and frequent collaborator approached us about adding support for a new chart. The IEEE p1858-2016 Cell Phone Image Quality standard requires images of at least five different standard test charts for the complete set of seven metrics. The customer wanted to significantly reduce the need for several charts by developing a test chart that provides a “good enough” measurement of all of the metrics from a single image. We worked together with them, jointly developing what would become the P1858 Variation Combo Chart and the Arbitrary Charts Module.

The immediate goal in this case was to reduce testing time of a device by at least a factor of five. (This is likely a low bound on the time savings, as switching between test targets is typically the most time-intensive part of a well-tuned automatic test bench.) The customer wanted results that, even if they weren’t technically in spec, would provide an indication of how a test device would score on a full-fledged CPIQ test, and wanted them to be produced quickly.

This would, of course, require a set of compromises on the CPIQ metrics:

SFR measurements from just outside of the center of field, instead of dead center
24 color patches based on the X-Rite ColorChecker instead of the 140 of the ColorChecker SG
Color Uniformity results measured from the chart background areas between the other features, rather than an entire flat-field image
Chromatic Aberration and Geometric Distortion measurements from slanted edges around the field, instead of the dot pattern

These changes from the CPIQ spec were deemed acceptable for this purpose, and the P1858 Combination Chart was born. This design is now publically available, with the approval of the original customer.

Using the Arbitrary Charts Module

We have been very happy to collaborate with Imatest on their new 5.0 release to allow us to design a combination chart that reduces our objective metric capture needs, improving our productivity and eliminating the challenge of correlating data from multiple capture conditions.

– Lead Imaging Scientist, Imatest Customer

At first glance, the P1858 Combination Chart looks very similar to Imatest’s standard eSFR-ISO chart.

Imatest eSFR-ISO Chart

P1858 Variant Combination Chart

The most obvious difference here is that there is the dead leaves texture pattern in the center of the chart instead of a slanted square. However, there are many other subtle differences which would make this completely unusable by our standard eSFR-ISO module:

The grayscale OECF patches have been enlarged and moved outward
Extra color patches have been added to approximate all of the patches on the X-Rite ColorChecker
Four individual slanted edges have been added on the inside of the OECF patches, so SFR measurements are still possible near center of field
In general, the positioning of wedges and slanted squares is different

These differences preclude any standard module in Imatest from performing an automatic analysis of the chart (though selections of slanted edges and wedges could be made manually in individual images using the SFR and Wedge modules, respectively – not a great solution).

Fortunately, this chart was designed in concert with the development of the Arbitrary Charts module, with the express purpose of making analysis of this sort of chart automatic and systematic. Moreover, not only can the combined results be produced from a single image but they are produced from a single module run in Imatest. Instead of running a module to analyze the color, a module for sharpness, a module for uniformity, etc., the Arbitrary Charts module simplifies the testing procedure and output parsing.

The ability to configure our own charts is revolutionizing our approach to quantitative image quality testing and test development.

-Lead Imaging Scientist, Imatest Customer

As an added bonus, the customer is able to make slight changes to this design at will and simply update the chart definition file supplied to the module, without having to wait for new builds of Imatest to catch up.

Learning to Use Arbitrary Charts

The capability to analyze user-defined charts is new in Imatest 5.0, and is under continual development. New capabilities are being added to the Arbitrary Charts module with each release.

Further information about the Arbitrary Charts module and its current capabilities can be found on its documentation page.

The most important novel aspect of working with this module is the introduction of the chart definition file which enables the software to understand images of the test chart. To help communicate the role and construction of this file, we are producing a series of short videos that describes the process.

September 21, 2017March 23, 2018 Imatest Admin

Five reasons to use Imatest 5.0

This week, we launched Imatest 5.0. We built Imatest 5.0 to provide users with a streamlined workflow for more efficient image quality analysis. Here are five reasons to start using Imatest 5.0 today.

Refined User Experience

Imatest 5.0 features several new tools and enhancements across our products to help you improve your workflow. Whether you want to reduce the number of images needed for complete system qualification with Arbitrary Charts; process a collection of images with automated analysis routines within the brand new Test Manager, or process batches of images with our enhanced image processing module, Imatest 5.0 is an improved experience for users needing an expedited testing process.

Innovative Image Quality Measurements

The newest release provides several new and enhanced measurements to help your focus on the image quality factors that matter to your business’ camera system.

Contrast Resolution was designed for the visualization and measurement of contrast separation. This is particularly important for the automotive industry where camera systems need to distinguish between low contrast objects in a larger field with varying ranges of brightness.
The ISO 18844 flare measurement added to the Uniformity module was created to address the need to accurately measure veiling glare.
The enhancements to the SFRreg Center Chart provides geometric utilities for measuring tilt and rotational measurements, an especially important factor for users needing to test wide-angled measurements.

Measuring Contrast Resolution using Multicharts.

Focus on the image quality factors that matter most

Imatest tools serve many industries with widely varying requirements, for example testing an automotive camera system is different from testing a consumer device. Even within one industry, companies need to test several image quality factors on several test charts, complicating the capture analysis workflow. Imatest 5.0 lets you zero in on the image quality factors and results that matter most to your business.

Customizable reports allows users to more efficiently filter and present select image quality data from a variety of image quality factors.
The new arbitrary charts functionality serves a similar purpose by letting users define efficient layouts of analysis features within one chart.

Enhanced support for industry standards

Industry standards provide a foundation for the testing and analysis of imaging systems and clearly define testing methods for reliable measurements of system performance. Imatest 5.0 goes along way to support and supplement industry testing standards.

Run predefined test plans for standard testing procedures such as CPIQ within the new Test Manager.

The new P1858 Variant Combo chart, supported by Arbitrary Charts, combines several analysis features required by the standard into one chart. This enables users to minimize the number of images needed in the testing process and increases testing efficiency and accuracy.

Imatest 5.0 also features an enhanced Uniformity module to support the ISO 18844 standard for image flare measurements on digital cameras. The standard was created to address some of the difficulties of making ISO 9358 measurements with cameras.

P1858 Variant Combo chart

Streamlined Product Line

The Imatest 5.0 release includes the streamlining of the Imatest product line. As of Imatest 5.0, Imatest Master will now includes image acquisition capabilities that were previously available in Imatest IS, which is now discontinued. Imatest users will now have access to the acquisition library. The library supports direct acquisition from a wide range of frame grabbers and cameras, as well as industry standard interfaces like CameraLink, GigE Vision, and USB through DirectShow (Windows) or QuickTime (OS X). Direct image acquisition cuts out several steps in the image quality testing process and allows for in-the-loop testing with Imatest.

In order to maximize performance in the new Test Manager, we utilized parallel processing. Previously, parallel processing was only available in Imatest IT-P. With the release of Imatest 5.0, Imatest IT-P functionality is now included in Imatest IT . This enables all Imatest users to rapidly process large volumes of images. By utilizing high-performance processors with 6+ cores, parallel processing can improve throughput by over 2.5x.

Learn More Start Free Trial

September 15, 2017September 29, 2017 Imatest Admin

Imatest IS Discontinued with Image Acquisition now Available in Imatest Master

As of Imatest 5.0, Imatest Master now features image acquisition capabilities. Previously, image acquisition capabilities were supported by Imatest IS, which has been discontinued as a separate product. This provides all of our customers with access to the acquisition library. The library supports direct acquisition from a wide range of frame grabbers and cameras, as well as industry standard interfaces. Direct image acquisition cuts out several steps in the image quality testing process and allows for in-the-loop testing with Imatest.

Learn more about Imatest’s image acquisition capabilities.

Sensor Evaluation Boards

OnSemi
Omnivision
Toshiba
STMicroelectronics
Graphin EasyLab

Capture Interfaces

Android
DirectShow
QuickTime
Analog
Camera Link
GigE Vision
DCAM IEEE 1394b
Blackmagic Design

Camera Vendors

Adimec
Allied Vision Tec
FLIR
Hamamatsu
Hitachi
Matrox Imaging
Microsoft Kinect
National Instruments
Sony
Toshiba Teli
GEViCAM
Imperx
Lumenera
Matrix Vision
Teledyne DALSA
QImaging
VRmagic
XIMEA

1 of 12
1
2
3
…
12
Next »