Recent growth in the automotive and security industries has increased the number of cameras designed for viewing both Near Infrared (NIR) and visible wavelengths of light. NIR illumination is invisible to the human eye and can light a dark scene without being visible or annoying. Because silicon sensors are sensitive out to 1100 nm — well beyond NIR wavelengths of interest — existing consumer systems require only a small modification, the removal of the infrared filter, to allow them to image NIR. This makes dual-band security cameras cost-effective and attractive to the industry.

The Response of Silicon Photodiode

Care is required when dual-band cameras are tested for image quality. In this post, we will outline some best practices, including a selection of light sources and charts for measuring sharpness of cameras in the visible and NIR bands.

Note: We chose to measure sharpness because a dramatic change between bands could greatly reduce the system’s effectiveness. A critical function in many automotive and security systems is object identification or detection. A security camera that cannot identify an intruder because the image is not sharp enough is not doing its job, therefore it is one of the most important factors to test.

Camera

The camera we will test is the Raspberry Pi Camera V2 NoIR version. It is ideal because its IR filter has been removed, and functions identically to the normal Raspberry Pi camera module, which leads us to believe that it has identical image processing.  

Lighting

As with any other test, the choice of lighting is crucial. In our setup, we will be testing under two specific lighting conditions: 5100K visible LED light and 850nm NIR LED light.

The first test is with the 5100K LED source.

The plot above shows the spectral output of the 5100K source. There is a strong peak near 450 nm, but the spectrum is fairly constant across the visual range (400 – 700 nm). For more information about this light source, see our ITI LED lightbox.

The second source is a lightbox engineered with NIR LEDs with the spectral output centered around 850 nm. The plot below shows this distribution.

Contact sales@imatest.com to learn more about our NIR lighting options.

It is important to avoid overlapping spectral response because it could confuse results from the two sources. We need to define how the system performs in each band. If the light sources overlap, we won’t be able to point to which band caused any change in quality.  

Chart & Material

We use an eSFR ISO 12233 on a black and white LVT film to measure sharpness. There are a number of ways to measure sharpness and Modulation Transfer Function (MTF) such as wedges and Siemens stars, but for our purpose, we will use the slanted edges on the eSFR chart. For a better understanding of the advantages and drawbacks of each feature when measuring MTF, see this blog post.

The chart and material on which it is printed or developed are two of the most overlooked aspects in testing at NIR wavelengths. Not all reflective and transmissive charts perform the same in NIR as they do for visible light. For example, there are two types of film charts: color, and black and white. Both types work well in the visible wavelengths with only subtle differences. However, the color film is nearly transparent at longer wavelengths. This is demonstrated below.

We use slanted-edges in the eSFR ISO (ISO 122334:2014) chart, printed on Black and White (B&W) LVT film as our preferred method for measuring sharpness in NIR. The choice of chart media is critical since NIR response for some media may be very different from the visible response. The B&W film has excellent NIR response (extending to shortwave IR). Color LVT film charts cannot be used because they are transparent in NIR wavelengths. This is demonstrated below.

We can further understand the properties of these charts by looking at the spectral transmittance for each material (illustrated below).

The above plot shows that the color film’s transmission increases dramatically at NIR wavelengths. In contrast, the B&W film maintains a similar transmission throughout visible and NIR, making it ideal for this test.

The table below compares the NIR suitability of several chart materials.

*For inkjet, a special print process is required. All inkjet charts are suitable for visible testing but not all are suitable for NIR. More info available at request by emailing charts@imatest.com.

Testing

We compare the sharpness of the same camera with visible and NIR illumination using a black and white film eSFR ISO chart. Images are analyzed using Imatest Master 5.1, with MTF50 (the spatial frequency where MTF drops to half its low-frequency value) as the image sharpness metric. Note that this is one of several summary metrics that could have been chosen. For more information, please visit our page, Sharpness, what is it and how is it measured?

We capture images of the same chart with the two different light sources (see below).

eSFR film with 5100K (visible) light source

eSFR film with the 850nm (NIR) light source

Even before running an analysis of these images, we inspect for qualitative differences. The immediate observation is that the visible image is much sharper.

(left) visible, (right) NIR

The above images show crops of the center square of both images. The visible image on the left has much sharper edges and a clearer focus aid feature in the center. The difference is important to quantify, so we run these images through the Rescharts module in Imatest Master for proper analysis.

Results

Below are the Multi-ROI summary results output by Imatest.

Multi-ROI Summary for 5100K image

Multi-ROI Summary for 850 nm image

We choose a large number of ROIs to increase the data points for each image in order to get more representative results. These plots can show a variety of metrics for each edge (MTF20, Chromatic Aberration, etc.), but we are showing the primary result which is the weighted mean of MTF50 (summarized in the graph below). For more information about understanding and interpreting these plots, read our documentation on it.

Conclusions

The plot shows that there is a 52% change in mean MTF50 between the two light sources. This difference is quite significant and correlates well with our visual observations. There are several explanations: NIR focuses at a different location from visible light and has intrinsically lower sharpness due to its longer wavelength. There is more diffusion inside the silicon sensor. The difference may be reduced by a better choice of optics, better focusing, and improved sensors, but in general, we should expect the sharpness of NIR images do not equal that of visible light.

Wrap Up

Before you test your cameras for NIR, there are a couple things you need to consider:

• The quality of the image can vary per band (visible & NIR)
• Not all chart and material types are visible in NIR. However, certain charts and materials when printed or developed for NIR can work for both visible and NIR testing

If you have additional questions about testing NIR, contact us at charts@imatest.com.

Our engineers can help you set up your lab to test NIR. Submit a solution request and we will fit you with the right equipment.

Learn more about the development of standards for automotive camera systems.

Overview

The IEEE-SA P2020 is a working group for automotive imaging standards. Their goal is to define a set of standards to resolve the current ambiguity in the measurement of image quality in automotive imaging systems. Generally, today’s image evaluation approaches do not adequately address the unique needs of either the human or computer-vision based automotive applications; therefore IEEE-SA P2020 is working with people in the field, understanding the gaps in current standards, and creating a coherent set of key performance indicators by which automotive camera systems might consistently be evaluated.

Our Involvement

Since the beginning of this working group, Imatest’s engineers continue to be heavily involved in the IEEE P2020 standard for automotive camera systems. The two areas of focus are Contrast Detection Probability and Flicker Measurement. We are actively working with major image sensor manufacturers to assess Contrast Detection Probability as a practical test measurement for evaluating the amount of contrast information present at various parts of the imaging chain. Additionally, we are the principal contributors towards the Flicker Measurement subgroup, and are preparing a publication to describe the effect further and why it is an essential consideration for ADAS and automated driving systems.

The Future

There is much to come as IEEE P2020 leads the creation of standards for automotive camera system testing. As automobiles evolve, autonomous vehicles mature, and the car becomes an artificial-intelligence platform connected to everyday life, it will be crucial to adapt and define proper evaluation metrics for the multi-sensor, multi-camera, and multi-application systems on an automobile. The IEEE P2020 working group involves leading camera and sensor manufacturers, leading automobile manufacturers, and leading imaging science engineers, and will leverage all the various backgrounds and experiences to create and adopt this standard for all automobile and part manufacturers. Download the most recent publication below from IEEE P2020.

If you have any questions for us on how this standard can help your projects, please email Ian Longton at ian@imatest.com.

DOWNLOAD THE WHITEPAPER

Distribution:

All rules about how the material can be used, copyrights, legal caveats etc. are contained within the document. It must only be shared in full so that these details are clearly indicated.

To stay informed on the work and progress of this standard, you can visit our standards page or look at the P2020 working group site directly.

By Robert Sumner

With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren

Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform.

Since Imatest makes measurements directly from the image pixels, any source that adds noise to the image can affect measurements. A primary source of noise in images is electronic sensor noise. Photon shot noise also contributes significantly in low-light situations. Other systemic sources of measurement variability, such as autofocus hysteresis, will not be addressed in this post.  

In order to reduce variation in your sharpness results and increase test repeatability, you should take steps to decrease the amount of noise in your image.

Here are 5 tips to limit noise in your test results: (more…)

We now offer a complete, customizable image quality testing solution for Security camera systems to provide our customers with an easy, effective way to outfit their labs. While working with some of the top security camera manufacturers, our engineers have compiled all of the necessary lab materials in a convenient package. The package includes options for software;  reflective, transmissive, tunable, and ultra-wide light sources; test fixtures for a variety of fields-of-view and focus distances; and applicable test charts to analyze crucial image quality factors.

Software

The Security package includes two software options, Master and Ultimate. The Master version, designed for R&D testing, is a robust analysis tool to help you measure any and every image quality factor of your camera system. The Ultimate version combines Master and IT, an automated production line testing version, and is a cost-effective solution for customers who manage R&D and production testing. Read more about our software version and license options here.

Test Fixtures

We include several test fixtures that help test varying fields-of-view (FOV) and focus distances properly. For ultra-wide testing with fields-of-view up to 270˚, our Ultra-Wide Test Fixture works perfectly to help analyze image quality factors such as sharpness and chromatic aberration using our eSFR ISO and SFRreg charts. If you have long-range testing requirements, we’ve included several collimator lenses and target projection systems to help you to test at or beyond your focusing distance. To automate and provide more flexible positioning, add one of the robotic automation fixtures from our partners Sofica and IQL.

Light Sources

Choose from a variety of reflective, transmissive, tunable, and ultra-wide lighting system options. We have included our most widely used reflective light source; the Kino Flo FreeStyle 31 LED DMX for its adjustability, portability, and high-illumination output. To test dynamic range, and high-resolution systems, select a uniform, transmissive LED light box with Lux levels ranging from 10,000 – 100,000. The brightest lightbox enables you to achieve IEC 62676 compliance. For the most adjustability in spectral and Lux levels, and for ultra-wide field-of-view uniformity testing, choose one of our tunable light sources such as the wide-FOV version of the Gamma Scientific RS-7.

Test Charts

To make your selections easier, we have included the most appropriate test charts to meet a variety of requirements for security cameras. For reflective resolution needs, select from our ISO 12233:2014 compliant multi-feature test charts, a Checkboard chart, or sinusoidal siemens star pattern chart. For use with the transmissive, uniform light sources and dynamic range and high-resolution testing, choose to use our ISO 12233:2014 chart printed on High-Precision LVT Film, and our Ultra-High Dynamic Range test chart.

View the package, the options included, and receive a quote by visiting our store.

Of course, we know that not everyone’s requirements are the same, so if the package doesn’t seem to be a good fit for you talk to a solution expert and we will customize a solution for your unique needs.

Our engineers are continually adding new features and updating Imatest software to provide you with the best analysis tool on the market. In our most recent release, our team has added several new modules to provide you better analysis tools. Here are the top three new features, and why should upgrade to Imatest 5.1:

• Compensate for chart MTF to perform accurate high-resolution or close-distance sharpness tests

MTF Compensation reduces the impact of the chart on measured imaging system MTF.

• Geometrically calibrate single or multi-camera devices obtaining camera intrinsic and extrinsic matrices for accurate machine vision and ADAS applications

A geometrically calibrated device provides a mapping between pixel coordinates and real-world position.

• Direct image acquisition for Sony sensors

Imatest now supports direct image acquisition from Sony sensor development boards through the Sony AYA software tool.

Learn More

It is important to test your camera system in environments which reproduce lighting conditions similar to where you intend to use the camera in the real world. Failure to test a camera under low light conditions may lead to overstating the camera’s performance.

Image sensors collect light (signal) into pixel wells, then convert the resulting analog voltage levels into digital numbers. Dark current is where electrons are released from thermal activity which becomes indistinguishable from electrons released via photoresponse. The dark current that exists in uncooled image sensors leads to dark noise. At low light levels, exposures are longer to gather more light, which gives more time for dark-current electrons to be gathered. This leads to dark noise and read noise representing a larger portion of the overall response, which reduces the signal to noise ratio (SNR). Low-light conditions are most challenging for higher resolution sensors with small pixel pitches where the particle and wave nature of light can seriously impact the performance of your camera.

The definition of “low light” depends on the application. For mobile devices (compact camera modules) low light is defined by the IEEE CPIQ standard as 25 lux, which resembles a dimly lit indoor space representative of the worst cases where people expect their phones to function properly. As mobile devices get better in low light scenarios, customer expectations may shift. For security or automotive industries, the levels are much lower based on how dark the outdoor environment may get^[1]:

In order to achieve light levels similar to the darkest scenes, most lab lighting setups that contain fluorescent or LED sources may not be able to be directly dimmed to very low levels, so the dim light levels may be achieved by a variety of methods shown in the following table. Ultra-low-light levels can be achieved by combining these approaches:

Eliminating additional sources of light within your lab is also important: Putting felt around doors to block light from other rooms, covering exit signs, covering indicator LEDs and power strips and installing baffling and light traps to block unwanted reflections.

Note: Verifying the low light levels can be challenging as many illuminance meters do not go to extreme low light levels.

Once you can reproduce low-light levels in your lab, you can perform tests to answer a number of questions about your system performance:

• Does the radiometric calibration for black level successfully reduce the effects of dark current without also eliminating useful signal that might impact the dynamic range of your system?
• Does the black level compensation work across the range of nominal operating temperatures that your sensor will experience?
• Does your ISP react appropriately to dark scenes by increasing the sensitivity (gain)?
• How consistently does your autofocus system (if you have one) perform under low light?
• What happens to moving objects captured under low light conditions?
• For security cameras with near-infrared ‘night-vision’ support: does the Infrared cutoff filter engage and disengage at the appropriate light level?
• Do the applied gamma curve(s) and tone mapping help to boost the perceived quality of the dark scene or assist the observer’s ability to recognize objects in the dark?
• Does the noise reduction strike the right balance of reducing visual noise without also blurring the appearance of relevant textures (skin and foliage) within the image?

A comprehensive tuning and testing regimen will involve performing a full battery of objective tests in a wide range of lighting conditions. By increasing the range of light levels you can reproduce in your lab, you can provide the most challenging condition in which you can validate the performance of your camera system.

Photo by Tuur Tisseghem from Pexels

Outline

Intro   |   Setup   |   Capture Procedure   |   Analysis   |   Results   |   Conclusion

The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio, These are typically produced at the maximum dynamic range of a printer which can be anywhere from 40:1 to 80:1.  The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values.

Some camera manufacturers who want better MTF results may take advantage of this anomaly to overstate the quality of the cameras they produce. This is why it is critical to validate cameras with a proper measurement system that includes a low-contrast target. (more…)