Increasing the Repeatability of Your Sharpness Tests

By Robert Sumner
With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren

Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform.

Since Imatest makes measurements directly from the image pixels, any source that adds noise to the image can affect measurements. A primary source of noise in images is electronic sensor noise. Photon shot noise also contributes significantly in low-light situations. Other systemic sources of measurement variability, such as autofocus hysteresis, will not be addressed in this post.  

In order to reduce variation in your sharpness results and increase test repeatability, you should take steps to decrease the amount of noise in your image.

Here are 5 tips to limit noise in your test results: (more…)

Read More

Undefined variable “py” or class “py.model.Message”.

With the addition of python routines in Imatest 5.0, several components require a working python installation. When some installation problems occur, the user may encounter the error message:

Undefined variable “py” or class “py.model.Message”.

This is generally caused by Imatest being unable to locate the correct python interpreter, or python failing to install properly.

Solution 1:

Make sure the python interpreter was installed properly and that you have permissions to access it. It should install to:
 
C:\Program Files\Imatest\v5.0\Master\bin\python35
 

Solution 2:

Some customers get this error when they try to run python:
 
image.png
 
To resolve, Reinstall the Visual Studio C++ 2013 redistributable, this is in the Imatest installer or you can install it directly from Microsoft: VS 2013
Read More

High-contrast edge-SFR test targets produce invalid MTF results

The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio, These are typically produced at the maximum dynamic range of a printer which can be anywhere from 40:1 to 80:1.  The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values.

Some camera manufacturers who want better MTF results may take advantage of this anomaly to overstate the quality of the cameras they produce. This is why it is critical to validate cameras with a proper measurement system that includes a low-contrast target. (more…)

Read More

How to capture frames from a video stream or RTSP camera

Imatest is still working on implementing RTSP streaming support and other video stream protocols within the software. In the meantime if your streaming protocol has the capability of outputting MP4s or other common video file formats, you can load this video into Imatest to analyze. Alternatively, you can use a program like FFMPEG or VLC to split the video into its individual frames, then analyze the frame as an image in Imatest. Heres how.

 

Using FFMPEG

  1. Download a static FFMPEG build from a reputable source.
  2. Install FFMPEG according to the directions of your current OS:

Windows:

  1. Use a program like 7-Zip to unpack the files to your preferred location.
  2. Open up the command line with administrative privileges.
  3. Run the command:

setx /M PATH "path\to\ffmpeg\bin;%PATH%"

OSX:

a. Install homebrew by running the following in a terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

b. Once homebrew is installed, just enter in a terminal:


brew install ffmpeg

Linux:

a. Install ffmpeg


sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install ffmpeg
sudo apt-get install frei0r-plugins

b. With FFMPEG installed you can now call it from a command line or bash terminal to split your video file into frames:

ffmpeg -i myfile.avi -f image2 image-%05d.png

Using VLC

coming soon

Read More

Megapixel suitability for test charts

Megapixel suitability is based on analysis of Modulation Transfer Function (MTF) obtained from a chart image captured using a 1:1 magnification lens (Canon 65mm f/2.8 1-5x macro) and a 6.5µm pixel size sensor (Canon EOS 6D). Conversion to megapixels is based on the Imatest Chart Quality Index (CQI) calculation which determines sensor height suitability using the equation:

2 * MTFxx (cycles per object mm) * vertical chart height (mm) where xx is 90, 70 or 50.

In order to consider performance throughout the camera’s range of expected sharpness, we photograph slanted-edges of transmissive and reflective substrates and take the sum of:

  • MTF90 (the spatial frequency where MTF is 90%) multiplied by 0.5,
  • MTF70 multiplied by 0.35, and
  • MTF50 multiplied by 0.15

Megapixel suitability calculations assume that:

  1. the lens is of high quality
  2. that the chart fills the vertical field of view (vFoV) of the camera system
  3. that the sensor aspect ratio is 3:2.

For 16:9 aspect ratio sensors (with pillarboxed framing, if applicable), multiply the megapixel suitability by 1.185.
For 4:3 aspect ratio sensors (with left/right sides of chart cropped), multiply megapixel suitability by 0.889.

Charts can be suitable for significantly higher megapixel counts if the minimum resolvable feature size of the lens is larger than the pixel size or the chart fills less than the full sensor vertical field of view.

Additions to Imatest 5.1

Imatest 5.1, to be released in April 2018, has some important enhancements that increase the megapixel suitability of most Imatest charts by up to a factor of 2. The MTF for most charts, which is a function of the chart media and printing technique, has been measured, and the measurements have been fit to a simple two-parameter function which can be used to correct MTF measurements by deconvolution (by dividing the measured camera MTF by the chart MTF function projected on the image sensor). The correction can be applied by entering an MTF correction file into the settings windows for any MTF calculations. For more details, see Compensating camera MTF measurements for chart and sensor MTF.

Chart Quality Calculator that uses the new MTF functions is also available. It provides a clearer and more accurate estimate of MTF suitability (including the expected MTF loss from the media without and with the correction) than the older Chart Quality Index.

See Also

Read More

Lightbox Uniformity Comparison

 

Lightbox Brightness Uniformity* CRI (spec.)
Viewing Area Dimensions Controls
ITI LED Lightbox 1 – 100,000 Lux ** 95.2% Over 97 260 mm x 220 mm 400 (W) x 380 (H) x 100mm (D) Bluetooth, USB
IQL LED Lightbox 10 – 40,000 Lux     254mm x 279.4mm 472.4 x 383.5 x 129.5mm Wireless via Android
GL-16e Lightbox Viewer 5750 Lux 63.6% 96-98 10 x 18″ (25 x 46 cm) 15x25x5″ (38x64x13cm) Manual
GL-20e Lightbox Viewer            
GL-30e Lightbox Viewer            
GL-44e Lightbox Viewer            
GLX-3044 Lightbox Viewer N/A N/A 96-98 30×42″ (76x107cm) 35x49x5″ (89x124x13cm) Manual
GLX-3856 LIghtbox Viewer N/A N/A 96-98 38×56″ (96x142cm) 43x63x5″ (109x160x13cm) Manual
GLE-10 Lightbox Viewer N/A TBC 96-98 8×10″ (20x25cm) 15.5×12.25×3.25″ (39x31x8cm) Manual
GLE GLX-30 Lightbox Viewer 5000 Lux 70.1% 96-98 16″ x 30″ (41 x 76cm) 21 x 37.5 x 5″  Manual
Artograph LightPad® 930 2820 Lux  77%   12×9″   On/Off
Artograph LightPad® 950 2740 Lux 78.5%   24×17″   On/Off

*measured using 9 rectangular regions, as described below.
** measured using 30-10,000 lux model, ultra bright version has 90% uniformity

 

A better color quality measurement?  The color quality of light sources is traditionally measured by CRI (Color Rendering Index), which has a maximum value of 100 (%). Recently, doubt has been cast on the accuracy of CRI, and a new Color Rendition measurement has been proposed: IES TM-30-15. It’s unfamiliar and the linked document doesn’t have an equation or algorithm for calculating it from the light spectrum. We’ll wait and see…

Lightbox Uniformity- Detailed Measurements

For the key measurement, the definition of uniformity is

Uniformity = 100%*(1 – (maximum of 9 regions- minimum of 9 regions) / maximum of 9 regions)

where the 9 rectangular regions (shown in the figures below) include the top, bottom, left, right, 4 corners, and center. The rectangular region dimensions are 10% of the crop width and height and (except for the center region) they are located 5% of the width and height from the boundaries, as described below.

 

Click on any of the images below to view full-sized.

ITI LED Lightbox

Uniformity = 95.2%

Response is very even, but unusual in that the center is slightly dimmer than the top and bottom.

Note that the contour line increments are 0.01 (1%), lower than the other lightboxes because the ITI is much more uniform. (0.02 contour increments wouldn’t show very much.)

The lightbox spectra for the standard 3100K and 5100K settings, provided by ITI, are shown below.

ITI_uniformity_contours
ITILED-3100k ITILED-5100k

GTI GL-16e Lightbox

Uniformity = 63.6%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox. Both the GL-16e and GLX-30 have very wide aspect ratios. Their uniformity would be much better if less of the sides were included in the measurement.

 

GTI_GLX16e_uniformity_contours

   

Artograph 930 12×9 inch Light Pad

We use this inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 77%

Artograph_12x9_uniformity_contours

Artograph 950 24×17 inch Light Pad

We use this large, relatively inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 78.5%

(Figure is darker because image was less exposed.) 

Arto_950_uniformity_contours

The control

The control for these measurements was made by capturing images immediately in front of the ITI lightbox (no more than 1cm distant). Results were repeatable when the camera was moved around the lightbox. Contour increments are very small: only 0.002 (0.5%).

Uniformity = 98.75%

EOS-6D_closeup_uniformity

GTI GLX-30 Lightbox

(No longer available in the Imatest Store)

Uniformity = 70.1%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX30_uniformity_contours

GTI GLE 12e Lightbox

(No longer available in the Imatest Store)

Uniformity = 65.8%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX12e_uniformity_contours

How we made the measurements

We developed a methodology for measuring lightbox uniformity because we were not aware of any relevant standards.

  • Photograph the lightbox using a camera with a long focal length marco lens. Such lenses tend to be highly uniform, i.e., have very low vignetting. We used the Canon EOS-6D with the highly-regarded 100mm f/2.8 macro lens set at f/8. Be sure to capture raw images. We used manual focus because the EOS-6D didn’t focus well on this image.
  • Frame the lightbox so it occupies about the central 30% of the image (10% by area). This makes the already low vignetting insignificant. Here is the framing (and region selection) for two lightboxes.
Click on the images to display them larger.
ITI_uniformity_cropITI Lightbox GTI_uniformity_cropGTI Lightbox
  • Open Uniformity (or Uniformity Interactive) and read in the raw file, converting it to a gamma = 1 (linear) file. (This means it’s not a standard file, but pixel level will be proportional to illumination.) Here are the recommended settings from the Imatest dcraw GUI window. The key settings are Output gamma = 1.0 (Linear), Auto white level checked, and Normalize by 1.0.

lightbox_dcraw_settings

  • Crop the images just inside the bright areas of the lightbox, as shown above. If there are areas of rapid illumination falloff close to the edges of the lightbox image, it’s OK to omit them.
  • Click Yes to open the Uniformity settings box. The key settings are shown inside the red rectangles. The corner and side regions (the rectangles) are 10% of the ROI (linearly), and the location of the regions is 5% (of the ROI size) from the ROI boundaries. We feel this is a reasonable summary metric since most tone and color measurements are made in the central two-thirds of the image. Uniformity is more important when measuring tone and color than it is for MTF, even though Imatest corrects for patch nonuniformity due to vignetting and uneven illumination.

lightbox_uniformity_settings

  • After you click OK (not shown) Uniformity runs and the results figures appear. The key nonuniformity summary metric does not appear in the figures it’s in the CSV and JSON file output. Here is the relevant CSV output.

    Uniformity = 100%*(1 – (maximum of 9 squares – minimum of 9 squares) / maximum of 9 squares)

Nonuniformity LRTB sides ctr (%) 36.44
Uniformity LRTB sides ctr (%) 63.56
  • and here is the JSON output:

         “nonuniformity_LRTB_sides_ctr_pct”: [36.4],
         “uniformity_LRTB_sides_ctr_pct”: [63.56],

 

Read More

Measuring the impact of flare light on dynamic range

Abstract

The dynamic range of recent HDR image sensors, defined as the range of exposure between saturation and 0 dB SNR, can be extremely high: 120 dB or more. But the dynamic range of real imaging systems is limited by veiling glare (flare light), arising from reflections inside the lens, and hence rarely approaches this level. Veiling glare measurements, such as ISO 18844, made with black cavities on white fields, result in large numbers that are difficult to relate to dynamic range. Camera dynamic range is typically measured from grayscale charts, where veiling glare depends on the design and layout of the chart, leading to inconsistent results. (more…)

Read More

How to convert a color image to grayscale

Sometimes an undemosaiced Bayer raw image gets improperly saved into a BMP or TIFF format that includes color channels that are all equal. In this case, Imatest will recognize the image as color, and not give you the monochrome read options that allow you to perform demosaicing or analyze the image as a Bayer raw.

Here are a variety of ways to convert a color image to grayscale:

Imagemagick command line utility

$ convert -type Grayscale rawcolor.bmp mono.bmp

GIMP / Photoshop

Image -> Mode -> Grayscale

 

ImageJ

From Image -> Tyoe -> Select the desired output bit depth

Irfanview

Ctrl-G or from the image menu select Convert to Grayscale

MATLAB

img = imread('rawcolor.bmp');
r = img(:, :, 1);
imwrite(r, 'mono.bmp');

 

Read More

Avoiding reflections on transmissive charts for dynamic range and flare testing

The testing of dynamic range and flare requires a proper test chart and environmental setup to get accurate results.High precision and high-density chart technologies such as photographic film or chrome on glass come with the disadvantage of glossiness which makes them susceptible to specular reflections on the target. This affects the use of dynamic range or contrast resolution and backlit flare targets that lack anti-reflective patches. 

The goal of the backlit test setup is to maximize the fraction of light that is coming directly from your light source, through your test target, and into your lens, Any additional light that is present in your system can disrupt your measurements either by causing reflections on the analysis patch of your target, or otherwise introducing additional flare (veiling glare) into the image.

As the optimal lens and camera body housing will be manufactured with anti-reflective coating, it is good to take a similar approach with your test environment.  Nothing is actually entirely black, but dark, opaque materials come in many different forms. These can be characterized by their total hemispherical reflectance (THR).

Here are how some dark materials compare:

Product THR (vis+NIR) Notes
Black Paint 5-6% Not dark enough
Black Felt 2.5% Flexible and low-cost
Acktar black 1.5% Comes on adhesive foil
Vantablack 0.17% Costly, not durable, dangerous to humans

For general testing purposes, we use black felt for covering our darkened test box

Block all light from entering or reflecting inside the testing environment

For accurate testing, you should create a dark space where no outside light is able to enter, and as much internally generated light is absorbed. Here is a diagram of a test environment:

Minimize internal direct reflections

The more that the surfaces in your test environment are lit up, the more they will become reflections in the glossy chart.

The body of the camera under test or even just glass of a camera lens can reflect back onto a test chart.This is especially important incident towards dark analysis regions.

If using Imatest charts that are properly centered, the reflection might not fall on to a dark analysis region for dynamic range measurement. Here is an example of a reflection of a dev kit PCB that would disrupt dynamic range or contrast resolution measurement:

Reflective camera parts should be blocked by black masks. The entire area behind your camera will directly be reflected back, so this environment should be made as dark as possible.

Minimize other internal light emissions

Sensor development kits often have blinking LED’s that reflect directly, Front facing cameras with displays that are not disabled or blocked off can be particularly problematic.

Read More

DMX Lighting Control Software

Lights such as the Kino Flo Select 31 LED use a DMX control interface to signal changes of light level and color temperature.

DMX is an open protocol and there are a large number of possibilities for hardware and software control. I’ve tried out several of these are more oriented to stage control
 
For DMX controller hardware I have personally tried and will recommend the Enttec Open DMX USB controller worked well for us, the DMX USB Pro seems nice also.
For computer control software with a user Interface, the one that has worked best for me so far is the Q Light Controller Plus (QLC+) which has nice cross-platform support.
 
For light control software that provides an automated API, the best one I have found, and the one we use to automatically control lights in our lab is the Open Lighting Architecture (OLA). The downside of this is that it only has good support for Linux. I have attached a python script that includes an example OLA calls, along with calls to an isolight puck, which you can overlook.  Here is a basic call to set the lights to middle color temp and maximum intensity:
 
from ola.ClientWrapper import ClientWrapper
intensity = 255
cct       = 127
universe = 1
data      = array.array('B') 
data.append(intensity)
data.append(cct)
data.append(intensity)
data.append(cct)
wrapper = ClientWrapper()
client = wrapper.Client()
client.SendDmx(universe, data, DmxSent)
 
Our goal for Imatest 5.2 is to have an instrument control interface built into Imatest Master that can directly communicate with DMX hardware.

 

Read More