Reducing the cross-lab variation of image quality metrics

Abstract

As imaging test labs seek to obtain objective performance scores of camera systems, many factors can skew the results. IEEE Camera Phone Image Quality (CPIQ) Conformity Assessment Steering Committee (CASC) working group members performed round-robin studies where an assortment of mobile devices was tested within heterogeneous imaging labs. This paper investigates how the existence of near-infrared energy in light sources that attempt to simulate CIE illuminants can influence test results. Numerous other impacts, including the influence opal diffusers used for uniformity testing, how test scene framing can alter white balance and exposure, and how chart quality and texture frequency distribution can skew results. We introduce a test procedure which is intended to reduce intra-lab variability and a method for assessing an independent lab’s competence in conforming with the IEEE testing standards.

Talk

Standards Documents

Read More

New 2019 Imatest Licensing – Update Required to Activate

We are migrating to a better, faster licensing platform for our software on January 1, 2019. You may need to update your software to activate it on a new device, depending on your software version. Updating your software is free of charge.

Who is affected by this?

If your license code starts with 2848, you will need to update your software after January 1, 2019 if you need to install it on other devices or reactivate it on a current device. Your software will continue to work as normal until you need to reinstall or reactivate.

What actions do you need to take?

You will need to update your software in order install it on other devices or reactivate it on your current device. Check below for your software build download link. If you’re not sure which version you need, contact us.

What happens if you don’t update?

You can continue to use your software as it is without updating it. However, if you attempt to reactivate it or install your software on another device after January 1, 2019, you won’t be able to do so. You’ll need to update the software to continue using it.

What are the benefits of this?

  • Our licenses will now run properly on OSX versions High Sierra and up.
  • Our licenses will have less issue activating on machines with heavy security and those that run in offline environments.

What version do you have?

As a reminder, this applies to licensing codes starting in 2848 only.

On opening Imatest, the version you are running will be displayed at the top of the main screen, in the command window, or in the Help > About window.   imatest software not upgrading

Download your software update today.

Select your software version from the drop-down menu below to access your download link.

Read More

Increasing the Repeatability of Your Sharpness Tests

By Robert Sumner
With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren

Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform.

Since Imatest makes measurements directly from the image pixels, any source that adds noise to the image can affect measurements. A primary source of noise in images is electronic sensor noise. Photon shot noise also contributes significantly in low-light situations. Other systemic sources of measurement variability, such as autofocus hysteresis, will not be addressed in this post.  

In order to reduce variation in your sharpness results and increase test repeatability, you should take steps to decrease the amount of noise in your image.

Here are 5 tips to limit noise in your test results: (more…)

Read More

Imatest does not start, unable to read MAT-file

If upon opening Imatest you get something like this:

 

Error: Error using load
Unable to read MAT-file C:\ProgramData\Imatest\mcr_cache\5.0\Master\mcrCache9.2\imates0\.matlab\matlabprefs.mat. File might be corrupt.

Some of your MATLAB files have become corrupted. You can solve this by deleting your CTF folder found at C:\ProgramData\Imatest\mcr_cache\5.0\Master\, then try starting Imatest again.

 

If you continue to have problems please email us with details of the issue at support@imatest.com

Read More

Undefined variable “py” or class “py.model.Message”.

With the addition of python routines in Imatest 5.0, several components require a working python installation. When some installation problems occur, the user may encounter the error message:

Undefined variable “py” or class “py.model.Message”.

This is generally caused by Imatest being unable to locate the correct python interpreter, or python failing to install properly.

Solution 1:

Make sure the python interpreter was installed properly and that you have permissions to access it. It should install to:
 
C:\Program Files\Imatest\v5.0\Master\bin\python35
 

Solution 2:

Some customers get this error when they try to run python:
 
image.png
 
To resolve, Reinstall the Visual Studio C++ 2013 redistributable, this is in the Imatest installer or you can install it directly from Microsoft: VS 2013
Read More

High-contrast edge-SFR test targets produce invalid MTF results

The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio, These are typically produced at the maximum dynamic range of a printer which can be anywhere from 40:1 to 80:1.  The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values.

Some camera manufacturers who want better MTF results may take advantage of this anomaly to overstate the quality of the cameras they produce. This is why it is critical to validate cameras with a proper measurement system that includes a low-contrast target. (more…)

Read More

Running Imatest With High Sierra

Due to High Sierra’s new file system, old Imatest licenses no longer have the access they need to write properly.

If your Imatest license begins with the numbers 2848 and you are trying to run on OSX High Sierra or later please let us know with your license included at support@imatest.com

Read More

How to capture frames from a video stream or RTSP camera

Imatest is still working on implementing RTSP streaming support and other video stream protocols within the software. In the meantime if your streaming protocol has the capability of outputting MP4s or other common video file formats, you can load this video into Imatest to analyze. Alternatively, you can use a program like FFMPEG or VLC to split the video into its individual frames, then analyze the frame as an image in Imatest. Heres how.

 

Using FFMPEG

  1. Download a static FFMPEG build from a reputable source.
  2. Install FFMPEG according to the directions of your current OS:

Windows:

  1. Use a program like 7-Zip to unpack the files to your preferred location.
  2. Open up the command line with administrative privileges.
  3. Run the command:

setx /M PATH "path\to\ffmpeg\bin;%PATH%"

OSX:

a. Install homebrew by running the following in a terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

b. Once homebrew is installed, just enter in a terminal:


brew install ffmpeg

Linux:

a. Install ffmpeg


sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install ffmpeg
sudo apt-get install frei0r-plugins

b. With FFMPEG installed you can now call it from a command line or bash terminal to split your video file into frames:

ffmpeg -i myfile.avi -f image2 image-%05d.png

Using VLC

coming soon

Read More

Megapixel suitability for test charts

Megapixel suitability is based on analysis of Modulation Transfer Function (MTF) obtained from a chart image captured using a 1:1 magnification lens (Canon 65mm f/2.8 1-5x macro) and a 6.5µm pixel size sensor (Canon EOS 6D). Conversion to megapixels is based on the Imatest Chart Quality Index (CQI) calculation which determines sensor height suitability using the equation:

2 * MTFxx (cycles per object mm) * vertical chart height (mm) where xx is 90, 70 or 50.

In order to consider performance throughout the camera’s range of expected sharpness, we photograph slanted-edges of transmissive and reflective substrates and take the sum of:

  • MTF90 (the spatial frequency where MTF is 90%) multiplied by 0.5,
  • MTF70 multiplied by 0.35, and
  • MTF50 multiplied by 0.15

Megapixel suitability calculations assume that:

  1. the lens is of high quality
  2. that the chart fills the vertical field of view (vFoV) of the camera system
  3. that the sensor aspect ratio is 3:2.

For 16:9 aspect ratio sensors (with pillarboxed framing, if applicable), multiply the megapixel suitability by 1.185.
For 4:3 aspect ratio sensors (with left/right sides of chart cropped), multiply megapixel suitability by 0.889.

Charts can be suitable for significantly higher megapixel counts if the minimum resolvable feature size of the lens is larger than the pixel size or the chart fills less than the full sensor vertical field of view.

Additions to Imatest 5.1

Imatest 5.1, to be released in April 2018, has some important enhancements that increase the megapixel suitability of most Imatest charts by up to a factor of 2. The MTF for most charts, which is a function of the chart media and printing technique, has been measured, and the measurements have been fit to a simple two-parameter function which can be used to correct MTF measurements by deconvolution (by dividing the measured camera MTF by the chart MTF function projected on the image sensor). The correction can be applied by entering an MTF correction file into the settings windows for any MTF calculations. For more details, see Compensating camera MTF measurements for chart and sensor MTF.

Chart Quality Calculator that uses the new MTF functions is also available. It provides a clearer and more accurate estimate of MTF suitability (including the expected MTF loss from the media without and with the correction) than the older Chart Quality Index.

See Also

Read More

Lightbox Uniformity Comparison

 

Lightbox Brightness Uniformity* CRI (spec.)
Viewing Area Dimensions Controls
Imatest LED Lightbox 1 to 100,000 lux ** 90 to 95% Over 97

260 x 220 mm to 1440 x 1100 mm (9 Sizes)

400 (W) x 380 (H) x 200mm (D) to 1655 x 1296 x 200 mm (9 Sizes) WiFi, USB, Manual
Imatest LED Light Panel 30 to 1,000 lux 90 to 95%   229 x 152mm to 907 x 680mm (5 Sizes)
289 x 212 x 40mm to 967 x 740 x 55mm (5 Sizes)
WiFi, USB, Manual
IQL LED Lightbox 10 to 40,000 lux     254 x 279.4mm 472.4 x 383.5 x 129.5mm Wireless via Android
GL-16e Lightbox Viewer 5750 lux 63.6% 96-98 10 x 18″ (25 x 46 cm) 15x25x5″ (38x64x13cm) Manual
GL-20e Lightbox Viewer            
GL-30e Lightbox Viewer            
GL-44e Lightbox Viewer            
GLX-3044 Lightbox Viewer N/A N/A 96-98 30×42″ (76x107cm) 35x49x5″ (89x124x13cm) Manual
GLX-3856 LIghtbox Viewer N/A N/A 96-98 38×56″ (96x142cm) 43x63x5″ (109x160x13cm) Manual
GLE-10 Lightbox Viewer N/A TBC 96-98 8×10″ (20x25cm) 15.5×12.25×3.25″ (39x31x8cm) Manual
GLE GLX-30 Lightbox Viewer 5000 lux 70.1% 96-98 16″ x 30″ (41 x 76cm) 21 x 37.5 x 5″  Manual
Artograph LightPad® 930 2820 lux  77%   12×9″   On/Off
Artograph LightPad® 950 2740 lux 78.5%   24×17″   On/Off

*measured using 9 rectangular regions, as described below.
** measured using 30-10,000 lux model, ultra bright version has 90% uniformity

 

A better color quality measurement?  The color quality of light sources is traditionally measured by CRI (Color Rendering Index), which has a maximum value of 100 (%). Recently, doubt has been cast on the accuracy of CRI, and a new Color Rendition measurement has been proposed: IES TM-30-15. It’s unfamiliar and the linked document doesn’t have an equation or algorithm for calculating it from the light spectrum. We’ll wait and see…

Lightbox Uniformity- Detailed Measurements

For the key measurement, the definition of uniformity is

Uniformity = 100%*(1 – (maximum of 9 regions- minimum of 9 regions) / maximum of 9 regions)

where the 9 rectangular regions (shown in the figures below) include the top, bottom, left, right, 4 corners, and center. The rectangular region dimensions are 10% of the crop width and height and (except for the center region) they are located 5% of the width and height from the boundaries, as described below.

 

Click on any of the images below to view full-sized.

ITI LED Lightbox

Uniformity = 95.2%

Response is very even, but unusual in that the center is slightly dimmer than the top and bottom.

Note that the contour line increments are 0.01 (1%), lower than the other lightboxes because the ITI is much more uniform. (0.02 contour increments wouldn’t show very much.)

The lightbox spectra for the standard 3100K and 5100K settings, provided by ITI, are shown below.

ITI_uniformity_contours
ITILED-3100k ITILED-5100k

GTI GL-16e Lightbox

Uniformity = 63.6%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox. Both the GL-16e and GLX-30 have very wide aspect ratios. Their uniformity would be much better if less of the sides were included in the measurement.

 

GTI_GLX16e_uniformity_contours

   

Artograph 930 12×9 inch Light Pad

We use this inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 77%

Artograph_12x9_uniformity_contours

Artograph 950 24×17 inch Light Pad

We use this large, relatively inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 78.5%

(Figure is darker because image was less exposed.) 

Arto_950_uniformity_contours

The control

The control for these measurements was made by capturing images immediately in front of the ITI lightbox (no more than 1cm distant). Results were repeatable when the camera was moved around the lightbox. Contour increments are very small: only 0.002 (0.5%).

Uniformity = 98.75%

EOS-6D_closeup_uniformity

GTI GLX-30 Lightbox

(No longer available in the Imatest Store)

Uniformity = 70.1%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX30_uniformity_contours

GTI GLE 12e Lightbox

(No longer available in the Imatest Store)

Uniformity = 65.8%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX12e_uniformity_contours

How we made the measurements

We developed a methodology for measuring lightbox uniformity because we were not aware of any relevant standards.

  • Photograph the lightbox using a camera with a long focal length marco lens. Such lenses tend to be highly uniform, i.e., have very low vignetting. We used the Canon EOS-6D with the highly-regarded 100mm f/2.8 macro lens set at f/8. Be sure to capture raw images. We used manual focus because the EOS-6D didn’t focus well on this image.
  • Frame the lightbox so it occupies about the central 30% of the image (10% by area). This makes the already low vignetting insignificant. Here is the framing (and region selection) for two lightboxes.
Click on the images to display them larger.
ITI_uniformity_cropITI Lightbox GTI_uniformity_cropGTI Lightbox
  • Open Uniformity (or Uniformity Interactive) and read in the raw file, converting it to a gamma = 1 (linear) file. (This means it’s not a standard file, but pixel level will be proportional to illumination.) Here are the recommended settings from the Imatest dcraw GUI window. The key settings are Output gamma = 1.0 (Linear), Auto white level checked, and Normalize by 1.0.

lightbox_dcraw_settings

  • Crop the images just inside the bright areas of the lightbox, as shown above. If there are areas of rapid illumination falloff close to the edges of the lightbox image, it’s OK to omit them.
  • Click Yes to open the Uniformity settings box. The key settings are shown inside the red rectangles. The corner and side regions (the rectangles) are 10% of the ROI (linearly), and the location of the regions is 5% (of the ROI size) from the ROI boundaries. We feel this is a reasonable summary metric since most tone and color measurements are made in the central two-thirds of the image. Uniformity is more important when measuring tone and color than it is for MTF, even though Imatest corrects for patch nonuniformity due to vignetting and uneven illumination.

lightbox_uniformity_settings

  • After you click OK (not shown) Uniformity runs and the results figures appear. The key nonuniformity summary metric does not appear in the figures it’s in the CSV and JSON file output. Here is the relevant CSV output.

    Uniformity = 100%*(1 – (maximum of 9 squares – minimum of 9 squares) / maximum of 9 squares)

Nonuniformity LRTB sides ctr (%) 36.44
Uniformity LRTB sides ctr (%) 63.56
  • and here is the JSON output:

         “nonuniformity_LRTB_sides_ctr_pct”: [36.4],
         “uniformity_LRTB_sides_ctr_pct”: [63.56],

 

Read More