Increasing the Repeatability of Your Sharpness Tests

By Robert Sumner
With contributions from Ranga Burada, Henry Koren, Brienna Rogers and Norman Koren

Consistency is a fundamental aspect of successful image quality testing. Each component in your system may contribute to variation in test results. For tasks such as pass/fail testing, the primary goal is to identify the variation due to the component and ignore the variation due to noise. Being able to accurately replicate test results with variability limited to 1-5% will give you a more accurate description of how your product will perform. (more…)

Read More

Imatest does not start, unable to read MAT-file

If upon opening Imatest you get something like this:

 

Error: Error using load
Unable to read MAT-file C:\ProgramData\Imatest\mcr_cache\5.0\Master\mcrCache9.2\imates0\.matlab\matlabprefs.mat. File might be corrupt.

Some of your MATLAB files have become corrupted. You can solve this by deleting your CTF folder found at C:\ProgramData\Imatest\mcr_cache\5.0\Master\, then try starting Imatest again.

 

If you continue to have problems please email us with details of the issue at support@imatest.com

Read More

Undefined variable “py” or class “py.model.Message”.

With the addition of python routines in Imatest 5.0, several components require a working python installation. When some installation problems occur, the user may encounter the error message:

Undefined variable “py” or class “py.model.Message”.

This is generally caused by Imatest being unable to locate the correct python interpreter, or python failing to install properly.

Solution 1:

Make sure the python interpreter was installed properly and that you have permissions to access it. It should install to:
 
C:\Program Files\Imatest\v5.0\Master\bin\python35
 

Solution 2:

Some customers get this error when they try to run python:
 

To resolve, Reinstall the Visual Studio C++ 2013 redistributable, this is in the Imatest installer or you can install it directly from Microsoft: VS 2013
Read More

High-contrast edge-SFR test targets produce invalid MTF results

The obsolete ISO 12233:2000 standard defines a resolution test target with a high contrast ratio. These are typically produced at the maximum dynamic range of a printer, which can be anywhere from 40:1 to 80:1.  The high contrast can lead to clipping of the signal which leads to overstated invalid MTF values. (more…)

Read More

Running Imatest With High Sierra

Due to High Sierra’s new file system, old Imatest licenses no longer have the access they need to write properly.

If your Imatest license begins with the numbers 2848 and you are trying to run on OSX High Sierra or later please let us know with your license included at support@imatest.com

Read More

How to capture frames from a video stream or RTSP camera

Imatest is still working on implementing RTSP streaming support and other video stream protocols within the software. In the meantime if your streaming protocol has the capability of outputting MP4s or other common video file formats, you can load this video into Imatest to analyze. Alternatively, you can use a program like FFMPEG or VLC to split the video into its individual frames, then analyze the frame as an image in Imatest. Heres how.

 

Using FFMPEG

  1. Download a static FFMPEG build from a reputable source.
  2. Install FFMPEG according to the directions of your current OS:

Windows:

  1. Use a program like 7-Zip to unpack the files to your preferred location.
  2. Open up the command line with administrative privileges.
  3. Run the command:

setx /M PATH "path\to\ffmpeg\bin;%PATH%"

OSX:

a. Install homebrew by running the following in a terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

b. Once homebrew is installed, just enter in a terminal:


brew install ffmpeg

Linux:

a. Install ffmpeg


sudo add-apt-repository ppa:mc3man/trusty-media
sudo apt-get update
sudo apt-get install ffmpeg
sudo apt-get install frei0r-plugins

b. With FFMPEG installed you can now call it from a command line or bash terminal to split your video file into frames:

ffmpeg -i myfile.avi -f image2 image-%05d.png

Using VLC

coming soon

Read More

Megapixel suitability for test charts

Megapixel suitability is based on the analysis of the Modulation Transfer Function (MTF) that is obtained from slanted-edge chart images captured at magnifications of around 0.5× (for inkjet charts) or 1× (for photographic charts) using a high quality DSLR or mirrorless camera and macro lens.

The calculations are described in the following links:

Note: the old Chart Quality Index (CQI) calculation has been deprecated.

Megapixel suitability is based on the spatial frequencies where the projected chart MTF on sensor is 0.9 at the Nyquist frequency (0.5 cycles/pixels). We’ll omit the details of the calculations here. The beauty of the new approach is that megapixel suitability can be determined from just three items:

  1. File (the chart MTF compensation file) that has a model of the chart MTF (measured in cycles per object mm).
  2. Pixel height of the image sensor.
  3. Vertical Field of View of the camera, typically somewhat larger than the height of charts designed to fill the frame (SFRplus, eSFR ISO, etc.).

We use the Chart suitability display for Black and White LVT film (which is very much sharper than inkjet prints) as an example.

The standard Black & White LVT chart, printed on 12 inch × 20 inch film, is designed to have a 27cm vertical field of view; 27cm (on the x-axis) corresponds to MTF@Nyquist = 0.9 (the green diagonal line, below) for sensor height = 5500 pixels (left y-axis) or 45 megapixels at 3:2 aspect ratio (right y-axis). At this high quality level, MTF compensation is not required. If we push the chart to MTF@Nyquist = 0.7 (the olive diagonal line), which is still reasonably good, but requires MTF compensation, we reach sensor height = 8500 pixels (about 100 Megapixels at 3:2 aspect ratio).

Chart suitability display for B&W LVT film. Click image for full-size view.

The Megapixel suitability calculation shown here assumes that:

  1. The lens is of high quality.
  2. The sensor aspect ratio is 3:2, which can be changed in the box on the right. The change only affects the Megapixel numbers; i.e., Megapixel suitability, on the right y-axis.

For 16:9 aspect ratio sensors, multiply the megapixel suitability by 1.185.
For 4:3 aspect ratio sensors (with left/right sides of chart cropped), multiply megapixel suitability by 0.889.

Additions to Imatest 5.1

Imatest 5.1, released in April 2018, has an important enhancement that increases the megapixel suitability of most Imatest charts by up to a factor of 2. The MTF for most charts, which is a function of the chart media and printing technique, has been measured and fit to a simple two-parameter function that can be used to correct MTF measurements by deconvolution (by dividing the measured camera+chart MTF by the chart MTF function projected on the image sensor). The correction can be applied by entering an MTF correction file into the settings windows for any MTF calculations. For more details, see Compensating camera MTF measurements for chart and sensor MTF.

Chart Quality Calculator that uses the new MTF functions is also available. It provides a clearer and more accurate estimate of MTF suitability (including the expected MTF loss from the media with and without the correction) than the older Chart Quality Index.

Chart suitability displays for several media types are found in Test chart suitability for MTF measurements.

See Also

Read More

Lightbox Uniformity Comparison

Summary tableDetailed uniformity measurementsHow we made the measurements

Lightbox summary table. Many are no longer available from Imatest
Lightbox Brightness Uniformity* CRI (spec.)
Viewing Area Dimensions Controls
Imatest LED Lightbox 1 to 100,000 lux ** 90 to 95% Over 97

260 x 220 mm to 1440 x 1100 mm (9 Sizes)

400 (W) x 380 (H) x 200mm (D) to 1655 x 1296 x 200 mm (9 Sizes) WiFi, USB, Manual
Imatest LED Light Panel 30 to 1,000 lux 90 to 95%   229 x 152mm to 907 x 680mm (5 Sizes)
289 x 212 x 40mm to 967 x 740 x 55mm (5 Sizes)
WiFi, USB, Manual
IQL LED Lightbox 10 to 40,000 lux     254 x 279.4mm 472.4 x 383.5 x 129.5mm Wireless via Android
GL-16e Lightbox Viewer 5750 lux 63.6% 96-98 10 x 18″ (25 x 46 cm) 15x25x5″ (38x64x13cm) Manual
GL-20e Lightbox Viewer            
GL-30e Lightbox Viewer            
GL-44e Lightbox Viewer            
GLX-3044 Lightbox Viewer N/A N/A 96-98 30×42″ (76x107cm) 35x49x5″ (89x124x13cm) Manual
GLX-3856 LIghtbox Viewer N/A N/A 96-98 38×56″ (96x142cm) 43x63x5″ (109x160x13cm) Manual
GLE-10 Lightbox Viewer N/A TBC 96-98 8×10″ (20x25cm) 15.5×12.25×3.25″ (39x31x8cm) Manual
GLE GLX-30 Lightbox Viewer 5000 lux 70.1% 96-98 16″ x 30″ (41 x 76cm) 21 x 37.5 x 5″  Manual
Artograph LightPad® 930 2820 lux  77%   12×9″   On/Off
Artograph LightPad® 950 2740 lux 78.5%   24×17″   On/Off

*measured using 9 rectangular regions, as described below.
** measured using 30-10,000 lux model, ultra bright version has 90% uniformity

 

A better color quality measurement?  The color quality of light sources is traditionally measured by CRI (Color Rendering Index), which has a maximum value of 100 (%). Recently, doubt has been cast on the accuracy of CRI, and a new Color Rendition measurement has been proposed: IES TM-30-15. It’s unfamiliar and the linked document doesn’t have an equation or algorithm for calculating it from the light spectrum. We’ll wait and see…

Lightbox Uniformity- Detailed Measurements

For the key measurement, the definition of uniformity is

Uniformity = 100%*(1 – (maximum of 9 regions- minimum of 9 regions) / maximum of 9 regions)

where the 9 rectangular regions (shown in the figures below) include the top, bottom, left, right, 4 corners, and center. The rectangular region dimensions are 10% of the crop width and height and (except for the center region) they are located 5% of the width and height from the boundaries, as described below.

 

Click on any of the images below to view full-sized.

ITI LED Lightbox

Uniformity = 95.2%

Response is very even, but unusual in that the center is slightly dimmer than the top and bottom.

Note that the contour line increments are 0.01 (1%), lower than the other lightboxes because the ITI is much more uniform. (0.02 contour increments wouldn’t show very much.)

The lightbox spectra for the standard 3100K and 5100K settings, provided by ITI, are shown below.

ITI_uniformity_contours
ITILED-3100k ITILED-5100k

GTI GL-16e Lightbox

Uniformity = 63.6%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox. Both the GL-16e and GLX-30 have very wide aspect ratios. Their uniformity would be much better if less of the sides were included in the measurement.

 

GTI_GLX16e_uniformity_contours

   

Artograph 930 12×9 inch Light Pad

We use this inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 77%

Artograph_12x9_uniformity_contours

Artograph 950 24×17 inch Light Pad

We use this large, relatively inexpensive lightbox for non-critical applications like MTF measurements and for trade show demonstrations. It’s uniformity is quite good.

Uniformity = 78.5%

(Figure is darker because image was less exposed.) 

Arto_950_uniformity_contours

The control

The control for these measurements was made by capturing images immediately in front of the ITI lightbox (no more than 1cm distant). Results were repeatable when the camera was moved around the lightbox. Contour increments are very small: only 0.002 (0.5%).

Uniformity = 98.75%

EOS-6D_closeup_uniformity

GTI GLX-30 Lightbox

(No longer available in the Imatest Store)

Uniformity = 70.1%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX30_uniformity_contours

GTI GLE 12e Lightbox

(No longer available in the Imatest Store)

Uniformity = 65.8%

The contour line increments are 0.02 (2%), double that of the ITI LED Lightbox.

GTI_GLX12e_uniformity_contours

How we made the measurements

We developed a methodology for measuring lightbox uniformity because we were not aware of any relevant standards.

  • Photograph the lightbox using a mirrorless or DSLR camera with a long focal length marco lens. Such lenses tend to be highly uniform, i.e., have very low vignetting. We used the Canon EOS-6D with the highly-regarded 100mm f/2.8 macro lens set at f/8. Be sure to capture raw images. We used manual focus because the EOS-6D didn’t focus well on this image.
  • Frame the lightbox so it occupies about the central 30% of the image (10% by area). This makes the already low vignetting insignificant. Here is the framing (and region selection) for two lightboxes.
Click on the images to display them larger.
ITI_uniformity_cropITI Lightbox GTI_uniformity_cropGTI Lightbox
  • Open Uniformity (or Uniformity Interactive) and read in the raw file, converting it to a gamma = 1 (linear) file. (This means it’s not a standard color space file, but pixel level will be proportional to illumination.) Here are the recommended settings from the Imatest libraw/dcraw GUI window. The key settings are Output gamma = 1.0 (Linear), Auto white level checked, and Normalize by 1.0.

lightbox_dcraw_settings

  • Crop the images just inside the bright areas of the lightbox, as shown above. If there are areas of rapid illumination falloff close to the edges of the lightbox image, it’s OK to omit them.
  • Click Yes to open the Uniformity settings box. The key settings are shown inside the red rectangles. The corner and side regions (the rectangles) are 10% of the ROI (linearly), and the location of the regions is 5% (of the ROI size) from the ROI boundaries. We feel this is a reasonable summary metric since most tone and color measurements are made in the central two-thirds of the image. Uniformity is more important when measuring tone and color than it is for MTF, even though Imatest corrects for patch nonuniformity due to vignetting and uneven illumination.

lightbox_uniformity_settings

  • After you click OK (not shown) Uniformity runs and the results figures appear. The key nonuniformity summary metric does not appear in the figures it’s in the CSV and JSON file output. Here is the relevant CSV output.

    Uniformity = 100%*(1 – (maximum of 9 squares – minimum of 9 squares) / maximum of 9 squares)

Nonuniformity LRTB sides ctr (%) 36.44
Uniformity LRTB sides ctr (%) 63.56
  • and here is the JSON output:

         “nonuniformity_LRTB_sides_ctr_pct”: [36.4],
         “uniformity_LRTB_sides_ctr_pct”: [63.56],

 

Read More

Measuring the impact of flare light on dynamic range

Abstract

The dynamic range of recent HDR image sensors, defined as the range of exposure between saturation and 0 dB SNR, can be extremely high: 120 dB or more. But the dynamic range of real imaging systems is limited by veiling glare (flare light), arising from reflections inside the lens, and hence rarely approaches this level. (more…)

Read More

How to convert a color image to grayscale

Sometimes an undemosaiced Bayer raw image gets improperly saved into a BMP or TIFF format that includes color channels that are all equal. In this case, Imatest will recognize the image as color, and not give you the monochrome read options that allow you to perform demosaicing or analyze the image as a Bayer raw.

Here are a variety of ways to convert a color image to grayscale:

Imagemagick command line utility

$ convert -type Grayscale rawcolor.bmp mono.bmp

GIMP / Photoshop

Image -> Mode -> Grayscale

 

ImageJ

From Image -> Tyoe -> Select the desired output bit depth

Irfanview

Ctrl-G or from the image menu select Convert to Grayscale

MATLAB

img = imread('rawcolor.bmp');
r = img(:, :, 1);
imwrite(r, 'mono.bmp');

 

Read More

Avoiding reflections on transmissive charts for dynamic range and flare testing

The testing of dynamic range and flare requires a proper test chart and environmental setup to get accurate results.High precision and high-density chart technologies such as photographic film or chrome on glass come with the disadvantage of glossiness which makes them susceptible to specular reflections on the target. This affects the use of dynamic range or contrast resolution and backlit flare targets that lack anti-reflective patches. 

The goal of the backlit test setup is to maximize the fraction of light that is coming directly from your light source, through your test target, and into your lens, Any additional light that is present in your system can disrupt your measurements either by causing reflections on the analysis patch of your target, or otherwise introducing additional flare (veiling glare) into the image.

As the optimal lens and camera body housing will be manufactured with anti-reflective coating, it is good to take a similar approach with your test environment.  Nothing is actually entirely black, but dark, opaque materials come in many different forms. These can be characterized by their total hemispherical reflectance (THR).

Here are how some dark materials compare:

Product THR (vis+NIR) Notes
Black Paint 5-6% Not dark enough
Black Felt 2.5% Flexible and low-cost
Acktar black 1.5% Comes on adhesive foil
Vantablack 0.17% Costly, not durable, dangerous to humans

For general testing purposes, we use black felt for covering our darkened test box

Block all light from entering or reflecting inside the testing environment

For accurate testing, you should create a dark space where no outside light is able to enter, and as much internally generated light is absorbed. Here is a diagram of a test environment:

Minimize internal direct reflections

The more that the surfaces in your test environment are lit up, the more they will become reflections in the glossy chart.

The body of the camera under test or even just glass of a camera lens can reflect back onto a test chart.This is especially important incident towards dark analysis regions.

If using Imatest charts that are properly centered, the reflection might not fall on to a dark analysis region for dynamic range measurement. Here is an example of a reflection of a dev kit PCB that would disrupt dynamic range or contrast resolution measurement:

Reflective camera parts should be blocked by black masks. The entire area behind your camera will directly be reflected back, so this environment should be made as dark as possible.

Minimize other internal light emissions

Sensor development kits often have blinking LED’s that reflect directly, Front facing cameras with displays that are not disabled or blocked off can be particularly problematic.

Read More

DMX Lighting Control Software

Lights such as the Kino Flo Select 31 LED use a DMX control interface to signal changes of light level and color temperature.

DMX is an open protocol and there are a large number of possibilities for hardware and software control. I’ve tried out several of these are more oriented to stage control
 
For DMX controller hardware I have personally tried and will recommend the Enttec DMX USB Pro controller worked well for us.
For computer control software with a user Interface, the one that has worked best for me so far is the Q Light Controller Plus (QLC+) which has nice cross-platform support.
 
For light control software that provides an automated API, the best one I have found, and the one we use to automatically control lights in our lab is the Open Lighting Architecture (OLA). The downside of this is that it only has good support for Linux. I have attached a python script that includes an example OLA calls, along with calls to an isolight puck, which you can overlook.  Here is a basic call to set the lights to middle color temp and maximum intensity:
 
from ola.ClientWrapper import ClientWrapper
intensity = 255
cct       = 127
universe = 1
data      = array.array('B') 
data.append(intensity)
data.append(cct)
data.append(intensity)
data.append(cct)
wrapper = ClientWrapper()
client = wrapper.Client()
client.SendDmx(universe, data, DmxSent)
 
Our goal for Imatest 2020.1 is to have an instrument control interface built into Imatest Master that can directly communicate with DMX hardware.

 

Read More

Error in Extracting CTF file

Error Message: Error in extracting CTF file to ‘C:\ProgramData\Imatest\mcr_cache\4.5\Master\mcrCache9.0.1\imates0’. Details: ‘Could not set last modified time: C:/ProgramData/Imatest/mcr_cache/4.5/Master/mcrCache9.0.1/imates0/toolbox/imaq/imaqextern/drivers/win64/genicam/genICam_v2_4/redist/Microsoft_VC80_CRT_x86_x64.msi: Broken pipe

Solution:

Your system is having trouble extracting Mathwork’s CTF archive. Here is what the possible causes are:
 
  1. Permissions issue. For example if the application is unable to delete old directory or create a new one.
  2. Time stamp issue. The CTF-file is re-extracted only when the time stamp on the CTFarchive is newer than the time stamp on the _mcr directory. If somehow an out-of-date _mcr folder has a newer time stamp than the CTF-file, the new CTF-file will not be extracted and there will be errors when running the application.
  3. Extraction issue. If some error occurred during CTF extraction, there could be a problem where some of the encrypted M-files are of an old version and others are the new versions. This could cause the application to partially run and then only have an error when it runs into an old encrypted M-file.
  4. Installer issue. If the _mcr directory is already in the package, the installer may not be updating all files properly leaving some older files in the _mcr directory thinking that the files already exist.
So, here are the steps you should try to resolve this:
 
1. Delete old MCR directories on your machine. These can be found at C:\Program Files\MATLAB\MATLAB Runtime, delete the contents of this folder.
2. Run the Imatest installer again as administrator, ensuring that the “Matlab Libraries” checkbox is checked and that you are running the full installer rather than the upgrade installer:
 
Inline image 1
 
3. After the installer finishes, run Imatest as administrator.
 
If this process does not fix the issue please email us, support@imatest.com.
Read More

Imatest Stops Working – Advanced Troubleshooting

Recovering Mathworks Log Files

Please recover this log file and send to support@imatest.com

The log file name is called “mathworks_<username>.log”. Where <username> is the username of the user that ran the installer. Please see the instructions below to locate the log file depending on your OS.

Windows 10 \ Windows 8 \ Windows 7 \ Windows Vista

The installer log file will be located in the AppData\Local\Temp directory under the user account who ran the installer. The full path, as an example, is:

C:\Users\<username>\AppData\Local\Temp

A shortcut to get to this path is to enter the following into the address bar of File Explorer window, then press Enter:

%tmp%

Windows XP:

The directory is:

C:\Documents and Settings\<username>\Local Settings\Temp\mathworks_<username>.log

Linux:

The log file will be located at the following location:

/tmp/mathworks_<username>.log

Mac OS X:

Follow these steps to retrieve the installer log file on a Mac:

1. Log in as the user who ran the installer.

2. Open a Terminal window (you can use Spotlight Search to find Terminal)

3. Copy and paste the following command, and press enter to run it in Terminal. Do not edit the command, run it exactly how it appears below:

cp $TMPDIR/mathworks_$USER.log ~/Desktop

This will create a copy of the installer log file on the desktop. If the command fails, this means the installer log file is not being created. This could indicate that the installer is starting at all, and is thus not able to write the log file. If that is the case, additional information may be able to be obtained from the Console:

Applications – Utilities – Console (or use Spotlight Search to find the Console application)

Use the ‘Clear display’ button to clear out previous console messages. Then try running the installer again and see what is recorded in the console. You can save this information from console. This output may be helpful in addition to the installer log file.

All operating systems:

Forcing the creation of the log file in a specific location:

If the installer is not creating a log file, then you may need to create your own. You can run the installer in command prompt or terminal with a flag to force a log file to be created. You could use a command like the following to do this:

Windows:

/full/path/to/setup.exe -outputfile “C:\path\to\logfile.log”

or

Linux/Mac:

./install -outputfile /path/to/logfile.log

As a note, you can drag the installer into command prompt or terminal and command prompt will automatically populate the path of the installer for you. When defining where the log file will go, make sure it is a directory that you have write permissions. You may need to run command prompt as administrator (Windows) or the installer as sudo (Linux/Mac) in order to do this. 

Read More

Handling an Lightroom for iPhone RAW DNG file

Adobe DNG converter for windows & mac can be used to re-save iPhone RAW files into supported DNG formats using the following custom settings:

We soon hope to will be adding native support for LightroomiPhone DNG RAW.

 

Read More

Calculating Field of View (FoV) using distance settings

For accurate calculation of field of view (FoV) from using Imatest’s SFRplus, eSFR ISO, or Checkerboard, modules, it is critical to use proper distance measurements entered into in centimeters into the more settings window.

Lens to chart distance (in centimeters) can be best measured by a laser range finder.

For SFRplus it’s the bar to bar height is measured from the top of the top bar to the bottom of the bottom bar.

For eSFR ISO this is the vertical distance between the registration marks in centimeters.

Checkerboard uses the size of the squares.

Read More

Automating CPIQ analysis Using Imatest IT and Python

The Python interface to Imatest IT provides a simple means of invoking Imatest’s tests. This post will show how Imatest runs can be automated, then the results of those tests can be collected and easily processed. For this example, we will run five Imatest modules across three light levels, then extract CPIQ quality loss metrics.

Development Setup

Image Collection

For each device, we capture the following set of images of different test charts at different light levels:   (Note: full details will be available with the publication of the standard in March of 2017)

  • DeviceX_eSFR_5000KLED_1000lux.JPG
  • DeviceX_eSFR_TL84_100lux.JPG
  • DeviceX_eSFR_Tung_10lux.JPG
  • DeviceX_Dot_5000KLED_1000lux.JPG
  • DeviceX_Dot_TL84_100lux.JPG
  • DeviceX_Dot_Tung_10lux.JPG
  • DeviceX_SG_5000KLED_1000lux.JPG
  • DeviceX_SG_TL84_100lux.JPG
  • DeviceX_SG_Tung_10lux.JPG
  • DeviceX_Coins_5000KLED_1000lux.JPG
  • DeviceX_Coins_TL84_100lux.JPG
  • DeviceX_Coins_Tung_10lux.JPG
  • DeviceX_Unif_100lux.JPG

Running All The Tests

Once we have these images collected in a folder named “DeviceX”, we define our lighting conditions, initialize the Imatest IT library and run our processing function on this and a number of other devices:

led5k = "5000KLED_1000lux"               # Light Levels
tl84 = "TL84_100lux"
tung = "Tung_10lux"
light_conditions = [led5k, tl84, tung]
imatest = ImatestLibrary()
phones = ['DeviceX',                     # Just for example...
          'DeviceY',
          'DeviceZ']
for phone in phones:
    calc_phone(phone)

imatest.terminate_library()

The main processing script invokes the five modules, collects their data outputs and saves them to a JSON file:

def calc_phone(phone):
    print('Calculating Metrics for '+phone+ ' *****************************')
    uniformity_results = uniformity(phone) # Uniformity
    multitest_results  = multitest(phone) # Multitest
    dotpattern_results = dotpattern(phone) # Dot Pattern
    esfriso_results    = esfriso(phone) # eSFR ISO
    random_results     = random(phone) # Random

    # Output data to disk
    file_name = os.path.join(images_dir, phone, phone + '.json')
    with open(file_name, 'w') as outfile:
        full_data = {'multitest'  : multitest_results,
                     'dotpattern' : dotpattern_results,
                     'esfriso' : esfriso_results,
                     'random' : random_results,
                     'uniformity' : uniformity_results}
        json.dump(full_data, outfile, indent=4, separators=(',',': '))

    return full_data

We call the Imatest eSFR ISO module for each specified lighting condition:

# eSFR ISO (ISO 12233:2014) is used to test Visual Noise (VN) 
# and Spatial Frequency Response (SFR)
def esfriso(base):
    global images_dir, ini_file, light_conditions, op_mode
    output = {}
    for light_source in light_conditions:
        print('Testing ' + base + ' eSFR ISO at ' + light_source)
        input_file = os.path.join(images_dir, base, base+'_eSFR_'+light_source+'.jpg')
        if os.path.exists(input_file) != True:
            raise Exception('File ' + input_file + 'not found!')
        print('opening ' + input_file)
        result = imatest.esfriso_json(input_file=input_file, root_dir=root_dir, 
                                      op_mode=op_mode, ini_file=ini_file)
        data = json.loads(result)
        output[light_source] = data['esfrisoResults'];

return output

On the fly INI file changes

The most difficult part of this processing is to manually update region selections for the calls to Multitest, which does not yet have automatic region detection for the Colorchecker SG target (coming soon…).

In this case, we are we replacing the [sg] roi, nwid_save, and nht_save keys with the values stored in this multicharts-rois.json file:

{
"DeviceX_SG_5000KLED_1000lux.JPG": "468 417 2122 1578 468 1588 2102 397",
"DeviceX_SG_TL84_100lux.JPG":"482 376 2142 1571 482 1561 2157 376",
"DeviceX_SG_Tung_10lux.JPG":"386 336 2016 1492 401 1497 2021 321",
"width":"2592",
"height":"1936"
}

We use this script to modify INI files to dynamically insert different settings:

import ConfigParser

def get_config(path): # Used make ConfigParser case sensitive
    config = ConfigParser.ConfigParser()
    config.optionxform=str  # make configparser preserve case
    try:
        config.read(os.path.expanduser(path))
        return config
     except Exception, e:
        log.error(e)

def setROI(roi,width,height):
    global ini_file
    imatestini = get_config(ini_file)

    imatestini.set('ccsg', 'roi', roi)
    imatestini.set('ccsg', 'nwid_save', width)
    imatestini.set('ccsg', 'nht_save', height)
    with open(tempINI, 'w') as outfile:
        imatestini.write(outfile)
    return

These functions are then called using the values loaded from the JSON file inside of our multicharts function call:

    roi_overrides = {}
    roi_overrides_lower = {}
    roi_override_file = os.path.join(images_dir,base,'multicharts-rois.json')
    if os.path.exists(roi_override_file):
        with open(roi_override_file,'r') as fh: # read ROI override file
            roi_overrides = json.load(fh)
        for override, rois in roi_overrides.iteritems(): # convert to lowercase
            roi_overrides_lower[override.lower()] = rois
    else:
        raise Exception('ROI override file ' + roi_override_file + ' not found.')
 
    if image_file.lower() in roi_overrides_lower:
         setROI(roi_overrides_lower[image_file.lower()], roi_overrides['width'], roi_overrides['height'])
         selectedINI = tempINI
    else:
         raise Exception('Missing ROI override for ' + image_file)

Extracting Scores

Finally, we extract Key Performance Indicators (KPI), which are all in perceptual quality loss (QL) in units of just noticeable differences (JND).  This script is a bit long for this post, so view the source on Github if you are interested./

Here is the resulting set of calculated metrics in JSON format:

{ 'DeviceX': 
    { 'CL_QL': { '5000KLED_1000lux': 0.9142,
                 'TL84_100lux': 0.1803,
                 'Tung_10lux': 2.565},
      'CU_QL': 0.07498,
      'LCD_QL': 0,
      'LGD_QL': 0,
      'SFR_QL': { '5000KLED_1000lux': 0.0,
                  'TL84_100lux': 0.1969,
                  'Tung_10lux': 0.3946},
      'TB_QL': { '5000KLED_1000lux': 1.2,
                 'TL84_100lux': 6.25,
                 'Tung_10lux': 9.1},
      'VN_QL': { '5000KLED_1000lux': 1.113,
                 'TL84_100lux': 2.409,
                 'Tung_10lux': 8.531}},
      'Combined_QL': 11.0245
}

Once test procedures and combined score calculations are finalized by the IEEE Conformity Assessment Program, these outputs will be combined into a simple camera phone rating score.

The full python source code for this example is posted on GitHub.

To add these automation capabilities to your existing Imatest Master license, upgrade to Imatest Ultimate. You can also request a free 30-day trial of Imatest IT to see how this works for you before you purchase.

 

Related Webinars

Join us March 1 for Automating Lab and Manufacturing Processes: Defining image quality parameters on automated test equipment. Register now

Read More

The Effects of misregistration on the dead leaves cross-correlation texture blur analysis

This paper was given as part of the Electronic Imaging 2017 Image Quality and System Performance XIV and Digital Photography and Mobile Imaging XIII sessions.

When: Tuesday, January 31, 2017, at 12:10 pm

By: Robert Sumner with support from Ranga Burada, Noah Kram (more…)

Read More

Measuring MTF with wedges: pitfalls and best practices

This paper was given as part of the Electronic Imaging 2017 Autonomous Vehicles and Machine session.

When: Monday, January 30, 2017, at 10:10 am

By: Norman Koren with support from Henry Koren, Robert Sumner (more…)

Read More

Best practices for using transmissive test charts

 

Light Source

Side chart largeThe most uniform light source available would be an integrating sphere, but these come with considerable size and expense. After considering how the light source uniformity impacts their results, most lab & manufacturing tests use LED or fluorescent lightboxes with flat-panel diffusers. (more…)

Read More