Correlating the Performance of Computer Vision Algorithms with Objective Image Quality Metrics

by Henry Koren

1. Why care about the quality of your cameras?

The task of computer vision (CV) involves analyzing a stream of images from an imaging device. Some simple applications such as object counting may be less dependent on good camera quality. But for more advanced CV applications where there is limited control of lighting & distance, the quality of your overall vision system will depend on the quality of your camera system. This is increasingly important when an error made by the vision system could lead to a decision that impacts safety. Along with proper optimization of a CV model, ensuring that that model is fed by imagery from a high-quality camera system is critical to maximizing your system’s performance.

Read More

Imatest not vulnerable to Apache log4j security compromise

Security researchers disclosed the following vulnerabilities in the Apache Log4j Java logging library:

  • CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker-controlled LDAP
    and other JNDI related endpoints
  • CVE-2021-45046: The fix for CVE-2021-44228 was incomplete in certain non-default
  • CVE-2021-45105: Apache Log4j2 Context Lookup features do not protect against uncontrolled
    recursion from self-referential lookups in certain non-default configurations

No Imatest software includes the affected versions of Log4j, no dependency used, such as the MATLAB compiler runtime includes an affected version either.

Internal Imatest systems which included Log4j were promptly patched when the vulnerability was discovered. These were never publically accessable.

Thank you for your concern.

Read More

Test Chart Life Span

 Best storage practices would be in a clean, dark, dry area at (~ 23°C / 72°F) or below.  Exposure to UV will break down pigments and cause loss of contrast and saturation.

Color Test Charts

Such as X-Rite colorcheckers.  Certified to last two years if stored properly.  You can use a spectrophotometer to generate a reference file for an older/faded chart.

Color LVT Film

The main factors affecting the films are humidity and UV light.

If the chart is used heavily (exposed to light every day) it can last up to one year.

If they are stored dark place and not used very often they can last up to 5 years.

Significant heat exposure ( > 52°C / 125°F) will decouple the dyes and significantly hasten the fading time.


Read More

How to compute resolution in TV Lines (TVL)

Television Lines (TVL) are derived from the EIA 1956 testing standard, where they are defined as the number of light and dark vertical lines that are visible (i.e., distinguishable) over the height of the screen. TVL defines “lines” as distinct dark and light lines. This is equivalent to Line Widths (LW), and should not to be confused with Line Pairs (LP), which represent a complete cycle, i.e., one TV line is equivalent to half a line pair. Although TVL is not the best measurement of digital imaging system quality, it can be useful for comparing digital systems with older analog systems.

The definition of TVL is problematic because “resolved” or “visible” are not clearly defined. There is no description of the viewing conditions, no objective definition of the contrast representing “light” and “dark”, and no description of how much contrast is lost before detail is considered “no longer resolved”.  Is a 90% loss of the signal (MTF10) enough to be considered “no longer resolved?”  This is a relatively common definition of resolution (derived from the Rayleigh diffraction limit), and it has been standardized for Camera Monitor Systems in the ISO 16505:2015 standard.  We have proposed an improved definition of resolution, but it has only gained limited traction. Using wedge patterns, some observers state that lines can no longer be resolved at 70% (MTF30) or 80% (MTF20) loss of contrast. Relying on subjective assessment of observers makes it difficult to get a consistent objective definition of TV lines.

TVL measurements do not consider changes in sharpness across the image that are typical of conventional lenses. Measuring the center-only may produce optimistic results. The EIA 1956 and ISO-12233:2000 charts had wedges in the corners that were easy to ignore. You should consider points all across the image to get a full understanding of your system’s resolution. You can use a mean of these values if you only want a single number. Remember that the tilt of your lens can degrade resolution in some corners, but boost it in opposite corners, without having much effect on the mean MTF.

A convenient, stable method of measuring TV lines in displays (i.e.,monitors) is given in Display (Monitor) Sharpness. For most other images we recommend using MTF20 (or better, MTF20P) in units of LW/PH (Line Widths per Picture Height), measured with (near) vertical slanted edges. (MTF20P is less sensitive than MTF20 to excessive software sharpening.)

note: for cropped images enter the original picture height into the more settings dimensions input.

If you can control the processing of your imaging system so you can disable sharpening or analyze unsharpened RAW images, you can make an approximate TVL measurement using “MTF10 LW/PH” or “MTF20 LW/PH” from vertical edges across the image. You can use the Multi-region summary plot from SFR, SFRplus, eSFR ISO, Checkerboard, or SFRreg modules to obtain a weighted mean MTF10 or MTF20 in LW/PH units, which are good objective approximations to the TVL metric. This is superior to the wedge measurement described below since it is less affected by aliasing, but it may not be the best measurement if you cannot disable your cameras sharpening.
If you have a “black box camera” system where you cannot disable sharpening, we recommend the hyperbolic wedge pattern to measure high-frequency MTF or vanishing resolution. Otherwise, the slanted edge MTF10 or MTF20 numbers may be exaggerated by the sharpening. Our best target for measuring multiple wedges across the field is our ISO 12233:2017 target with “extra wedges”. The best Output is our Multi-Wedge plot. The Weighted Mean Wedge MTF10 or MTF20 in units of LW/PH is very close to TV Lines.

Read More

Interactive vs. Auto(Batch) Analyses

Imatest has two types of analysis module: Fixed (in the left-most column of the main window) and Interactive (in the second column).
  • Fixed modules require that all settings be entered prior to running the analysis. Stored settings are read from imatest.ini and can be changed by user input. Results are displayed as figures and can be saved as image (usually PNG), CSV, XML, and JSON files.
  • Interactive modules are run from GUIs (Graphic User Interfaces) that allow results to be queried and modified after the analysis has been run. This allows you to explore results in great depth. Results can be saved as image (usually PNG), CSV, XML, and JSON files.
Auto(Batch) / Fixed vs. Interactive module summary
Auto(Batch) / Fixed Interactive
Graphic results in Figures allow limited manipulation (zooming or rotation for 3D images). Graphic results displayed in GUI windows allow a high degree of manipulation: You can change calculation and display settings and you can select any available display. You can analyzed results in great depth.
Batches of files can be run. Only a single file can be analyzed (one exception: several files can be combined (signal-averagted) to improve signal-to-noise ratio (SNR)).
Most are available as Industrial Testing (IT) modules (EXE or DLL programs that can run in production/quality control environments). Not directly available in IT.
Images cannot be directly acquired (in the Image Sensor edition). In the Imatest Image Sensor Edition, Images can be acquired directly from devices (development boards from Aptina, Omnivision and others as well as devices supported by the Matlab Image Acquisition toolbox). Directly-acquired images can be continuously refreshed in realtime.


In most (but not all) cases there are corresponding Fixed and Interactive modules.


Coresponding Interactive and Auto(Batch) / Fixed modules
Interactive module
Auto(Batch) / Fixed module

Interactive interface for several separate sharpness modules.

SFRplus Auto All settings must be made in Rescharts (SFRplus Setup). SFRplus Auto is highly automated (with automated region detection based on criteria set in Rescharts), requiring no user input.
SFR Fixed SFR can analyze several regions (ROIs); Rescharts SFR can only analyze a single region.
Random/Dead Leaves  
Log F-Contrast  
Any image sharpness  

Analyze color/grayscale charts.

ColorTest Can analyze a large variety of color charts, including the Colorchecker 24-patch and SG, the DSC Labs ChromaduMonde, the IT8.7, and many others (all charts supported by Multicharts). Before running ColorTest, the chart type should be selected in ColorTest settings (in the Fixed Modules or Settings dropdown menus). Plots are identical to Multicharts. There is considerable overlap with Stepchart and Colorcheck, both of which were developed earlier.
Stepchart Can analyze a large variety of grayscale stepcharts, including the Q-13/Q-14, ISO-14524, ISO-15739, Imatest 36-patch Dynamic Range chart, and many others. Figures are slightly different from Multicharts, but Multicharts chart seleections 6. Grayscale Stepchart (linear) and 7. Special Grayscale & Color charts correspond to charts supported by Stepchart.
Uniformity Interactive Uniformity Settings made in either module are used by the other. Blemish Detect capability is not yet included in Uniformity Interactive, but Blemish (fixed module) has most Uniformity functions.


Read More

Imatest Releases Version 2021.1

Imatest Version 2021.1 adds several new features to Imatest Master and Imatest IT, including edge tracking during live focusing, alignment tools in device manager, new subjective exposure quality loss calculation, OpenEXR support, live exposure and white balance measurements and several customer-requested features.

Read More

Calculating Exposure Quality Loss per IEEE CPIQ v2

CPIQ Auto exposure is in draft status. It will be added to the 2021 revision of the CPIQ Standard. You should use Imatest v2021.1 or later to perform this calculation.

Auto exposure error is based on deviation from a nominal luminance value for the neutral grey patch in an X-Rite Colorchecker SG. In a variant calculation, we can also calculate this metric from a 24-patch X-Rite Colorchecker.

A psychophysical study of the subjective quality based on luminance level was how the following model for exposure quality loss was developed:

CPIQ Auto exposure is measured with the Imatest Color/Tone module, which produces a Quality Loss (QL) score in units of Just Noticeable Differences (JND).

Results are available in display #4 2D a*b:

and display #7 B&W density & White Balance:


For more about Imatest’s CPIQ support click here

Read More

Imatest Releases Version 2020.2

Imatest Version 2020.2 adds several new features to Imatest Master and Imatest IT, including Improvements in Calculation Speed, Uniformity Statistics based on EMVA 1288 Standards, Uniformity measurements from ISO 17957:2015, IT For Mac OS, and Tunable Checkerboard Detection. (more…)

Read More

Obtaining Uniform Flat-Field Images

When measuring uniformity, in order to minimize cosine falloff in imaging systems with wider field of view lenses, it is important to have a light source that produces uniform illumination across the entire field of view.  The equipment listed below can be deployed to produce uniform images that fit the spectral requirements and field of view of your imaging system:

Field of View Visual and/or Narrow Band  Broad Spectrum VIS+IR / SWIR
<= (80°, ~110°) Imatest Lightbox Gamma Scientific RS-7-1 (80°)
<= 180°   Gamma Scientific RS-7-6 (180°)
<= 300° Colorspace LB2  

Learn More



Read More

Some Imatest calculations up to v5.1 inappropriately averaged zeros with summary metrics


Certain imatest calculations in version 5.1 and below produced zeros when the calculation was not determined. These zero values could be inappropriately averaged with summary metrics which lead to these summary metrics being underreported.

The problem includes, but is not limited to the following summary metrics:

  • Chromatic Aberration (CA) summary calculations CA_areaPCT_summary, CA_crossingPCT_summary, CA_R_G_PCT_summary, CA_B_G_PCT_summary where the individual outputs in CA_area_Pct_corner, CA_cross_Pct_corner, CA_crossing_R_G_PCT_corner, CA_crossing_B_G_PCT_corner, CA_crossing_R_G_Pxls, CA_crossing_B_G_Pxls, include zeros.
  • MTF Summary metrics where higher frequency MTF values that are not achieved, and incorrectly reported as zero (corrected in v5.1.28)


  • Take care not to use any summary metrics where the calculations output zeros
  • Upgrade to Imatest 5.2 where these indeterminate calculations are reported as NaN (Not a Number), and will not be included in any averages
Read More

5.2 MacOS Mojave system freezes during file operations

On some MacOS systems, the 5.2 Imatest release has an issue where the system temporarily freezes for under 60 seconds when file dialogs are opened.

We have traced this to the new 2019a MATLAB compiler runtime and are working with MathWorks to resolve the issue. (more…)

Read More

Which Imatest Charts Support Automatic Detection?

Using testing charts (also known as targets) improves the efficiency and repeatability of your testing workflow. Imatest offers several charts that support automatic detection as well as charts with registration marks.


Read More

Connectivity requirements for online or floating license activation

In order to perform online license activation or to use a floating license, Imatest needs to be able to perform name resolution or connect through a network to the following licensing servers:

Server Name IP Address Port Protocol
login 443  https
license 443 https


Read More

Error -4304: Users having problems activating licenses or starting trials

Recently we have been receiving reports of end-users in China being blocked from accessing the Imatest license servers (IP address

A message such as “SOAP couldn’t connect to host; Error code -4304″ may be displayed.

Here are the current workarounds to resolve this issue for this subset of end-users:

  • End-users have had success using a VPN to tunnel to either another location within China that is not impacted by an ISP block, or outside of China. This is typically the easiest solution to address the issue.
  • Use offline activation: The offline activation portal accessible through is still functional.

Note: this offline activation approach works both for individual node-locked licenses as well as offline activation of network-based licenses within the Nalpeiron LAN Daemon – which then allows floating licensing, element pools, floating features, etc. all to be used within the application.

We apologize for the inconvenience. Please contact us at with any difficulties you have.

Read More

Color & density reference files in CSV or CGATS format

Color reference files read by Imatest can be CSV or CGATS format containing L*a*b* color and luminance reference data.

The reference illuminant is assumed to be the same as the selected color space (D65 for sRGB, etc.). This is at variance with current practice, where L*a*b* files are assumed to have D50 data. The new LAB D50 file (Imatest 4.4+) is recommended because it will transform the reference values using a Bradford Transform to move them to the selected color space’s white point. This is the recommended approach if you have spectrophotometer measurements for your individual chart.

Color Reference files

Color CSV reference files files consist of the number of lines that the test chart has patches, with L*, a*, b* values on each line separated by spaces, commas (,), or semicolons (;). Example (first 3 lines of 24):

38.08, 12.09, 14.39
66.38, 13.22, 17.14
51.06, 0.38, -22.06

If you have an Excel .CSV file with extra rows or columns, you can easily edit it Excel by selecting the key region (3 columns, 24 rows), copying it to a new file, and saving it in .CSV format. L*a*b* data is preferred to xyY data below because it is independent of white point color temperature, hence less error-prone. 

Color reference files can also be derived from measured values from a camera you determine to be a “gold standard” by running Color/Tone Interactive, then pressing FileSave L*a*b* results as CSV reference (making sure to check CSV reference file of L*a*b* results…). This can be useful for manufacturing testing because the camera under test is compared with the attainable values of a golden sample rather than unattainable ideal values.

The CGATS file format is also supported.

Color/Tone Interactive General mxn charts (4×12 shown on the left). Any chart with an m x n grid can be analyzed. You must specify the grid size (click SettingsISO speed & mxn chart settings to set the number of rows (m) and columns (n). You must also enter a CSV or CGATS file with reference patch settings (typically L*, a*, b* values, one set per line). If you have a spectrophotometer you can create a custom reference file (CGATS format) using techniques described in Measuring test chart patches.

Obtaining the color reference

In Color/Tone Interactive you can perform a number of operations that relate to the reference file using the File dropdown menu, shown on the right.

The operation of interest here is Copy L*a*b* reference to clipboard. This shows the currently selected reference, whether it’s a standard or custom file. You can also save it as a named file for future use as a reference— perhaps for a “gold standard”. Here are the first three values for the post-Nov 2014 X-Rite Colorchecker default values.

37.31 13.39 14.58
64.37 18.05 17.05
49.62 -1.162 -22.16

These are close, but not identical, to the values shown above, which were derived from a different source.

Density Reference Files

Density CSV reference files are much simpler, they contain one optical density value per line, going from least-dense (smallest) patch, up to the most dense patch.  

Imatest will provide you with a density reference file when you purchase a 36-patch dynamic range or contrast resolution chart. You should always use this reference file while performing dynamic range measurements with multicharts or color-tone instead of using the default values.

Note that Copy density reference to clipboard is available in the File dropdown menu on the right.

See Also


Read More

Understanding collimator MTF loss using bronze and golden sample testing

by Henry Koren, inspired by Paul Romanczyk, edited by Norman Koren

Not all MTF measurement systems will necessarily provide the same results. The quality of the test target can impact the measurements you obtain. Long distance tests are ideally performed at the hyperfocal distance, where there is enough depth of field to have acceptable focus at infinity. (more…)

Read More

Henry’s Imatest Training Links

Here are Henry’s favorite Imatest Links that he often mentions during the Imatest Training Class:

Day 1

Day 2

Read More

Test Chart Substrate and Sizing Guide

This guide enumerates the possibilities and constraints of our test charts and provides resolution (MTF) information for the different options. Imatest has introduced Chart MTF Compensation, which can as much as double the resolution suitability of a given test chart.


Inkjet Substrate - Close up at 1mm

Close up of inkjet print quality

Inkjet is the lowest resolution process we offer, but also the least expensive and most versatile. The minimum print size with inkjet is approximately 8” x 10”, and the maximum print size for a single sheet is 44” x 80”.

Inkjet charts are available in three different substrates: Matte and Semigloss for reflective testing and DisplayTrans for transmissive tests using a backlight. All inkjet options are roughly the same resolution.

Inkjet is best suited for cameras tested at a moderate distance in order to compensate for the low resolution of the print. It is also suitable for low-resolution or wide-angle systems. As the resolution of the target is relatively low, the size of the chart needs to be larger to accommodate a higher resolution sensor.

We can produce custom charts, and custom sizes of most of our standard charts. Contact with your requested size and information about your lens (field of view and testing distance) and sensor (pixel count and aspect ratio.) We need the pixel count to confirm the size you are requesting will be adequate for your imaging system. Our Chart Finder Tool can help you determine the size of your imaging plane at a specific testing distance.

Black and White Photographic Paper (Silver Halide)

Photographic Paper Print Quality

Close up of photographic paper print quality

Photographic media offers high-resolution reflective prints when a transmissive target is not an option. These prints are over twice the resolution of inkjet media.

The minimum size of a photographic print is approximately 8” x 10” and the maximum size is approximately 40” x 50”.

Our standard photographic targets include our eSFR ISO chart (3 sizes) and a multi-size option of our SFRplus chart. This multi-size is best suited for testing low-resolution systems at close distances. We can offer custom charts on this media for an additional fee. Contact for information. 

Transmissive Photographic Film

Our Color and Black & White film targets are printed using high-precision LVT Technology. These transmissive targets require a lightbox in order to be used properly. 

Black and White

Black and White Film Print Quality

Close up of Black and White Film print quality

Black and White film is over twice the resolution of our inkjet targets, and can optionally include a color patch for color measurements. The size of the film is 12” x 20” or 16” x 20” (with ½” on each side as unprintable area). The minimum target size on this media is roughly 2” x 3”. We can print multiple targets on a single sheet of film depending on the size and resolution requirements to help make this option more cost-effective.

This media is suited for customers testing in a small area with low to medium resolution cameras.


Color Film print quality

Close up of Color Film print quality

Our high-precision color film targets are printed on 8” x 10” film with a maximum printed area of 9.25” x 7.75”. The minimum target size on this media is approximately 0.5” x 1”. As with the Black and White film, we can print multiple targets to a single sheet of film to make this a more cost-effective option.

This media is suited for customers requiring a very compact testing system or those using conversion optics to simulate infinity within a very small space. The fine detail of the film allows us to test mid to high-resolution cameras at close distances and with a small field of view. Color film has the highest dynamic range of all of the targets we offer. 

Chrome on Glass Photomask

Chrome on Glass Print Quality

Close up of Chrome on Glass print quality

Chrome Photomasks are our sharpest option for resolution testing. We use a lithographic photomasking process to print chrome directly onto glass. This process is single tone, so there is no gray-scale or tonal variance.

Chrome on Glass offers superior resolution when color and tonal measurements are not needed. We can make targets as small as 2mm x 3mm for measuring devices such as endoscopes or microscopy equipment. When printing to glass, the chrome has an optical density of 1, which gives a contrast ratio of approximately 10:1, suitable for resolution measurement.

Our standard glass plates are 4” x 4” with a non-printable border of 1 cm. We can source plates up to 20” x 20” and as small as 1” x 1”. Any plates larger than 4” x 4” require an extended lead time of 6-8 weeks. 

We can customize the size of various targets such as our SFRplus or ISO-eSFR targets on request. We can also print most custom design requests to glass, though some may require an engineering fee depending on the complexity of the design. Contact

Test chart suitability

Detailed information on test chart suitability — relating camera resolution to chart size — can be found on Test chart suitability for MTF Measurements

General information on sharpness, MTF, and the wide range of available charts can be found in the Sharpness section of the Imatest Documentation page

Read More

Reducing the cross-lab variation of image quality metrics


As imaging test labs seek to obtain objective performance scores of camera systems, many factors can skew the results. (more…)
Read More

New 2019 Imatest Licensing – Update Required to Activate

We are migrating to a better, faster licensing platform for our software on January 1, 2019. You may need to update your software to activate it on a new device, depending on your software version. Updating your software is free of charge.

Who is affected by this?

If your license code starts with 2848, you will need to update your software after January 1, 2019 if you need to install it on other devices or reactivate it on a current device. Your software will continue to work as normal until you need to reinstall or reactivate.

What actions do you need to take?

You will need to update your software in order install it on other devices or reactivate it on your current device. Check below for your software build download link. If you’re not sure which version you need, contact us.

What happens if you don’t update?

You can continue to use your software as it is without updating it. However, if you attempt to reactivate it or install your software on another device after January 1, 2019, you won’t be able to do so. You’ll need to update the software to continue using it.

What are the benefits of this?

  • Our licenses will now run properly on OSX versions High Sierra and higher.
  • Our licenses will have fewer issues activating on machines with heavy security and those that run in offline environments.

What version do you have?

As a reminder, this applies to licensing codes starting in 2848 only.

On opening Imatest, the version you are running will be displayed at the top of the main screen, in the command window, or in the Help > About window.   imatest software not upgrading

Download your software update today.


Version 4.1+ : 

Visit our download page for the latest builds of these versions which support the new licensing system.


Read More