Handling an Lightroom for iPhone RAW DNG file

Adobe DNG converter for windows & mac can be used to re-save iPhone RAW files into supported DNG formats.

We soon hope to will be adding native support for iPhone RAW 

Read More

Test Chart Substrate and Sizing Guide

This guide is intended to enumerate the various possibilities and constraints of our various printed materials, and give a rough idea of the resolution of each material.

Inkjet

Inkjet is the lowest resolution process we have available, but also the cheapest and most versatile. The minimum print size with inkjet is approximately 8” x 10”, and the maximum print size for a single sheet is 44” x 80”.

We have three different substrates available for inkjet prints. We can use Matte or Semigloss paper, both of which are reflective. We also have the option of using DisplayTrans, which is a transmissive media intended for use with a backlight. All inkjet options are roughly the same resolution.

Inkjet is best suited to cameras tested at a moderate distance in order to compensate for the low quality of the print. It also works for low-resolution or wide-angle systems where chart quality is less important. As the resolution is relatively low, the size of the chart needs to be much larger to accommodate a sensor of higher resolution.

We can produce custom charts and custom sizes of any of the charts we offer. Simply email charts@imatest.com with the requested sizing and information about your lens (Field of View, and testing distance) and sensor (pixel count and aspect ratio.) We need the pixel count to be certain that the size you are requesting will be adequate for your imaging system. If needed you can use our Chart Finder Tool to determine the size of your imaging plane at a specific testing distance.

We also have available the option to tile together multiple pieces of paper to create even larger charts for extremely wide angle imaging systems.  The constraints of the multi-sheet process are the number of columns that can print on each sheet, and the chart must be assembled on site by the customer due to the extremely large sizes. We have currently produced charts as large as 80” x 120” using this process. 

Photographic Paper (Silver Halide)

Photographic media offers high-resolution reflective prints for when using a transmissive target is not an option. These prints are over twice the resolution of inkjet media.

The minimum size of a photographic print is approximately 8” x 10” and the maximum size is approximately 40” x 50”.

We can do customized sizes on this media, however there is a substantial engineering fee ($3000) for the work required to customize these targets. We offer 3 standard sizes: 1x, 2x, and 4x. Soon we wil offer a small multi-size option which has 5 different sizes on an 8” x 10” print. This multi-size is best suited for those testing low-resolution systems at close distances. Contact charts@imatest.com if you are interested.

Transmissive Photographic Film

Our film is printed using high-precision LVT Technology. We have two different types of film available. Black and White and Color. These targets are transmissive and require a lightbox in order to be used properly. 

Black and White

The Black and White film is over twice the resolution of our inkjet targets, and can optionally include a color patch for customers needing to do color measurements. The size of the film is 12” x 20” or 16” x 20” (with ½” on each side as unprintable area). The minimum target size on this media is roughly 2” x 3”. We can print multiple targets on a single sheet of film depending on the size and resolution requirements to help make this option more cost-effective.

This media is suited for customers testing in a small area with low to medium resolution cameras.

For high-resolution systems, we can tile together multiple pieces of LVT film to fit the size of your lightbox. The maximum dimensions using this option would be approximately 19” x 32”. This option is best suited for customers testing high-resolution cameras such as DSLR’s. 

Color

Our High-Precision Color film is printed on 8” x 10” film with a maximum printed area of 9.25” x 7.75”. The minimum target size on this media is approximately 0.5” x 1”. As with the Black and White film, we can print multiple targets to a single sheet of film to make this a more cost-effective option.

This media is suited for customers requiring a very compact testing system or those using conversion optics to simulate infinity within a very small space. The fine detail of the film allows us to test mid to high-resolution cameras at close distances and with a small field of view. Color film also has the highest dynamic range of all of the targets we offer. 

Chrome on Glass Photomask 

Chrome Photomasks are by far the absolute sharpest option we have available for resolution testing, though it is also our most expensive option. We use a lithographic photomasking process to print chrome directly onto glass. This process is single tone, so there is no gray-scale or tonal variance.

Chrome on Glass offers superior resolution at the cost of color and tonal measurements. Using chrome on glass, we can make targets as small as 2mm x 3mm for measuring devices such as endoscopes or microscopy equipment. When printing to glass, the chrome has an optical density of 1, which gives a contrast ratio of approximately 10:1, suitable for resolution measurement.

Our standard glass plates are 4” x 4” with a non-printable border of 1 cm. At maximum size on our 4” glass plates, we can measure imaging systems with a resolution above 25 Megapixels.

We can source plates up to 20” x 20” and as small as 1” x 1”. Any plates larger than 4” x 4” will require an extended lead time of 6-8 weeks. A 20” plate with our SFRplus pattern would be suitable for testing cameras over 100MP.

We can customize the size of various targets such as our own SFRplus or ISO-eSFR targets on request. We can also print most custom design requests to glass, though some may require an engineering fee depending on the complexity of the design.  Contact charts@imatest.com.

 

Read More

Calculating Field of View (FoV) using distance settings

For accurate calculation of field of view (FoV) from using Imatest’s SFRplus, eSFR ISO, or Checkerboard, modules, it is critical to use proper distance measurements entered into in centimeters into the more settings window.

Lens to chart distance (in centimeters) can be best measured by a laser range finder.

For SFRplus it’s the bar to bar height is measured from the top of the top bar to the bottom of the bottom bar.

For eSFR ISO this is the vertical distance between the registration marks in centimeters.

Checkerboard uses the size of the squares.

Read More

Automating CPIQ analysis Using Imatest IT and Python

The Python interface to Imatest IT provides a simple means of invoking Imatest’s tests. This post will show how Imatest runs can be automated, then the results of those tests can be collected and easily processed. For this example, we will run five Imatest modules across three light levels, then extract CPIQ quality loss metrics.

Development Setup

Image Collection

For each device, we capture the following set of images of different test charts at different light levels:   (Note: full details will be available with the publication of the standard in March of 2017)

  • DeviceX_eSFR_5000KLED_1000lux.JPG
  • DeviceX_eSFR_TL84_100lux.JPG
  • DeviceX_eSFR_Tung_10lux.JPG
  • DeviceX_Dot_5000KLED_1000lux.JPG
  • DeviceX_Dot_TL84_100lux.JPG
  • DeviceX_Dot_Tung_10lux.JPG
  • DeviceX_SG_5000KLED_1000lux.JPG
  • DeviceX_SG_TL84_100lux.JPG
  • DeviceX_SG_Tung_10lux.JPG
  • DeviceX_Coins_5000KLED_1000lux.JPG
  • DeviceX_Coins_TL84_100lux.JPG
  • DeviceX_Coins_Tung_10lux.JPG
  • DeviceX_Unif_100lux.JPG

Running All The Tests

Once we have these images collected in a folder named “DeviceX”, we define our lighting conditions, initialize the Imatest IT library and run our processing function on this and a number of other devices:

led5k = "5000KLED_1000lux"               # Light Levels
tl84 = "TL84_100lux"
tung = "Tung_10lux"
light_conditions = [led5k, tl84, tung]
imatest = ImatestLibrary()
phones = ['DeviceX',                     # Just for example...
          'DeviceY',
          'DeviceZ']
for phone in phones:
    calc_phone(phone)

imatest.terminate_library()

The main processing script invokes the five modules, collects their data outputs and saves them to a JSON file:

def calc_phone(phone):
    print('Calculating Metrics for '+phone+ ' *****************************')
    uniformity_results = uniformity(phone) # Uniformity
    multitest_results  = multitest(phone) # Multitest
    dotpattern_results = dotpattern(phone) # Dot Pattern
    esfriso_results    = esfriso(phone) # eSFR ISO
    random_results     = random(phone) # Random

    # Output data to disk
    file_name = os.path.join(images_dir, phone, phone + '.json')
    with open(file_name, 'w') as outfile:
        full_data = {'multitest'  : multitest_results,
                     'dotpattern' : dotpattern_results,
                     'esfriso' : esfriso_results,
                     'random' : random_results,
                     'uniformity' : uniformity_results}
        json.dump(full_data, outfile, indent=4, separators=(',',': '))

    return full_data

We call the Imatest eSFR ISO module for each specified lighting condition:

# eSFR ISO (ISO 12233:2014) is used to test Visual Noise (VN) 
# and Spatial Frequency Response (SFR)
def esfriso(base):
    global images_dir, ini_file, light_conditions, op_mode
    output = {}
    for light_source in light_conditions:
        print('Testing ' + base + ' eSFR ISO at ' + light_source)
        input_file = os.path.join(images_dir, base, base+'_eSFR_'+light_source+'.jpg')
        if os.path.exists(input_file) != True:
            raise Exception('File ' + input_file + 'not found!')
        print('opening ' + input_file)
        result = imatest.esfriso_json(input_file=input_file, root_dir=root_dir, 
                                      op_mode=op_mode, ini_file=ini_file)
        data = json.loads(result)
        output[light_source] = data['esfrisoResults'];

return output

On the fly INI file changes

The most difficult part of this processing is to manually update region selections for the calls to Multitest, which does not yet have automatic region detection for the Colorchecker SG target (coming soon…).

In this case, we are we replacing the [sg] roi, nwid_save, and nht_save keys with the values stored in this multicharts-rois.json file:

{
"DeviceX_SG_5000KLED_1000lux.JPG": "468 417 2122 1578 468 1588 2102 397",
"DeviceX_SG_TL84_100lux.JPG":"482 376 2142 1571 482 1561 2157 376",
"DeviceX_SG_Tung_10lux.JPG":"386 336 2016 1492 401 1497 2021 321",
"width":"2592",
"height":"1936"
}

We use this script to modify INI files to dynamically insert different settings:

import ConfigParser

def get_config(path): # Used make ConfigParser case sensitive
    config = ConfigParser.ConfigParser()
    config.optionxform=str
    try:
        config.read(os.path.expanduser(path))
        return config
     except Exception, e:
        log.error(e)

def setROI(roi,width,height):
    global ini_file
    imatestini = get_config(ini_file)

    imatestini.set('ccsg', 'roi', roi)
    imatestini.set('ccsg', 'nwid_save', width)
    imatestini.set('ccsg', 'nht_save', height)
    with open(tempINI, 'w') as outfile:
        imatestini.write(outfile)
    return

These functions are then called using the values loaded from the JSON file inside of our multicharts function call:

    roi_overrides = {}
    roi_overrides_lower = {}
    roi_override_file = os.path.join(images_dir,base,'multicharts-rois.json')
    if os.path.exists(roi_override_file):
        with open(roi_override_file,'r') as fh: # read ROI override file
            roi_overrides = json.load(fh)
        for override, rois in roi_overrides.iteritems(): # convert to lowercase
            roi_overrides_lower[override.lower()] = rois
    else:
        raise Exception('ROI override file ' + roi_override_file + ' not found.')
 
    if image_file.lower() in roi_overrides_lower:
         setROI(roi_overrides_lower[image_file.lower()], roi_overrides['width'], roi_overrides['height'])
         selectedINI = tempINI
    else:
         raise Exception('Missing ROI override for ' + image_file)

Extracting Scores

Finally, we extract Key Performance Indicators (KPI), which are all in perceptual quality loss (QL) in units of just noticeable differences (JND).  This script is a bit long for this post, so view the source on Github if you are interested./

Here is the resulting set of calculated metrics in JSON format:

{ 'DeviceX': 
    { 'CL_QL': { '5000KLED_1000lux': 0.9142,
                 'TL84_100lux': 0.1803,
                 'Tung_10lux': 2.565},
      'CU_QL': 0.07498,
      'LCD_QL': 0,
      'LGD_QL': 0,
      'SFR_QL': { '5000KLED_1000lux': 0.0,
                  'TL84_100lux': 0.1969,
                  'Tung_10lux': 0.3946},
      'TB_QL': { '5000KLED_1000lux': 1.2,
                 'TL84_100lux': 6.25,
                 'Tung_10lux': 9.1},
      'VN_QL': { '5000KLED_1000lux': 1.113,
                 'TL84_100lux': 2.409,
                 'Tung_10lux': 8.531}},
      'Combined_QL': 11.0245
}

Once test procedures and combined score calculations are finalized by the IEEE Conformity Assessment Program, these outputs will be combined into a simple camera phone rating score.

The full python source code for this example is posted on GitHub.

To add these automation capabilities to your existing Imatest Master license, upgrade to Imatest Ultimate. You can also request a free 30-day trial of Imatest IT to see how this works for you before you purchase.

 

Related Webinars

Join us March 1 for Automating Lab and Manufacturing Processes: Defining image quality parameters on automated test equipment. Register now

Read More

The Effects of misregistration on the dead leaves cross-correlation texture blur analysis

This paper was given as part of the Electronic Imaging 2017 Image Quality and System Performance XIV and Digital Photography and Mobile Imaging XIII sessions.

When: Tuesday, January 31, 2017, at 12:10 pm

By: Robert Sumner with support from Ranga Burada, Noah Kram

Abstract: The dead leaves image model is often used for measurement of the spatial frequency response (SFR) of digital cameras, where response to fine texture is of interest. It has a power spectral density (PSD) similar to natural images and image features of varying sizes, making it useful for measuring the texture-blurring effects of non-linear noise reduction which may not be well analyzed by traditional methods. The standard approach for analyzing images of this model is to compare observed PSDs to the analytically known one. However, recent works have proposed a cross-correlation based approach which promises more robust measurements via full-reference comparison with the known true pattern. A major assumption of this method is that the observed image and reference image can be aligned (registered) with sub-pixel accuracy.

Read Full Paper:

Effects of misregistration on the dead leaves cross-correlation texture blur analysis

Read More

Measuring MTF with wedges: Pitfalls and best practices

This paper was given as part of the Electronic Imaging 2017 Autonomous Vehicles and Machine session.

When: Monday, January 30, 2017, at 10:10 am

By: Norman Koren with support from Henry Koren, Robert Sumner

Abstract:  The ISO 16505 standard for automotive Camera Monitor Systems uses high contrast hyperbolic wedges instead of slanted-edges to measure system resolution, defined as MTF10 (the spatial frequency where MTF = 10% of its low frequency value). Wedges were chosen based on the claim that slanted-edges are sensitive to signal processing. While this is indeed the case, we have found that wedges are also highly sensitive and present a number of meas­ure­ment challenges: Sub-pixel location variations cause unavoidable inconsistencies; wedge saturation makes results more stable at the expense of accuracy; MTF10 can be boosted by sharpening, noise, and other artifacts, and may never be reached. Poor quality images can exhibit high MTF10. We show that the onset of aliasing is a more stable performance indicator, and we discuss methods of getting the most accurate results from wedges as well as misun­derstandings about low contrast slanted-edges, which cor­relate better with system performance and are more repre­sentative of objects of interest in automotive and security imaging.

Full Text: Measuring MTF with Wedges: Pitfalls and best practices

Slides: Wedge_measurements_N_Koren_2017

Imatest upgrades based on the paper

The recommended metric to replace MTF10 (min(MTF10, onset of aliasing, Nyquist frequency)) is displayed in the Wedge MTF plot as well as the (new in Imatest 5.0) Multi-Wedge plot, shown below.

Multi-Wedge plot for eSFR ISO, including recommended metric.

Read More

Best practices for using transmissive test targets

Light Source

 

The most uniform light source available would be an integrating sphere, but these come with considerable size and expense. After considering how the light source uniformity impacts their results, most lab & manufacturing tests use LED or fluorescent lightboxes with flat-panel diffusers. Some customers choose a larger lightbox because that leads to greater uniformity within the central image plane area. We have performed a study of lightbox uniformity to see how different models compare. 

To perform tests with a transmissive chart setup, devices with built-in light sources will need to have that light source disabled. (more…)

Read More

How to get support for your image acquisition hardware

Not all hardware is supported by Imatest IS or Imatest IT’s acquire_image routine. In order to obtain support there are a number of possibilities:

1. Implement a standard image acquisition library

The quickest path to getting support for your device within the current version of Imatest is to implement a standard acquisition interface such as DirectShow or GenICam.

2. Integration Consulting

We can add support for your custom image acquisition SDK on a statement of work basis. Contact testing@imatest.com for a quote.

3. Get Support in MATLAB Image Acquisition Toolbox Adaptor

Many of our supported devices come courtesy of the Image Acquisition Toolbox. If you create a MATLAB imaq adaptor a future version of Imatest IS will be able to load that adaptor from a plugin folder. Please contact us if you are interested in this option.  

You might also request hardware support from Mathworks, but that could be a long process  for them to implement your device, release a new version of MATLAB and for us to upgrade to it.

 

Read More

Gamma, Chart Contrast and MTF Calculations

Gamma (the average slope of log pixel levels as a function of log exposure for light through dark gray tones) is used, per the ISO 12233 standard, to linearize the input data, i.e., to remove the gamma encoding applied by the camera or RAW converter. Gamma defaults to 0.5 = 1/2, which is typical of digital cameras, but may be affected by camera or RAW converter settings. 

Imatest Settings

For accurate calculation of edge-SFR from using Imatest’s SFRplus, eSFR ISO, SFRreg, Checkerboard, or SFR modules, it’s important to use a proper gamma for sharpness calculations.

If the chart contrast is known and is ≤10:1 (medium or low contrast), you can enter the contrast in the Chart contrast (for gamma calc.) box, then check the Use for MTF checkbox. Gamma will be then be calculated from the chart and displayed in the Edge/MTF plot.

Measuring Gamma

If chart contrast is not known you should measure gamma by obtaining the OECF (Opto-Electric Conversion Function) per the ISO 14524 standard.  This can be obtained using an image of a grayscale stepchart and running Colorcheck, Stepchart , Multicharts (interactive), or Multitest. A nominal value of gamma should be entered, even if the value of gamma derived from the chart (described above) is used to calculate MTF.

Errors in Gamma

Small errors in gamma have a minor effect on MTF measurements (a 10% error in gamma results in a 2.5% error in MTF50 for a normal contrast target). Gamma should be set to 0.45 when dcraw is used to convert RAW images into sRGB or a gamma=2.2 (Adobe RGB) color space. If gamma is set to less than 0.3 or greater than 0.8, the background will be changed to pink to indicate an unusual (possibly erroneous) selection.

Technical Details

Capture One LE with Film Curve
1. Capture One LE set to Film standard
(the default). Gamma = 0.679.

Capture One LE with Linear Curve
2. Capture One LE, Linear response. Gamma =
0.508. Recommended for SFR runs.

Canon FVU with Standard contrast

3. Canon FVU set to Standard contrast.
Gamma = 0.642.

Gamma is the exponent of the equation that relates image pixel level to luminance. For a monitor or print,

     Output luminance = (pixel level)gamma_display

When the raw output of the image sensor, which is linear, is converted to image file pixels for a standard color space, the approximate inverse of the above operation is applied.

     pixel level = (RAW pixel level)gamma_camera ~= exposuregamma_camera

The total system gamma is gamma_display * gamma_camera. Standard values of display gamma are 1.8 for older color spaces used in the Macintosh and 2.2 for color spaces used in Windows, such as sRGB (the default) and Adobe RGB (1998).

The three curves on the right, produced by Stepchart for the Canon EOS-10D, show how Gamma varies with RAW converter settings.In characteristic curves for film and paper, which use logarithmic scales (e.g., density (–log10(absorbed light) vs. log10(exposure)), gamma is the average slope of the transfer curve (excluding the “toe” and “shoulder” regions near the ends of the curve), i.e.,

Gamma is contrast.

See Kodak’s definition in Sensitometric and Image-Structure Data.

To obtain the correct MTF, Imatest must linearize the pixel levels— the camera’s gamma encoding must be removed. That is the purpose of Gamma in the SFR input dialog box, which defaults to 0.5, typical for digital cameras. It can, however, vary considerably with camera and RAW converter settings, most notably contrast.

Characteristic curves for the Canon EOS-10D with three RAW converter settings are shown on the right. Gamma deviates considerably from 0.5. Gamma = 0.679 could result in a 9% MTF50 error. For best accuracy we recommend measuring gamma using Colorcheck or Stepchart, which provides slightly more detailed results.

Confusion factor: Digital cameras rarely apply an exact gamma curve: A “tone reproduction curve” (an “S” curve) is often superposed on the gamma curve to extend dynamic range while maintaining visual contrast. This reduces contrast in highlights and (sometimes) deep shadows while maintaining or boosting it in middle tones. You can see it in curves 1 and 3, on the right. For this reason, “Linear response” (where no S-curves is applied on top of the gamma curve) is recommended for SFR measurements.

The transfer function may also be adaptive: camera gamma may be higher for low contrast scenes than for contrasty scenes. This can cause headaches with SFR measurements. But it’s not a bad idea generally; it’s quite similar to the development adjustments (N-1, N, N+1, etc.) in Ansel Adams’ zone system. For this reason it’s not a bad idea to place a Q-13 or Q-14 chart near the slanted edges.

To learn more about gamma, read Tonal quality and dynamic range in digital cameras and Monitor calibration.

Read More

Testing a macro lens using Checkerboard and Micro Multi-slide

Testing a 1-5x Macro Canon MP‑E 65mm Lens

Imatest’s Checkerboard module is our new flagship module for automated analysis of sharpness, distortion and chromatic aberration from a checkerboard (AKA chessboard) pattern. The big benefit of using the checkerboard is that there are looser framing requirements than with other kinds of test targets. While checkerboard lacks the color and tone analysis provided by SFRplus and eSFR ISO, these features are not available on the high precision chrome on glass substrate, so the checkerboard is the optimal pattern for this test.

The Imatest Micro Multi-Slide contains high precision checkerboard patterns with many different scales of frequencies. This makes a single target capable of effectively testing a wide range of magnifications.

We obtained the below cycles per object mm by dividing the LW/PH obtained from the image by 2*image height in mm.

Magnification Image height Center MTF50 Best Aperture
1x 23.71 mm 29.61 cycles / object mm f/5.6
2x 11.86 mm 57.04 cycles / object mm f/5.6
3x 7.90 mm 74.55 cycles / object mm f/5.6
4x 6.07 mm 81.52 cycles / object mm f/4
5x 4.74 mm 87.61 cycles / object mm f/4

 

Thanks to this study, we now know to select the f/4 aperture at 5x magnification to maximize the resolving power of this lens. Detailed results, capture and analysis procedures are available below.


1x Magnification

This report is shown in line widths per picture height (LW/PH):

1x

1x-3d


2x Magnification

2x

2x-3d


3x Magnification

3x

3x-3d


4x Magnification

4x

4x-3d


5x Magnification

At the limit of this lens’ magnification we obtain the most spatial detail on the target at f/4

5x

5x-3d


Capture Procedure

We use this camera setup to take high precision photographs of the test targets on a LED lightbox:  

micromulti_setup_scaled

We mounted the camera on an adjustable arm and used a Manfrotto 454 Micro Positioning plate for adjusting distance and focus. We masked off the extra parts of the lightbox using opaque material to prevent additional stray light from increasing flare.

We used the following framing for the various magnifications:

micromulti_site_blog_customized

To get good sharpness measurements we selected an area of the slide that had under 10 vertical squares in the image. This yields a reasonably large SFR region for the most reliable calculations and prevents there from being an excessive number of regions.

We centered the chart on the 3-dot mark in the center of the frequency zone. For optimal slanted-edge analysis, we rotated the chart by about 5° according to the ISO 12233 standard. Our current checkerboard routines can automatically detect the checkerboard, but they look for complete rows and columns before including the set of regions. Which means that rotation can make the framing a little more difficult, and if a corner of a complete row/column of squares gets clipped off, then the automatic region detection will skip some regions you may have wanted to test, producing less than optimal region availability around the periphery of your image. We will be improving these selection routines in future releases of our software.

If we wanted more detail about lateral chromatic aberration and distortion (which are very low for this lens) we would have analyzed the dot pattern regions of the chart.

We used 5 megapixel downsampled JPG’s from the camera to perform this analysis, which gives us the following table of image sizes:

Magnification Image height Image pixel size
1x 23.71mm 13µm
2x 11.86mm 6.5µm
3x 7.90mm 4.33µm
3.9x 6.07mm 3.33µm
5x 4.74mm 2.6µm

Analysis Procedure

We performed our Checkerboard Setup and selected all regions:

setup

After initially determining our range of expected MTF values, we disabled auto-scaling on our 3D plots and set our range to the total:

micromultiscaling

We used Imatest Batchview to coalesce the large volume of tests in order to produce the above bar graphs.  We also used ImageMagick to assemble the nifty animated GIF’s from the collections of 3D plots:

convert -delay 100 mag1/Results/*3D.png 1x-3D.gif

We hope that you find this write-up to be helpful in testing your own equipment. You can purchase Micro-multi Slide on our store or contact charts@imatest.com for customizing a test target that fits your unique requirements.  See our Macro Solutions for other close-range testing items.

Read More