Instructions and comments
We are updating this page for the latest Skype/Lync specification.
An index of of the Skype/Lync specifications can be found on http://technet.microsoft.com/en-us/lync/gg278181.aspx.
This document contains instructions for using Imatest with the Skype Hardware Certification Specification — For all Skype Video Devices Version 5.0. It also contains comments and suggestions (some of which we hope might be adopted in a future release of the spec).
The Skype spec uses only a tiny fraction of Imatest’s powerful capabilities. To learn more, see Image Quality Factors and SFRplus (which allows many factors to be measured from a single image). In addition to the GUI-based Master version, non-GUI versions— IT/EXE and IT/DLL— that can interface with manufacturing testing systems are available. Imatest also supports the Microsoft Lync Logo Video Capture Specification and the I3A CPIQ (Camera Phone Image Quality) Phase 2.0 specification.
Results of Skype tests are divided among three sections of the test specs: 2.2 Minimum requirements (required level), 2.3 HQV Requirements (preferred level; for 640×480 video resolution), and 2.4 HD Requirements (preferred level; for 720×1280 video resolution).
We are making a number of enhancements to Imatest 3.9 to make it easier to obtain Skype results. These will be in Imatest 3.9-Beta prior to the official 3.9 release in February or March 2012.
2.1.2 Acuity – MTF50 and Oversharpening, 2.1.14 Depth of field
Starting with Imatest 3.9, SFRplus has a region (ROI) selection setting that conforms to these tests.
SFRplus input settings, with Skype region (ROI) selection
MTF50 and Oversharpening ( (–)undersharpening) are displayed in the Edge and MTF plot. In order to display Over/undersharpening, Display oversharpening only should be selected in the SFRplus More settings window (in the bottom-left of the Settings area).
SFRplus Edge and MTF plot, showing Under/Oversharpening and MTF50
[Modify the Multi-ROI summary to display PW50, Oversharpening, and 1st Secondary readout?]
MTF50 (LW/PH) and Oversharpening (%) are written to the multi-region CSV output file, which has a name or the form filename_extension_multi.csv (4_JPG_multi.csv for the above example.)
2.2.3 (300lux 5000K) MTF50 ≥ 0.25 cycles/pixel. 2.3.3 (30 lux 3000K) MTF50 ≥ 0.33 cycles/pixel. 2.3.4 (300 lux 5000K) MTF50 ≥ 0.38 cycles/pixel. Fairly strong sharpening is usually required to meet the numbers in 2.3.3 and 2.3.4. 2.3.5 (300 lux 5000K) Oversharpening ≤ 40%. This number is associated with strong sharpening: edge artifacts (“halos”) should be highly visible with Oversharpening > 40%.
2.1.3 Lens distortion – SMIA distortion
Imatest uses a different method to measure SMIA distortion, with slightly different (generally larger) results because the measure is extrapolated to represent lines whose separation is 98% of the image height. Distortion can be measured with SFRplus, Distortion, and Dot Pattern.
SFRplus Image & Geometry plot, showing SMIA TV Distortion (-1.46%)
2.3.6 (300 lux 5000K, though lighting does not matter): |DSMIA| ≤ 3%. This spec should be easy to achieve, especially since distortion can be easily corrected in software (many cameras now do this).
2.1.4 Lens distortion – Stretch distortion
This can be derived from the Horizontal/Vertical aspect ratio, displayed (above) in the SFRplus Image & Geometry plot. Aspect ratio is A = Δw/Δh.
In the Skype spec, DStretch = 100%(w – h)/w = 100%(1 – h/w) = 100%(1 – 1/A).
2.3.7 (300 lux, 5000K, though lighting does not matter): |DStretch| ≤ 5%. This should be easy to achieve.
2.1.5 SNR – Spatial noise deviation
SNR and related measurements can be obtained from a standard Colorcheck run using an image of an X-Rite Colorchecker.
The (normalized) pixel levels are shown in the upper-left plot, and the standard deviations σ, which are the same as noise, (in units of % of the maximum pixel level, 255) are shown on the lower-left. There are other display options: SNR could have been shown instead. All results are summarized in the CSV output file. Here are portions:
|Zone||Gray||Pixel||Pixel/255||Px/255 ideal||Log(exp)||Log(px/255)||WB Err Deg||WB Err Mired||WB ERR S(HSV)||WB Delta-C ab||WB Delta-C 94||WB Delta-C 00|
|(Zones 1-18 (Rows 1-3), which contain color (non-grayscale) patches, are omitted here.)|
|S/N (signal/noise) and SNR(dB) (20*log10(signal/noise)) for the 6 gray patches. A is Average of RBGY channels.|
2.3.8, 2.3.9 (30 lux 3000K and 5000K) σMAX < 2.7. 2.3.10, 2.3.11 (300 lux 3000K and 5000K): σMAX < 1.7. In the above tables, σMAX = 1.113%*2.55 = 2.83. This spec seems rather stringent.
2.1.6 SNR – Temporal noise
Temporal noise is measured by running Colorcheck, selecting two identical images, and indicating that this is for Temporal noise (not separate or combined analysis). A fifth figure is available for Temporal noise, and all results are written to the CSV output file. The key Skype results in the CSV output— the worst Y (luminance) channel SNR (dB) for gray patches 1-5— are shown below.
Skype Temporal noise result: worst Y-channel SNR (db) of gray patches 1-5
2.3.12, 2.3.13 (30 lux): SNRTemporal ≥ 30dB, 2.3.14, 2.3.15 (300 lux): SNRTemporal ≥ 35dB. This spec is quite stringent.
2.1.7 Color accuracy and Saturation
These are measured with standard Colorcheck runs. In the Colorcheck settings window, Color error display should be set to Delta-C, Delta-E (standard). (Note: we would have preferred Delta-C 94 or Delta-C 00, both of which are more recent and accurate as perceptual color difference measurements.)
C¯r in the Skype spec (with ¯ above C) is Mean camera chroma (saturation) in the plot.It is Mean camera chroma % in the CSV output file.
ΔC*avg-abcorr is on the second line (right side) of the plot. It is Mean Delta-C sat corr in the CSV output file.
ΔC*abcorr (patch no. 2) (light skin) is the second line below Delta-C(corr) in the CSV output file.
2.1.8 Exposure accuracy
Exposure accuracy can be obtained from a standard Colorcheck run. The standard Imatest calculation (Exposure error (f-stops) in the CSV output file; displayed in all figures except a*b* color error) is slightly different from the Skype spec. The Imatest measurement uses patches 2-5 from the bottom row of the ColorChecker, while the Skype spec uses all 6 patches. The logic behind the Imatest calculation is that patch 1 is sometimes saturated (P1 = 255) and is often in the highly nonlinear “shoulder” region of the response, while patch 6 is too dark to be visually significant in estimating exposure error (and on rare occasions Pi may be zero so that log(Pi) = -∞. For patch 6, common tonal response curves with nearly invisible irregularities can have a large effect on the measured results.
Additional details: Pi is the luminance channel = 0.3*Ri + 0.59*Gi + 0.11*Bi .
2.2.4-2.2.8 (all light intensities, color temps): -1.5 ≤ Errexp ≤ 1.5
2.1.9 Dynamic range
The Skype Dynamic range, which is quite different from other Imatest dynamic range measurements, can be measured in a standard Stepchart run with an image of the Kodak/Tiffen Q-14 test chart, which consists of 20 patches with a density increment of 0.1 (3 patches per f-stop or EV) for a total tonal range of 1.9 density units = 6.3 f-stops (EV). This is considerably less than the dynamic range of most sensors and cameras. Essentially this spec flags really bad cameras (which may have excessive lens flare, defective sensors, or poorly-designed signal processing).
The Skype Dynamic range is the number of patches where the “brightness” step between the patches is greater than 0.02* the (maximum – minimum) brightness. In Imatest 3.9+ it is written to the CSV output file as shown below (… linear), and comparable values are written to the JSON output.
Skype Dynamic Range from the CSV output (linear is from the spec)
We have included a logarithmic measurement because we believe that a linear measurement is far from optimum. The eye has logarithmic response, and gamma (contrast) is defined using a logarithmic curve, which is a better indicator of camera performance.
2.1.10 Capture gamma
The Skype equation (apparently) uses the first two patches in the bottom row of the ColorChecker to calculate gamma. This is not representative of the overall gamma of the image because the first two patches are typically in the “shoulder” region of the contrast curve, where the contrast is often less than the visually-significant middle tones. Imatest uses patches 2-5 to calculate gamma.
The Imatest equation for gamma, reported in Colorcheck results, is based on a first order fit to patches 2-5 of the bottom row (which should be very close to the Skype equation in the great majority of cases). It is approximately
γ = log10(P2/P5)/(d5–d2), where Pi is defined identically to the Skype spec.
(We will be discussing this with Skype shortly.)
2.1.11 Light falloff and Color uniformity
This test is performed with Light Falloff. A white board (or a uniform light source) should give identical results to a gray board. If a reflective target (i.e., any board) is used, extreme care should be taken to obtain uniform lighting.
In the Light Falloff (Uniformity) settings window, select 5% for Corner and side regions. Location should be 0% (on image border), though 1% could be useful when there are shadows (that may be hard to see) next to the borders.
If you would like to see results in plots (they’re also in the CSV output), select Pixel contours only for Contour plots and Red/Green and Blue/Green for Color shading plots. We recommend checking Extra smoothing (on the right of the Color shading area) for best plot appearance.
The Y (luminance) contours are shown in the figure on the right. Relative illumination is
Irel = Pcenter/Pcorner(worstcase) = 100/(worst corner %)
where worst corner % = 70.8; hence Irel = 1.412. In the CSV file (Imatest 3.9+),
The R/G and B/G ratios are shown at the bottom of the Color shading figures and in the CSV output files.
2.1.15 Field of view consistency
Please note that 2.1.12 Frame rate, 2.1.13 Delay, and 2.1.16 Video Encoding tests are not performed by Imatest.