The Imatest Test Lab

How to build a testing lab

The Imatest Laboratory was set up for several purposes.

• To demonstrate how to construct a cost-effective image quality testing lab.
• To validate new Imatest modules and image quality testing concepts.
• To perform tests for clients who need low volumes of testing or want to verify their own testing methods.
• To photograph artwork. The optimum setup for measuring image quality is, not coincidentally, optimum for photographing fine art.

The lab was designed to meet the following goals.

• Low cost, easily obtainable, high quality components.
• Portable: easy to move if needed.

Hardware

Substitutions can be made for most of the parts listed below. Creativity is encouraged in assembling a lab.

Lighting

To obtain high quality illumination covering up to 40x60 inch (100x150 cm) foam board with reasonably good uniformity and smaller areas with high uniformity, we selected a track lighting system that takes the outstanding SoLux 4700K (near daylight) 12V MR16 halogen lamps, which use long-lasting built-in dichroic filters. Fixture.  The Hampton Bay K15 expandable halogen track lighting kit (724-491), available from Home Depot, along with the A18 end connector with cord and switch (678 546) is inexpensive and excellent. It's not on their website. Similar products should work equally well, for example, the Pro Track 150 Watt Low Voltage Track Kit (65227) from Lamps Plus. Lamps.  We use 6 SoLux 50W 4700K/36 degree MR16 bulbs, along with 6 Plano Convex diffusers, which are necessary for even light. The diffusers are mounted in front of the lamps (rightmost image), in place of the clear glass supplied with the track lighting kit. "T" braces with self-adhesive felt pads were added to the sides of the lamp fixtures to block stray light from reaching the lens (rightmost image).

Lights & stand

Lamp detail

SoLux halogen bulbs are inexpensive and produce extremely high quality near-daylight 4700K illumination, with a CRI (Color Rendering Index) of 98 out of 100. They were featured in a recent article in Inkjetart.com. SoLux also sells track lighting components.

Stand.  The Radio Shack 33-335 folding microphone stand is simple, lightweight, sturdy, and cheap.

Assembly.  The track is attached to the stand with an 8-32 bolt. Here is the procedure.

Drill holes large enough for an 8-32 bolt about 1.25 inches (3.5 cm) below the top of the upper shaft of the microphone stand and about 10 inches (25cm) below the top of the track (the end without the cord). The head of a 1.25 or 1.5 inch 8-32 bolt is placed inside the track, then attached with two nuts on the back of the track to provide a little spacing with the stand. A wing nut (optional) is used to secure the bolt to the stand. [Parts sequence: 8-32 bolt (head), track, 2 8-32 nuts, upper shaft of microphone stand, 8-32 wing nut.] (CAUTION: do not do this unless you know how to work with electrical wiring safely. DO NOT do this when the track is plugged in! )

The remainder of the track is attached to the stand by a combination of materials that allow the stand to be raised and lowered.

Cambuckles and nylon webbing. Nice because they put tension on the webbing when they are closed. 1 inch and 2 inch Available from REI. Strapall heavy-duty Quick Tape, available from REI and many other sources. This is a strong double-sided Velcro tape. Sticky-back Velcro tape. Can attach to both the mic stand and the back of the track, but contact area is limited. The lights are mounted near the top, middle, and bottom of the 40 inch high targets. If you have a large budget and need high intensity illumination, check out the Imaging Resource laboratory-grade lighting system, which uses HMI lamps. Easel The Mabef M-10 easel was chosen because it was sturdy, could be adjusted vertically (checked with a small level), had just the right features, and was on display at a local art supply store. A self-adhesive felt pad on the bottom of the upper clamp helps secure targets.

Light measurement The BK Precision 615 Light meter (Lux meter) is an outstanding low-cost instrument (about $100 USD) for measuring incident light (illuminance). It has a flat diffuser and photopic response (similar to the human eye in bright light). It is particularly useful for measuring evenness of illumination. We purchased ours from Action Electronics, which offers several alternative choices. The BK was chosen over similar Easyview meters because it has a flat (rather than hemispheric) diffuser that better represents the geometry of the flat targets. Readings can be somewhat slow, with settling times around 3-5 seconds. Miscellaneous Several additional items can come in handy: a small level, a flashlight (LED models are cool), rulers, a few clamps, etc. A bubble level mounted on a camera's accessory shoe can be particularly helpful. A few Irwin Quick-Grip Handi-Clamps are useful for attaching the targets to the easel.

The $99 USD Stanley 77-910 - TLM 100 FatMax™ Tru-Laser™ Distance Measurer shown on the right was added to the lab in November. 2007. It is very convenient for measuring precise distances— much more accurate than the measuring tape shown on the left.

Tripod and accessories

The tripod should be sturdy, easy to adjust, and easy to move. Any good tripod with a pan/tilt head should work. Ball heads are not recommended for the lab. A head that takes quick-release plates is strongly recommended. Any tripod can be adapted for quick release with the Bogen/Manfrotto RC2 adaptor, which is supplied with one 3157N plate. We illustrate a few Manfrotto components, which offer good value for the price. These represent typical choices; many more are available to meet special needs.

Manfrotto 475 or 475B tripod legs with geared center column - supports 26.5 lbs. Height 16.6 – 74 in. A geared center column is not strictly necessary (we don't have one on our old Bogen/Manfrotto 3021BN), but can be convenient in a tripod dedicated to a test lab. The built-in spirit level is valuable for leveling the tripod: If the base (where the head is mounted) is level, fine adjustments are much easier to make because head movements (swivel, etc.) don't interact.

Manfrotto 804RC2 3-way pan-tilt head with quick release - supports 8.8 lbs. A very nice head, very reasonably priced. Uses the versatile RC2 quick release system. Get several 3157N mounting plates if you purchase this head.

Manfrotto 3275 / 410 compact geared head with quick release - supports 11.1 lbs. A more elegant but more expensive head: especially nice for making precise adjustments. Uses the less common 3271 quick release mounting plate.

Manfrotto 3137 variable leg spread portable dolly. Very useful for keeping legs in place while moving the tripod.

475B tripod legs

804RC2 pan/tilt head

3275/410 geared head

Manfrotto 3137 dolly

Manfrotto 3416 leveling head. Optional. The level built into the 475B (and several other) legs may suffice for most users. Mounts between the tripod legs and the head. Ensures that the base of the head is level so that the various movements (swivel, etc.) don't interact. This greatly facilitates fine adjustments.

Clamps

Although a regular quick-release tripod mount is fine for most cameras, a good sturdy clamping system is needed for testing cameraphones and webcams. We put two of them together after rummaging through the aisles of Home Depot. They consist of Irwin Quick-Grip Mini or Micro bar clamps mounted on Manfrotto quick release plates. The images below give a good idea of how they're constructed. A hole had to be drilled in the larger clamp (above) and enlarged in the lower clamp (below) to accommodate the 1/4 inch bolt.

The larger clamp (the Mini) is sturdier, and has proven to be more useful.

Putting it together

The image below summarizes how the lights should be positioned. The goal is even, glare-free illumination. The lamps represent the stacks of three SoLux track lamps, described above. This may involve a tradeoff, especially with wide-angle lenses. Lighting angles between 35 and 45 degrees work well for normal and long lenses. Avoid lighting behind the camera, which can cause glare. Ambient light should be kept subdued. Check carefully for glare and lighting uniformity before you expose, especially with wide angle lenses, where glare can be difficult to avoid: you may have to move the lights further than normal from the target— or allow glare in portions of the target, taking care that it doesn't affect critical areas.

Use the meter to check for the evenness of lighting. ±20% over the entire target is sufficient for SFR and Distortion measurements, where small portions of the target are analyzed and exposure is not critical. ±5% or better should be the goal for tonal and color measurements (Stepchart, Colorcheck, and Multicharts). This should be easy to achieve because illumination only needs to be even on the relatively small charts themselves, not on the entire target. The widest (the Kodak Q-14) is 14 inches (35 cm) wide; most are under 11 inches (28 cm). Lighting uniformity is strongly affected by the distance and orientation of the lamps. Our lamps are roughly 50 inches (125 cm) from the center of the target.

Targets

Charts are mounted on 40x60 inch (100x150 cm) or 30x40 inch (75x100 cm) 1/2 inch (1.25 cm) thick foam board, which is lightweight, rigid (unlikely to warp), and easy to find at art supply shops. The images below show entire targets. Framing would normally be tighter for test images (much tighter for the Q-14, ColorChecker, and OECF targets). In some cases, the charts are mounted on neutral, middle gray (~18% reflectance) mat board, which is attached to foam board, to ensure correct exposure in cameraphones and webcams with autoexposure.

Charts are mounted using permanent double-sided tape or spray adhesive, available from hardware and art supply stores. We have used 3M 45 and 3M Photo Mount sprays, both of which are "Photo safe." 3M Super 77™ is not labeled "Photo safe," (and can't be sold in California), but it should be OK for attaching gray mat board to foam board.

New standard sharpness targets 

Two standard sharpness targets have been used in the Imatest lab starting in December 2007. Both consist of several charts printed on Epson Enhanced Matte (Ultra Premium Presentation - Matte) paper, which resolves detail as well as semigloss and glossy inkjet papers, but is much less susceptible to glare. These charts support SFR as well as all Rescharts modules (SFR, Log F-Contrast, Log Frequency, and Star Chart). They are by no means the final charts: combined charts that would include color and distortion measurement are under development.

Sharpness target (30x40 inch) for cameras up to 13 megapixels

Sharpness target (40x60 inch), for cameras over 13 megapixels
(recommended for cameras over 8 megapixels)

Notes on the charts:

• MTF measurements for the two charts were compared using a 39-megapixel medium format camera with an excellent lens. MTF was approximately 10% higher when measured on the 40x60 inch target. That is the basis of the above recommendation that the larger chart be used for cameras with over 13 megapixels.

The mirrors are difficult to see in the above images because they are reflecting a darkened room on a dark background. The mirror above the center of the 40x60 inch target is illustrated on the right. The 20:1 slanted edges are used as the standard edges for SFR sharpness measurements. They are arranged so sharpness can be measured near the image center, half-way between the center and the four corners, and near the corners. This is important because lens response may be asymmetrical in lenses that are decentered due to sloppy manufacturing. This contrast is lower than the minimum specified by the ISO standard (40:1), which is now understood to result in large errors if gamma is poorly estimated or if the tonal response is nonlinear (clips or deviates from a straight gamma curve). Much lower contrast will be recommended in the revised ISO standard. The 2:1 slanted edges measure MTF in regions where contrasty edges are absent. In such regions there may be less software sharpening; software noise reduction (lowpass filtering; the opposite of sharpening) may be employed. Signal processing that differs in different parts of the image is called nonlinear.

Mirror, showing red dot • stickers marking the horizontal target center

Aligning the camera with the target

To obtain the most accurate and consistent results from Imatest's spatial measurements (sharpness, chromatic aberration, and distortion), the camera and target must be aligned presicely. This means that

• The axis of the lens should be normal (perpendicular) to the surface of the target,
• The edges of the image should be parallel with the edges of the charts in the target,
• In most cases, the center of the image corresponds to the center of the target.

Accurate alignment is easy to obtain with an optical bench, but optical benches large enough for Imatest testing are extremely expensive. For this reason users need to develop a collection of tricks to facilitate alignment for different cameras, lenses, or focal lengths.

Leveling (horizontal, vertical (front/back))

Level the target horizontally (illustrated above) and vertically (front/back) (illustrated on the right). Then mount a spirit (bubble) level on the camera (shown below) and level the camera horizontally and vertically. This gets the leveling very close to the final setting. The "Adorama Double Bubble Level" (sounds like chewing gum) costs about $29; identical levels from Cullmann, Hasselblad, or Manfrotto, all of which fit in the accessory shoe, cost about $33 at B&H. (They're probably all made in the same factory in China.)

Rope trick (for horizontal positioning) This trick uses rope and mirrors, as well as the red dots • tht mark the horizontal center of the target (or whatever marker you choose). This is as close to smoke and mirrors as Imatest gets. The cord should be about 1 meter long and have a weight on one end. A heavy nut— the type that mates with a bolt, not the type that grows on trees— serves well. Hold the cord in front of the camera (between the camera and mirror), and align your eye and the cord so the so the cord passes in front of the horizontal center markers (red dots • in the images), and so the reflection of the cord passes directly behind the cord. The camera is positioned properly if the center of the lens is aligned with the cord. If it isn't, shift the camera horizontally until it is. To complete the horizontal adjustment, rotate (pan) the tripod pan/tilt head so the chart is centered in the image frame. At this point you should be close enough to correct alignments so the final, fine adjustments should proceed quickly.

Other targets

The images below illustrate a problem that can arise using wide angle lenses. Glare appears in the outer patches, but it doesn't affect the critical outer edges. It's difficult to remove glare entirely: it can be impractical to place lights further from the target, and reducing the angle may make lighting too uneven. We have recently been dealing with glare by printing charts on high quality matte paper (Epson Ultra Premium Presentation - Matte), which resolves detail as fine as glossy and semigloss papers. Glare is rarely a problem with normal and telephoto lenses.

Modified ISO 12233 (QA-77) and slanted-edge charts

The QA-77 modified ISO chart, available from Applied Image, is printed on photographic media with higher resolution than is attainable with inkjet printers. It can be used with smaller image fields than inkjet-printed charts: valuable for close-up work and for analyzing telephoto lenses when space is limited. Note that the glare in the corner targets (no longer used) is doesn't affect the outer-most tangential edges. Careful positioning of lights is critical when working with wide-angle lenses.

Kodak Q-14 Step chart and GretagMacbeth™ ColorChecker^®

These charts (normally framed much closer) are analyzed with Imatest Stepchart, Colorcheck, and Multicharts. They are mounted on neutral middle gray (~18% reflectance) mat board, which is mounted on foam board for rigidity. Shown in front of the wider slanted-edge target. The Q-13/Q-14 can be mounted vertically to reduce glare when used with wide angle lenses.

OECF target, available from Applied Image

The OECF target is valuable for measuring tonal response in cameras that have significant light falloff (vignetting)— especially for cameraphones, which are constrained to be extremely thin.

Distortion target, printed on an Epson 9800 inkjet
using a file generated by Imatest Test Charts

Additional targets:  The veiling glare target is illustrated here.