The paper has been updated to remove some obsolete sections and include recent developments (like the InfoDR chart and Object Detection Error Probability).

Read the full updated paper (PDF)   (original paper for historical interest)

Author: Norman L. Koren (Imatest LLC)

Presented January 22, 2024 at EI2024, the IS&T Electronic Imaging Symposium, as the keynote of the joint Autonomous Vehicles & Machines (AVM) and Image Quality and Systems Performance (IQSP) session.

Abstract

There is increasing evidence that standard sharpness (MTF) and noise measurements correlate poorly with Machine Vision and Artificial Intelligence (MV/AI) system performance. This is not surprising because MV/AI algorithms operate on information rather than pixels.

We describe new techniques for measuring noise in the presence of slanted edge signals that enable the calculation of the key metric from information theory, information capacity, as well as several additional metrics.

We expect information capacity to be a strong predictor of MV/AI system performance, and because it is relatively unaffected by uniform image processing, it is the best metric for selecting (i.e., qualifying) cameras. Choosing a camera with the minimum number of pixels for the required information capacity should result in the fastest calculations and least power consumption.

The most important of the additional metrics, SNRi and Edge SNRi, measure the quality of object and edge detection, which can be enhanced by image processing. We show how to design filters that optimize object and edge detection, and we discuss the tradeoffs in applying them to real world scenarios. Finally, we discuss the mathematical framework that ties the new metrics together, resulting in a powerful and versatile toolkit of measurements.

The post Image information metrics from slanted edges appeared first on Imatest.

Authors: Dara Molloy, Patrick Müller, Brian Deegan, Darragh Mullins, Jonathan Horgan, Enda Ward, Edward Jones, Alexander Braun, and Martin Glavin (University of Galway, Hochschule Düsseldorf, and Valeo).

Published in IEEE Access, vol. 12, pp. 3554–3569, 2024. doi:10.1109/ACCESS.2023.3348663

Summary

Camera-based object detection underpins ADAS and autonomous driving, and how sharp the camera has to be is a question with direct safety consequences. This study measures that relationship rather than assuming it. Six current object detection models were run against a purpose-built raw dataset of pedestrians and cars at distances out to 100 m, captured across a range of lens defocus levels, and the minimum spatial frequency response (SFR) needed for reliable detection was derived for each case.

The headline result is that the relationship between detection performance and lens blur is considerably more complex than earlier work suggested. Real lenses do not defocus uniformly: field curvature, chromatic aberration, and astigmatism all shape how blur varies across the field, so a single global sharpness figure does not describe what the detector actually sees. The authors also report that smaller objects are disproportionately affected by blur, and that the detection models themselves differ markedly in how robust they are to it.

Why it matters for camera testing

Two of these findings line up closely with work we have been pursuing. The first is that a single summary sharpness number is not sufficient to predict machine vision performance. That is the same conclusion we reached comparing MTF50 against detection confidence in Validating Information Metrics Correlation with Object Detection. The second is that blur has to be characterized across the field rather than at a single point, because the aberrations that dominate off-axis are exactly the ones that move the detection result.

The overlap is not only thematic. Several of the authors (Brian Deegan, Dara Molloy, Enda Ward, Edward Jones, and Martin Glavin) are part of the University of Galway group we have worked with on Information Capacity as a Predictor of Perception Performance, which approaches the same question from the information-theoretic side.

See also

• Analysis of the Impact of Lens Blur on Safety-Critical Automotive Object Detection (IEEE Access, open access)
• Information Capacity as a Predictor of Perception Performance
• Validating Information Metrics Correlation with Object Detection
• Image Information Metrics
• Sharpness: What is it and how is it measured?

The post Analysis of the Impact of Lens Blur on Safety-Critical Automotive Object Detection appeared first on Imatest.