A toolkit of metrics to aid object recognition, machine vision, and artificial intelligence systems
The paper has been updated to remove some obsolete sections and include recent developments (like the InfoDR chart and Object Detection Error Probability).
Read the full updated paper (PDF) (original paper for historical interest)
Author: Norman L. Koren (Imatest LLC)
Presented January 22, 2024 at EI2024, the IS&T Electronic Imaging Symposium, as the keynote of the joint Autonomous Vehicles & Machines (AVM) and Image Quality and Systems Performance (IQSP) session.
Abstract
There is increasing evidence that standard sharpness (MTF) and noise measurements correlate poorly with Machine Vision and Artificial Intelligence (MV/AI) system performance. This is not surprising because MV/AI algorithms operate on information rather than pixels.
We describe new techniques for measuring noise in the presence of slanted edge signals that enable the calculation of the key metric from information theory, information capacity, as well as several additional metrics.
We expect information capacity to be a strong predictor of MV/AI system performance, and because it is relatively unaffected by uniform image processing, it is the best metric for selecting (i.e., qualifying) cameras. Choosing a camera with the minimum number of pixels for the required information capacity should result in the fastest calculations and least power consumption.
The most important of the additional metrics, SNRi and Edge SNRi, measure the quality of object and edge detection, which can be enhanced by image processing. We show how to design filters that optimize object and edge detection, and we discuss the tradeoffs in applying them to real world scenarios. Finally, we discuss the mathematical framework that ties the new metrics together, resulting in a powerful and versatile toolkit of measurements.
