Evaluating the Robustness of ML Models to Out-of-Distribution Data Through Similarity Analysis

• In Machine Learning systems, several factors impact the performance of a trained model. The most important ones include model architecture, the amount of training time, the dataset size and diversity. We present a method for analyzing datasets from a use-case scenario perspective, detecting and quantifying out-of-distribution (OOD) data on dataset level. Our main contribution is the novel use of similarity metrics for the evaluation of the robustness of a model by introducing relative Fréchet Inception Distance (FID) and relative Kernel Inception Distance (KID) measures. These relative measures are relative to a baseline in-distribution dataset and are used to estimate how the model will perform on OOD data (i.e. estimate the model accuracy drop). We find a correlation between our proposed relative FID/relative KID measure and the drop in Average Precision (AP) accuracy on unseen data.

Subject headings

NATURVETENSKAP  -- Data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Datavetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Computer Sciences (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Datorseende och robotik (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Computer Vision and Robotics (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Bioinformatik (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Bioinformatics (hsv//eng)

Keyword

accuracy estimation

datasets

neural networks

similarity metrics

Learning systems

Dataset

Distance measure

Frechet

Machine learning systems

Modeling architecture

Neural-networks

Performance

Similarity analysis

Drops

Publication and Content Type

ref (subject category)

kon (subject category)