Reliable Confidence Predictions Using Conformal Prediction

• Conformal classiers output condence prediction regions, i.e., multi-valued predictions that are guaranteed to contain the true output value of each test pattern with some predened probability. In order to fully utilize the predictions provided by a conformal classier, it is essential that those predictions are reliable, i.e., that a user is able to assess the quality of the predictions made. Although conformal classiers are statistically valid by default, the error probability of the prediction regions output are dependent on their size in such a way that smaller, and thus potentially more interesting, predictions are more likely to be incorrect. This paper proposes, and evaluates, a method for producing rened error probability estimates of prediction regions, that takes their size into account. The end result is a binary conformal condence predictor that is able to provide accurate error probability estimates for those prediction regions containing only a single class label.

Subject headings

Natural Sciences  (hsv)

Computer and Information Sciences  (hsv)

Computer Sciences  (hsv)

Naturvetenskap  (hsv)

Data- och informationsvetenskap  (hsv)

Datavetenskap (datalogi)  (hsv)

Business and IT  (hj)

Handel och IT  (hj)