Uncovering hidden reasoning of convolutional neural networks in biomedical image classification by using attribution methods

• Convolutional neural networks (CNNs) are very popular in biomedical image processing and analysis, due to their impressive performance on numerous tasks. However, the performance comes at a cost of limited interpretability, which may harm users' trust in methods and their results. Robust and trustworthy methods are particularly in demand in the medical domain due to the sensitivity of the matter. There is a limited understanding of what CNNs base their decisions on, and, in particular, how their performance is related to what they are paying attention to. In this study, we utilize popular attribution methods, with the aim to explore relations between properties of a network's attention and its accuracy and certainty in classification. An intuitive reasoning is that in order for a network to make good decisions, it has to be consistent in what to draw attention to. We take a step towards understanding CNNs' behavior by identifying a relation between the model performance and the variability of its attention map.We observe two biomedical datasets and two commonly used architectures. We train several identical models of the same architecture on the given data; these identical models differ due to stochasticity of initialization and training. We analyse the variability of the predictions from such collections of networks where we observe all the network instances and their classifications independently. We utilize Gradient-weighted Class Activation Mapping (Grad-CAM) and Layer-wise Relevance Propagation (LRP), frequently employed attribution methods, for the activation analysis. Given a collection of trained CNNs, we compute, for each image of the test set: (i) the mean and standard deviation (SD) of the accuracy, over the networks in the collection; (ii) the mean and SD of the respective attention maps. We plot these measures against each other for the different combinations of network architectures and datasets, in order to expose possible relations between them.Our results reveal that there exists a relation between the variability of accuracy for collections of identical models and the variability of corresponding attention maps and that this relation is consistent among the considered combinations of datasets and architectures. We observe that the aggregated standard deviation of attention maps has a quadratic relation to the average accuracy of the sets of models and a linear relation to the standard deviation of accuracy. Motivated by the results, we are also performing subsequent experiments to reveal the relation between the score and attention, as well as to understand the impact of different images to the prediction by using mentioned statistics for each image and clustering techniques. These constitute important steps towards improved explainability and a generally clearer picture of the decision-making process of CNNs for biomedical data.

Ämnesord

TEKNIK OCH TEKNOLOGIER  -- Medicinteknik -- Medicinsk bildbehandling (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Medical Engineering -- Medical Image Processing (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Annan data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Other Computer and Information Science (hsv//eng)

Nyckelord

Oral cancer

attribution methods

deep convolutional neural networks

Artificiell intelligens

Artificial Intelligence

Computerized Image Processing

Datoriserad bildbehandling

Publikations- och innehållstyp

vet (ämneskategori)

kon (ämneskategori)