Interpreting Video Features : A Comparison of 3D Convolutional Networks and Convolutional LSTM Networks

• A number of techniques for interpretability have been presented for deep learning in computer vision, typically with the goal of understanding what the networks have based their classification on. However, interpretability for deep video architectures is still in its infancy and we do not yet have a clear concept of how to decode spatiotemporal features. In this paper, we present a study comparing how 3D convolutional networks and convolutional LSTM networks learn features across temporally dependent frames. This is the first comparison of two video models that both convolve to learn spatial features but have principally different methods of modeling time. Additionally, we extend the concept of meaningful perturbation introduced by [1] to the temporal dimension, to identify the temporal part of a sequence most meaningful to the network for a classification decision. Our findings indicate that the 3D convolutional model concentrates on shorter events in the input sequence, and places its spatial focus on fewer, contiguous areas. 

Ämnesord

NATURVETENSKAP  -- Data- och informationsvetenskap -- Datorseende och robotik (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Computer Vision and Robotics (hsv//eng)

Nyckelord

3D modeling

Computer vision

Convolution

Convolutional neural networks

Deep learning

Classification decision

Convolutional model

Convolutional networks

Interpretability

Spatial features

Spatio temporal features

Temporal dimensions

Video architecture

Long short-term memory

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)