Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness

Beyond exploding and vanishing gradients : analysing RNN training using attractors and smoothness

Ribeiro, Antonio H. (author): Univ Fed Minas Gerais, Belo Horizonte, MG, Brazil

Aguirre, Luis A. (author): Univ Fed Minas Gerais, Belo Horizonte, MG, Brazil

Schön, Thomas B., Professor, 1977- (author): Uppsala universitet,Avdelningen för systemteknik,Artificiell intelligens

(creator_code:org_t)

2020
2020
English.
In: Proceedings of the 23<sup>rd</sup> International Conference on Artificial Intelligence and Statistics (AISTATS). ; , s. 2370-2380

Related links:: http://proceedings.m...; show more...; https://urn.kb.se/re...; show less...

Abstract Subject headings

The exploding and vanishing gradient problem has been the major conceptual principle behind most architecture and training improvements in recurrent neural networks (RNNs) during the last decade. In this paper, we argue that this principle, while powerful, might need some refinement to explain recent developments. We refine the concept of exploding gradients by reformulating the problem in terms of the cost function smoothness, which gives insight into higher-order derivatives and the existence of regions with many close local minima. We also clarify the distinction between vanishing gradients and the need for the RNN to learn attractors to fully use its expressive power. Through the lens of these refinements, we shed new light on recent developments in the RNN field, namely stable RNN and unitary (or orthogonal) RNNs.

By the author/editor: Ribeiro, Antonio ...; Tiels, Koen; Aguirre, Luis A.; Schön, Thomas B. ...

About the subject

ENGINEERING AND TECHNOLOGY: ENGINEERING AND ...; and Electrical Engin ...; and Embedded Systems

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.