Recurrent knowledge distillation

• Knowledge distillation compacts deep networks by letting a small student network learn from a large teacher network. The accuracy of knowledge distillation recently benefited from adding residual layers. We propose to reduce the size of the student network even further by recasting multiple residual layers in the teacher network into a single recurrent student layer. We propose three variants of adding recurrent connections into the student network, and show experimentally on CIFAR-10, Scenes and MiniPlaces, that we can reduce the number of parameters at little loss in accuracy.

Ämnesord

TEKNIK OCH TEKNOLOGIER  -- Elektroteknik och elektronik -- Kommunikationssystem (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Electrical Engineering, Electronic Engineering, Information Engineering -- Communication Systems (hsv//eng)

Nyckelord

Knowledge distillation

compacting deep representations for image classification

recurrent layers

NG J

2009

PROC CVPR IEEE

P248 J

2014

NIPS

P2654 X

2017

CVPR

P7370 ringenberg J T

2013

CoRR

ng Yunhe

2017

ICML

P3703 izhevsky Alex

2009

Learning multiple layers of features from tiny images

n M

2013

ICLR

m J

2017

CVPR

P4133 goruyko Sergey

2017

ICLR

eff K

2016

ICLR

odfellow I J

2013

JMLR

elhamer Evan

2017

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

V39

P640 bedev Vadim

2016

2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

JUN 27-30

2016

Seattle

WA

P2554 pta Saurabh

2016

2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

JUN 27-30

2016

Seattle

WA

P2827 stegari Mohammad

2016

COMPUTER VISION - ECCV 2016

PT IV14th European Conference on Computer Vision (ECCV)

OCT 08-16

2016

Amsterdam

NETHERLANDS

V9908

P525 ang Xiangyu

2016

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

V38

P1943 Kaiming

2015

2015 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)IEEE International Conference on Computer Vision

DEC 11-18

2015

Santiago

CHILE

P1026 eng Yu

2015

2015 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)IEEE International Conference on Computer Vision

DEC 11-18

2015

Santiago

CHILE

P2857 ang Ming

2015

2015 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

JUN 07-12

2015

Boston

MA

P3367 chreiter S

1997

NEURAL COMPUTATION

V9

P1735 attoni Ariadna

2009

CVPR: 2009 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION

VOLS 1-4IEEE-Computer-Society Conference on Computer Vision and Pattern Recognition Workshops

JUN 20-25

2009

Miami Beach

FL

P413 nton G

2015

NIPS Deep Learning and Representation Learning Workshop

i J

2016

NIPS

P379

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)