Sökning: onr:"swepub:oai:DiVA.org:kth-287769" >
SMALLER GENERALIZAT...
-
Kammonen, Aku,1984-KTH,Numerisk analys, NA
(författare)
SMALLER GENERALIZATION ERROR DERIVED FOR DEEP COMPARED TO SHALLOW RESIDUAL NEURAL NETWORKS
Förlag, utgivningsår, omfång ...
Nummerbeteckningar
-
LIBRIS-ID:oai:DiVA.org:kth-287769
-
https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-287769URI
Kompletterande språkuppgifter
-
Språk:engelska
-
Sammanfattning på:engelska
Ingår i deldatabas
Klassifikation
-
Ämneskategori:vet swepub-contenttype
-
Ämneskategori:ovr swepub-publicationtype
Anmärkningar
-
QC 20201221
-
Estimates of the generalization error are proved for a residual neural network with $L$ random Fourier features layers $\bar z_{\ell+1}=\bar z_\ell + \mathrm{Re}\sum_{k=1}^K\bar b_{\ell k}e^{\mathrm{i}\omega_{\ell k}\bar z_\ell}+\mathrm{Re}\sum_{k=1}^K\bar c_{\ell k}e^{\mathrm{i}\omega'_{\ell k}\cdot x}$. An optimal distribution for the frequencies $(\omega_{\ell k},\omega'_{\ell k})$ of the random Fourier features $e^{\mathrm{i}\omega_{\ell k}\bar z_\ell}$ and $e^{\mathrm{i}\omega'_{\ell k}\cdot x}$ is derived. This derivation is based on the corresponding generalization error for the approximation of the function values $f(x)$. The generalization error turns out to be smaller than the estimate ${\|\hat f\|^2_{L^1(\mathbb{R}^d)}}/{(LK)}$ of the generalization error for random Fourier features with one hidden layer and the same total number of nodes $LK$, in the case the $L^\infty$-norm of $f$ is much less than the $L^1$-norm of its Fourier transform $\hat f$. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network that shows promising results.
Ämnesord och genrebeteckningar
Biuppslag (personer, institutioner, konferenser, titlar ...)
-
Kiessling, JonasKTH,Numerisk analys, NA
(författare)
-
Petr, Plecháč
(författare)
-
Sandberg, MattiasKTH,Numerisk analys, NA(Swepub:kth)u17r6d2m
(författare)
-
Szepessy, Anders,1960-KTH,Numerisk analys, NA(Swepub:kth)u1mrbma3
(författare)
-
Tempone, Raúl
(författare)
-
KTHNumerisk analys, NA
(creator_code:org_t)
Internetlänk