Sökning: onr:"swepub:oai:DiVA.org:liu-203630" >
DINO as a von Mises...
DINO as a von Mises-Fisher mixture model
-
- Govindarajan, Hariprasath (författare)
- Linköpings universitet,Statistik och maskininlärning,Tekniska fakulteten
-
- Sidén, Per (författare)
- Arriver Sweden AB
-
- Roll, Jacob (författare)
- Arriver Sweden AB
-
visa fler...
-
- Lindsten, Fredrik, 1984- (författare)
- Linköpings universitet,Statistik och maskininlärning,Reglerteknik,Tekniska fakulteten
-
visa färre...
-
(creator_code:org_t)
- 2023
- 2023
- Engelska.
-
Ingår i: The Eleventh International Conference on Learning Representations.
- Relaterad länk:
-
https://openreview.n...
-
visa fler...
-
https://liu.diva-por... (primary) (Raw object)
-
https://urn.kb.se/re...
-
visa färre...
Abstract
Ämnesord
Stäng
- Self-distillation methods using Siamese networks are popular for self-supervised pre-training. DINO is one such method based on a cross-entropy loss between K-dimensional probability vectors, obtained by applying a softmax function to the dot product between representations and learnt prototypes. Given the fact that the learned representations are L2-normalized, we show that DINO and its derivatives, such as iBOT, can be interpreted as a mixture model of von Mises-Fisher components. With this interpretation, DINO assumes equal precision for all components when the prototypes are also L2-normalized. Using this insight we propose DINO-vMF, that adds appropriate normalization constants when computing the cluster assignment probabilities. Unlike DINO, DINO-vMF is stable also for the larger ViT-Base model with unnormalized prototypes. We show that the added flexibility of the mixture model is beneficial in terms of better image representations. The DINO-vMF pre-trained model consistently performs better than DINO on a range of downstream tasks. We obtain similar improvements for iBOT-vMF vs iBOT and thereby show the relevance of our proposed modification also for other methods derived from DINO.
Ämnesord
- NATURVETENSKAP -- Data- och informationsvetenskap -- Datorseende och robotik (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences -- Computer Vision and Robotics (hsv//eng)
Nyckelord
- self-supervised learning
- vision transformers
- mixture models
Publikations- och innehållstyp
- ref (ämneskategori)
- kon (ämneskategori)