| 1. |
- Kjær Høier, Rasmus, 1987, et al.
(författare)
-
Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons
- 2023
-
Ingår i: Proceedings of Machine Learning Research. - 2640-3498. ; 202, s. 13141-13156
-
Konferensbidrag (refereegranskat)abstract
- Activity difference based learning algorithms-such as contrastive Hebbian learning and equilibrium propagation-have been proposed as biologically plausible alternatives to error backpropagation. However, on traditional digital chips these algorithms suffer from having to solve a costly inference problem twice, making these approaches more than two orders of magnitude slower than back-propagation. In the analog realm equilibrium propagation may be promising for fast and energy efficient learning, but states still need to be inferred and stored twice. Inspired by lifted neural networks and compartmental neuron models we propose a simple energy based compartmental neuron model, termed dual propagation, in which each neuron is a dyad with two intrinsic states. At inference time these intrinsic states encode the error/activity duality through their difference and their mean respectively. The advantage of this method is that only a single inference phase is needed and that inference can be solved in layerwise closed-form. Experimentally we show on common computer vision datasets, including Imagenet32x32, that dual propagation performs equivalently to back-propagation both in terms of accuracy and runtime.
|
|
| 2. |
- Kjær Høier, Rasmus, 1987, et al.
(författare)
-
Lifted Regression/Reconstruction Networks
- 2020
-
Ingår i: 31st British Machine Vision Conference, BMVC 2020.
-
Konferensbidrag (refereegranskat)abstract
- In this work we propose lifted regression/reconstruction networks(LRRNs), which combine lifted neural networks with a guaranteed Lipschitz continuity property for the output layer. Lifted neural networks explicitly optimize an energy model to infer the unit activations and therefore—in contrast to standard feed-forward neural networks—allow bidirectional feedback between layers. So far lifted neural networks have been modelled around standard feed-forward architectures. We propose to take further advantage of the feedback property by letting the layers simultaneously perform regression and reconstruction. The resulting lifted network architecture allows to control the desired amount of Lipschitz continuity, which is an important feature to obtain adversarially robust regression and classification methods. We analyse and numerically demonstrate applications for unsupervised and supervised learning
|
|
| 3. |
- Le, Huu, 1988, et al.
(författare)
-
AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks
- 2022
-
Ingår i: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. - 1063-6919. ; 2022-June, s. 460-469
-
Konferensbidrag (refereegranskat)abstract
- We propose a new algorithm for training deep neural networks (DNNs) with binary weights. In particular, we first cast the problem of training binary neural networks (BiNNs) as a bilevel optimization instance and subsequently construct flexible relaxations of this bilevel program. The resulting training method shares its algorithmic simplicity with several existing approaches to train BiNNs, in particular with the straight-through gradient estimator successfully employed in BinaryConnect and subsequent methods. In fact, our proposed method can be interpreted as an adaptive variant of the original straight-through estimator that conditionally (but not always) acts like a linear mapping in the backward pass of error propagation. Experimental results demonstrate that our new algorithm offers favorable performance compared to existing approaches.
|
|
| 4. |
- Meirose, Bernhard, et al.
(författare)
-
Real-time accelerator diagnostic tools for the max iv storage rings
- 2020
-
Ingår i: Instruments. - : MDPI AG. - 2410-390X. ; 4:3
-
Tidskriftsartikel (refereegranskat)abstract
- In this paper, beam diagnostic and monitoring tools developed by the MAX IV Operations Group are discussed. In particular, beam position monitoring and accelerator tunes visualization software tools, as well as tools that directly influence the beam quality and stability, are introduced. An availability and downtime monitoring application is also presented.
|
|
