| 1. |
- Adib Yaghmaie, Farnaz, 1987-, et al.
(författare)
-
Online Optimal Tracking of Linear Systems with Adversarial Disturbances
- 2023
-
Ingår i: Transactions on Machine Learning Research. - 2835-8856. ; :04
-
Tidskriftsartikel (refereegranskat)abstract
- This paper presents a memory-augmented control solution to the optimal reference tracking problem for linear systems subject to adversarial disturbances. We assume that the dynamics of the linear system are known and that the reference signal is generated by a linear system with unknown dynamics. Under these assumptions, finding the optimal tracking controller is formalized as an online convex optimization problem that leverages memory of past disturbance and reference values to capture their temporal effects on the performance. That is, a (disturbance, reference)-action control policy is formalized, which selects the control actions as a linear map of the past disturbance and reference values. The online convex optimization is then formulated over the parameters of the policy on its past disturbance and reference values to optimize general convex costs. It is shown that our approach outperforms robust control methods and achieves a tight regret bound O(√T) where in our regret analysis, we have benchmarked against the best linear policy.
|
|
| 2. |
|
|
| 3. |
- Bånkestad, Maria, et al.
(författare)
-
Variational Elliptical Processes
- 2023
-
Ingår i: Transactions on Machine Learning Research. - 2835-8856.
-
Tidskriftsartikel (refereegranskat)abstract
- We present elliptical processes—a family of non-parametric probabilistic models that subsumes Gaussian processes and Student's t processes. This generalization includes a range of new heavy-tailed behaviors while retaining computational tractability. Elliptical processes are based on a representation of elliptical distributions as a continuous mixture of Gaussian distributions. We parameterize this mixture distribution as a spline normalizing flow, which we train using variational inference. The proposed form of the variational posterior enables a sparse variational elliptical process applicable to large-scale problems. We highlight advantages compared to Gaussian processes through regression and classification experiments. Elliptical processes can supersede Gaussian processes in several settings, including cases where the likelihood is non-Gaussian or when accurate tail modeling is essential.
|
|
| 4. |
- Bökman, Georg, et al.
(författare)
-
In search of projectively equivariant networks
- 2023
-
Ingår i: Transactions on Machine Learning Research. - 2835-8856.
-
Tidskriftsartikel (refereegranskat)abstract
- Equivariance of linear neural network layers is well studied. In this work, we relax the equivariance condition to only be true in a projective sense. Hereby, we introduce the topic of projective equivariance to the machine learning audience. We theoretically study the relation of projectively and linearly equivariant linear layers. We find that in some important cases, surprisingly, the two types of layers coincide. We also propose a way to construct a projectively equivariant neural network, which boils down to building a standard equivariant network where the linear group representations acting on each intermediate feature space are lifts of projective group representations. Projective equivariance is showcased in two simple experiments. Code for the experiments is provided in the supplementary material.
|
|
| 5. |
- Coelho Mollo, Dimitri, et al.
(författare)
-
Beyond the Imitation Game : Quantifying and extrapolating the capabilities of language models
- 2023
-
Ingår i: Transactions on Machine Learning Research. ; :5
-
Tidskriftsartikel (refereegranskat)abstract
- Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 442 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.
|
|
| 6. |
|
|
| 7. |
- Englesson, Erik, et al.
(författare)
-
Logistic-Normal Likelihoods for Heteroscedastic Label Noise
- 2023
-
Ingår i: Transactions on Machine Learning Research. - : Transactions on Machine Learning Research (TMLR). - 2835-8856. ; 8
-
Tidskriftsartikel (refereegranskat)abstract
- A natural way of estimating heteroscedastic label noise in regression is to model the observed (potentially noisy) target as a sample from a normal distribution, whose parameters can be learned by minimizing the negative log-likelihood. This formulation has desirable loss attenuation properties, as it reduces the contribution of high-error examples. Intuitively, this behavior can improve robustness against label noise by reducing overfitting. We propose an extension of this simple and probabilistic approach to classification that has the same desirable loss attenuation properties. Furthermore, we discuss and address some practical challenges of this extension. We evaluate the effectiveness of the method by measuring its robustness against label noise in classification. We perform enlightening experiments exploring the inner workings of the method, including sensitivity to hyperparameters, ablation studies, and other insightful analyses.
|
|
| 8. |
- Fay, Dominik, et al.
(författare)
-
Adaptive Hyperparameter Selection for Differentially Private Gradient Descent
- 2023
-
Ingår i: Transactions on Machine Learning Research. - 2835-8856.
-
Tidskriftsartikel (refereegranskat)abstract
- We present an adaptive mechanism for hyperparameter selection in differentially private optimization that addresses the inherent trade-off between utility and privacy. The mechanism eliminates the often unstructured and time-consuming manual effort of selecting hyperparameters and avoids the additional privacy costs that hyperparameter selection otherwise incurs on top of that of the actual algorithm.We instantiate our mechanism for noisy gradient descent on non-convex, convex and strongly convex loss functions, respectively, to derive schedules for the noise variance and step size. These schedules account for the properties of the loss function and adapt to convergence metrics such as the gradient norm. When using these schedules, we show that noisy gradient descent converges at essentially the same rate as its noise-free counterpart. Numerical experiments show that the schedules consistently perform well across a range of datasets without manual tuning.
|
|
| 9. |
- Gamba, Matteo, et al.
(författare)
-
Deep Double Descent via Smooth Interpolation
- 2023
-
Ingår i: Transactions on Machine Learning Research. - : Transactions on Machine Learning Research (TMLR). - 2835-8856. ; 4
-
Tidskriftsartikel (refereegranskat)abstract
- The ability of overparameterized deep networks to interpolate noisy data, while at the same time showing good generalization performance, has been recently characterized in terms of the double descent curve for the test error. Common intuition from polynomial regression suggests that overparameterized networks are able to sharply interpolate noisy data, without considerably deviating from the ground-truth signal, thus preserving generalization ability. At present, a precise characterization of the relationship between interpolation and generalization for deep networks is missing. In this work, we quantify sharpness of fit of the training data interpolated by neural network functions, by studying the loss landscape w.r.t. to the input variable locally to each training point, over volumes around cleanly- and noisily-labelled training samples, as we systematically increase the number of model parameters and training epochs. Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy labels. While small interpolating models sharply fit both clean and noisy data, large interpolating models express a smooth loss landscape, where noisy targets are predicted over large volumes around training data points, in contrast to existing intuition.
|
|
| 10. |
|
|
