| 1. |
- Horta Ribeiro, Antônio, et al.
(författare)
-
Regularization properties of adversarially-trained linear regression
- 2023
-
Ingår i: Advances in Neural Information Processing Systems 36 (NeurIPS 2023). - : Curran Associates, Inc.. ; , s. 23658-23670
-
Konferensbidrag (refereegranskat)abstract
- State-of-the-art machine learning models can be vulnerable to very small input perturbations that are adversarially constructed. Adversarial training is an effective approach to defend against it. Formulated as a min-max problem, it searches for the best solution when the training data were corrupted by the worst-case attacks. Linear models are among the simple models where vulnerabilities can be observed and are the focus of our study. In this case, adversarial training leads to a convex optimization problem which can be formulated as the minimization of a finite sum. We provide a comparative analysis between the solution of adversarial training in linear regression and other regularization methods. Our main findings are that: (A) Adversarial training yields the minimum-norm interpolating solution in the overparameterized regime (more parameters than data), as long as the maximum disturbance radius is smaller than a threshold. And, conversely, the minimum-norm interpolator is the solution to adversarial training with a given radius. (B) Adversarial training can be equivalent to parameter shrinking methods (ridge regression and Lasso). This happens in the underparametrized region, for an appropriate choice of adversarial radius and zero-mean symmetrically distributed covariates. (C) For l(infinity)-adversarial training-as in square-root Lasso-the choice of adversarial radius for optimal bounds does not depend on the additive noise variance. We confirm our theoretical findings with numerical examples.
|
|
| 2. |
- Hult, Ludvig
(författare)
-
Robust inference for systems under distribution shifts
- 2024
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- We use statistics and machine learning to make advanced inferences from data. Challenges may arise, invalidating inferences, if the context changes. Situations where the data generating process changes from one context to another is known as distribution shift, and may arise for several reasons. This thesis presents five articles on the topic of making robust inferences in the presence of distribution shifts.Paper 1 to 3 develop mathematical methods for robust inference. Paper 1 adresses the problem that when there is uncertainty about the structue of the underlying data generating process, confidence intervals are not generally valid for estimating the impact of interventions. We propose a method for constructing valid confidence intervals for the average treatment effect using linear structural causal models. Paper 2 addresses the problem of model evaluation under distribution shift, using nonparametric statistics. We show that with a small validation sample, one can make finite-samplevalid inference about a machine learning model performance on a new data set despite distribution shift. Paper 3 addresses the problem that inventory control policies may become invalid without assumptions on the demand. Using a deterministic feedback mechanism, we construct an order policy that guarantees any prescribed service level, with weak assumptions on the demand, allowing distribution shift.Paper 4 and 5 focus on applications to neurocritical care data. Paper 4 uses machine learning to predict intracranial pressure insults in neurocritical care. Since distribution shift may occur between patients and/or years, the validation methods takes this into account. Paper 5 explores the use of causal inference on neurointensive care data. While this may eventually lead to inferences valid under intervention distribution shift, several obstacles to effective application are identified and discussed.
|
|
| 3. |
|
|
| 4. |
- Osama, Muhammad, et al.
(författare)
-
Inferring Heterogeneous Causal Effects in Presence of Spatial Confounding
- 2019
-
Ingår i: Proceedings of the 36th International Conference on Machine Learning. ; , s. 4942-4950
-
Konferensbidrag (refereegranskat)abstract
- We address the problem of inferring the causal effect of an exposure on an outcome across space, using observational data. The data is possibly subject to unmeasured confounding variables which, in a standard approach, must be adjusted for by estimating a nuisance function. Here we develop a method that eliminates the nuisance function, while mitigating the resulting errors-in-variables. The result is a robust and accurate inference method for spatially varying heterogeneous causal effects. The properties of the method are demonstrated on synthetic as well as real data from Germany and the US.
|
|
| 5. |
- Osama, Muhammad, et al.
(författare)
-
Online Learning for Prediction via Covariance Fitting : Computation, Performance and Robustness
- 2023
-
Ingår i: Transactions on Machine Learning Research. - : Transactions on Machine Learning Research. - 2835-8856.
-
Tidskriftsartikel (refereegranskat)abstract
- We consider the online learning of linear smoother predictors based on a covariance model of the outcomes. To control its degrees of freedom in an appropriate manner, the covariance model parameters are often learned using cross-validation or maximum-likelihood techniques. However, neither technique is suitable when training data arrives in a streaming fashion. Here we consider a covariance-fitting method to learn the model parameters, initially used in spectral estimation. We show that this results in a computation efficient online learning method in which the resulting predictor can be updated sequentially. We prove that, with high probability, its out-of-sample error approaches the minimum achievable level at root-$n$ rate. Moreover, we show that the resulting predictor enjoys two different robustness properties. First, it minimizes the out-of-sample error with respect to the least favourable distribution within a given Wasserstein distance from the empirical distribution. Second, it is robust against errors in the covariate training data. We illustrate the performance of the proposed method in a numerical experiment.
|
|
| 6. |
- Osama, Muhammad
(författare)
-
Robust machine learning methods
- 2022
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- We are surrounded by data in our daily lives. The rent of our houses, the amount of electricity units consumed, the prices of different products at a supermarket, the daily temperature, our medicine prescriptions, our internet search history are all different forms of data. Data can be used in a wide range of applications. For example, one can use data to predict product prices in the future; to predict tomorrow's temperature; to recommend videos; or suggest better prescriptions. However in order to do the above, one is required to learn a model from data. A model is a mathematical description of how the phenomena we are interested in behaves e.g. how does the temperature vary? Is it periodic? What kinds of patterns does it have? Machine learning is about this process of learning models from data by building on disciplines such as statistics and optimization. Learning models comes with many different challenges. Some challenges are related to how flexible the model is, some are related to the size of data, some are related to computational efficiency etc. One of the challenges is that of data outliers. For instance, due to war in a country exports could stop and there could be a sudden spike in prices of different products. This sudden jump in prices is an outlier or corruption to the normal situation and must be accounted for when learning the model. Another challenge could be that data is collected in one situation but the model is to be used in another situation. For example, one might have data on vaccine trials where the participants were mostly old people. But one might want to make a decision on whether to use the vaccine or not for the whole population that contains people of all age groups. So one must also account for this difference when learning models because the conclusion drawn may not be valid for the young people in the population. Yet another challenge could arise when data is collected from different sources or contexts. For example, a shopkeeper might have data on sales of paracetamol when there was flu and when there was no flu and she might want to decide how much paracetamol to stock for the next month. In this situation, it is difficult to know whether there will be a flu next month or not and so deciding on how much to stock is a challenge. This thesis tries to address these and other similar challenges.In paper I, we address the challenge of data corruption i.e., learning models in a robust way when some fraction of the data is corrupted. In paper II, we apply the methodology of paper I to the problem of localization in wireless networks. Paper III addresses the challenge of estimating causal effect between an exposure and an outcome variable from spatially collected data (e.g. whether increasing number of police personnel in an area reduces number of crimes there). Paper IV addresses the challenge of learning improved decision policies e.g. which treatment to assign to which patient given past data on treatment assignments. In paper V, we look at the challenge of learning models when data is acquired from different contexts and the future context is unknown. In paper VI, we address the challenge of predicting count data across space e.g. number of crimes in an area and quantify its uncertainty. In paper VII, we address the challenge of learning models when data points arrive in a streaming fashion i.e., point by point. The proposed method enables online training and also yields some robustness properties.
|
|
| 7. |
- Svensson, Andreas, et al.
(författare)
-
How consistent is my model with the data? : Information-theoretic model check
- 2018
-
Konferensbidrag (refereegranskat)abstract
- The choice of model class is fundamental in statistical learning and system identification, no matter whether the class is derived from physical principles or is a generic black-box. We develop a method to evaluate the specified model class by assessing its capability of reproducing data that is similar to the observed data record. This model check is based on the information-theoretic properties of models viewed as data generators and is applicable to e.g. sequential data and nonlinear dynamical models. The method can be understood as a specific two-sided posterior predictive test. We apply the information-theoretic model check to both synthetic and real data and compare it with a classical whiteness test.
|
|
| 8. |
|
|
| 9. |
- Wågberg, Johan, et al.
(författare)
-
Prediction Performance After Learning in Gaussian Process Regression
- 2017
-
Ingår i: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. - Brookline : PMLR. ; , s. 1264-1272
-
Konferensbidrag (refereegranskat)abstract
- This paper considers the quantification of the prediction performance in Gaussian process regression. The standard approach is to base the prediction error bars on the theoretical predictive variance, which is a lower bound on the mean square-error (MSE). This approach, however, does not take into account that the statistical model is learned from the data. We show that this omission leads to a systematic underestimation of the prediction errors. Starting from a generalization of the Cramér-Rao bound, we derive a more accurate MSE bound which provides a measure of uncertainty for prediction of Gaussian processes. The improved bound is easily computed and we illustrate it using synthetic and real data examples.
|
|
| 10. |
- Wågberg, Johan, et al.
(författare)
-
Regularized parametric system identification : a decision-theoretic formulation
- 2018
-
Ingår i: 2018 Annual American Control Conference (ACC). - : IEEE. - 9781538654286 - 9781538654279 - 9781538654293 ; , s. 1895-1900
-
Konferensbidrag (refereegranskat)abstract
- Parametric prediction error methods constitute a classical approach to the identification of linear dynamic systems with excellent large-sample properties. A more recent regularized approach, inspired by machine learning and Bayesian methods, has also gained attention. Methods based on this approach estimate the system impulse response with excellent small-sample properties. In several applications, however, it is desirable to obtain a compact representation of the system in the form of a parametric model. By viewing the identification of such models as a decision, we develop a decision-theoretic formulation of the parametric system identification problem that bridges the gap between the classical and regularized approaches above. Using the output-error model class as an illustration, we show that this decision-theoretic approach leads to a regularized method that is robust to small sample-sizes as well as overparameterization.
|
|
