| 1. |
- Alenlöv, Johan, et al.
(författare)
-
Pseudo-Marginal Hamiltonian Monte Carlo
- 2021
-
Ingår i: Journal of machine learning research. - : MICROTOME PUBL. - 1532-4435 .- 1533-7928. ; 22
-
Tidskriftsartikel (refereegranskat)abstract
- Bayesian inference in the presence of an intractable likelihood function is computationally challenging. When following a Markov chain Monte Carlo (MCMC) approach to approximate the posterior distribution in this context, one typically either uses MCMC schemes which target the joint posterior of the parameters and some auxiliary latent variables, or pseudo-marginal Metropolis-Hastings (MH) schemes. The latter mimic a MH algorithm targeting the marginal posterior of the parameters by approximating unbiasedly the intractable likelihood. However, in scenarios where the parameters and auxiliary variables are strongly correlated under the posterior and/or this posterior is multimodal, Gibbs sampling or Hamiltonian Monte Carlo (HMC) will perform poorly and the pseudo-marginal MH algorithm, as any other MH scheme, will be inefficient for high-dimensional parameters. We propose here an original MCMC algorithm, termed pseudo-marginal HMC, which combines the advantages of both HMC and pseudo-marginal schemes. Specifically, the PM-HMC method is controlled by a precision parameter N, controlling the approximation of the likelihood and, for any N, it samples the marginal posterior of the parameters. Additionally, as N tends to infinity, its sample trajectories and acceptance probability converge to those of an ideal, but intractable, HMC algorithm which would have access to the intractable likelihood and its gradient. We demonstrate through experiments that PM-HMC can outperform significantly both standard HMC and pseudo-marginal MH schemes.
|
|
| 2. |
- Allerbo, Oskar, 1985, et al.
(författare)
-
Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net
- 2023
-
Ingår i: Journal of Machine Learning Research. - 1533-7928 .- 1532-4435. ; 24, s. 1-35
-
Tidskriftsartikel (refereegranskat)abstract
- The elastic net combines lasso and ridge regression to fuse the sparsity property of lasso with the grouping property of ridge regression. The connections between ridge regression and gradient descent and between lasso and forward stagewise regression have previously been shown. Similar to how the elastic net generalizes lasso and ridge regression, we introduce elastic gradient descent, a generalization of gradient descent and forward stagewise regression. We theoretically analyze elastic gradient descent and compare it to the elastic net and forward stagewise regression. Parts of the analysis are based on elastic gradient flow, a piecewise analytical construction, obtained for elastic gradient descent with infinitesimal step size. We also compare elastic gradient descent to the elastic net on real and simulated data and show that it provides similar solution paths, but is several orders of magnitude faster. Compared to forward stagewise regression, elastic gradient descent selects a model that, although still sparse, provides considerably lower prediction and estimation errors.
|
|
| 3. |
- Allerbo, Oskar, 1985, et al.
(författare)
-
Non-linear, sparse dimensionality reduction via path lasso penalized autoencoders
- 2021
-
Ingår i: Journal of Machine Learning Research. - : Microtome Publishing. - 1532-4435 .- 1533-7928. ; 22
-
Tidskriftsartikel (refereegranskat)abstract
- High-dimensional data sets are often analyzed and explored via the construction of a latent low-dimensional space which enables convenient visualization and efficient predictive modeling or clustering. For complex data structures, linear dimensionality reduction techniques like PCA may not be sufficiently flexible to enable low-dimensional representation. Non-linear dimension reduction techniques, like kernel PCA and autoencoders, suffer from loss of interpretability since each latent variable is dependent of all input dimensions. To address this limitation, we here present path lasso penalized autoencoders. This structured regularization enhances interpretability by penalizing each path through the encoder from an input to a latent variable, thus restricting how many input variables are represented in each latent dimension. Our algorithm uses a group lasso penalty and non-negative matrix factorization to construct a sparse, non-linear latent representation. We compare the path lasso regularized autoencoder to PCA, sparse PCA, autoencoders and sparse autoencoders on real and simulated data sets. We show that the algorithm exhibits much lower reconstruction errors than sparse PCA and parameter-wise lasso regularized autoencoders for low-dimensional representations. Moreover, path lasso representations provide a more accurate reconstruction match, i.e. preserved relative distance between objects in the original and reconstructed spaces. ©2021 Oskar Allerbo and Rebecka Jörnsten.
|
|
| 4. |
- Avelin, Benny, 1984-, et al.
(författare)
-
Deep Limits and a Cut-Off Phenomenon for Neural Networks
- 2022
-
Ingår i: Journal of machine learning research. - : MICROTOME PUBL. - 1532-4435 .- 1533-7928. ; 23
-
Tidskriftsartikel (refereegranskat)abstract
- We consider dynamical and geometrical aspects of deep learning. For many standard choices of layer maps we display semi-invariant metrics which quantify differences between data or decision functions. This allows us, when considering random layer maps and using non-commutative ergodic theorems, to deduce that certain limits exist when letting the number of layers tend to infinity. We also examine the random initialization of standard networks where we observe a surprising cut-off phenomenon in terms of the number of layers, the depth of the network. This could be a relevant parameter when choosing an appropriate number of layers for a given learning task, or for selecting a good initialization procedure. More generally, we hope that the notions and results in this paper can provide a framework, in particular a geometric one, for a part of the theoretical understanding of deep neural networks.
|
|
| 5. |
|
|
| 6. |
- Damaschke, Peter, 1963, et al.
(författare)
-
Linear programs for hypotheses selection in probabilistic inference models
- 2006
-
Ingår i: Journal of Machine Learning Research. - 1533-7928 .- 1532-4435. ; 7, s. 1339-1355
-
Tidskriftsartikel (refereegranskat)abstract
- We consider an optimization problem in probabilistic inference: Given n hypotheses, m possible observations, their conditional probabilities, and a particular observation, select a possibly small subset of hypotheses excluding the true target only with some given error probability. After specifying the optimization goal we show that this problem can be solved through a linear program in mn variables that indicate the probabilities to discard a hypothesis given an observation. Moreover, we can compute optimal strategies where only O(m+n) of these variables get fractional values. The manageable size of the linear programs and the mostly deterministic shape of optimal strategies make the method practicable. We interpret the dual variables as worst-case distributions of hypotheses, and we point out some counterintuitive nonmonotonic behaviour of the variables as a function of the error bound. One of the open problems is the existence of a purely combinatorial algorithm that is faster than generic linear programming. (Slightly adapted from the original abstract.)
|
|
| 7. |
- Dang, Khue-Dung, et al.
(författare)
-
Hamiltonian Monte Carlo with Energy Conserving Subsampling
- 2019
-
Ingår i: Journal of machine learning research. - : MIT Press. - 1532-4435 .- 1533-7928. ; 20, s. 1-31
-
Tidskriftsartikel (refereegranskat)abstract
- Hamiltonian Monte Carlo (HMC) samples efficiently from high-dimensional posterior distributions with proposed parameter draws obtained by iterating on a discretized version of the Hamiltonian dynamics. The iterations make HMC computationally costly, especially in problems with large data sets, since it is necessary to compute posterior densities and their derivatives with respect to the parameters. Naively computing the Hamiltonian dynamics on a subset of the data causes HMC to lose its key ability to generate distant parameter proposals with high acceptance probability. The key insight in our article is that efficient subsampling HMC for the parameters is possible if both the dynamics and the acceptance probability are computed from the same data subsample in each complete HMC iteration. We show that this is possible to do in a principled way in a HMC-within-Gibbs framework where the subsample is updated using a pseudo marginal MH step and the parameters are then updated using an HMC step, based on the current subsample. We show that our subsampling methods are fast and compare favorably to two popular sampling algorithms that use gradient estimates from data subsampling. We also explore the current limitations of subsampling HMC algorithms by varying the quality of the variance reducing control variates used in the estimators of the posterior density and its gradients.
|
|
| 8. |
- Dimitrakakis, Christos, 1975, et al.
(författare)
-
Differential privacy for Bayesian inference through posterior sampling
- 2017
-
Ingår i: Journal of Machine Learning Research. - 1533-7928 .- 1532-4435. ; 18:1 March 2017
-
Tidskriftsartikel (refereegranskat)abstract
- Differential privacy formalises privacy-preserving mechanisms that provide access to a database. Can Bayesian inference be used directly to provide private access to data? The answer is yes: under certain conditions on the prior, sampling from the posterior distribution can lead to a desired level of privacy and utility. For a uniform treatment, we define differential privacy over arbitrary data set metrics, outcome spaces and distribution families. This allows us to also deal with non-i.i.d or non-tabular data sets. We then prove bounds on the sensitivity of the posterior to the data, which delivers a measure of robustness. We also show how to use posterior sampling to provide differentially private responses to queries, within a decision-theoretic framework. Finally, we provide bounds on the utility of answers to queries and on the ability of an adversary to distinguish between data sets. The latter are complemented by a novel use of Le Cam's method to obtain lower bounds on distinguishability. Our results hold for arbitrary metrics, including those for the common definition of differential privacy. For specific choices of the metric, we give a number of examples satisfying our assumptions. © 2017 C Dimitrakakis, B. Nelson, Z. Zhang, A. Mitrokotsa, B. I. P. Rubinstein.
|
|
| 9. |
- Ekdahl, Magnus, et al.
(författare)
-
Bounds for the loss in probability of correct classification under model based approximation
- 2006
-
Ingår i: Journal of machine learning research. - 1532-4435 .- 1533-7928. ; 7, s. 2449-2480
-
Tidskriftsartikel (refereegranskat)abstract
- In many pattern recognition/classification problem the true class conditional model and class probabilities are approximated for reasons of reducing complexity and/or of statistical estimation. The approximated classifier is expected to have worse performance, here measured by the probability of correct classification. We present an analysis valid in general, and easily computable formulas for estimating the degradation in probability of correct classification when compared to the optimal classifier. An example of an approximation is the Naive Bayes classifier. We show that the performance of the Naive Bayes depends on the degree of functional dependence between the features and labels. We provide a sufficient condition for zero loss of performance, too.
|
|
| 10. |
- Feyzmahdavian, Hamid Reza, et al.
(författare)
-
Asynchronous Iterations in Optimization : New Sequence Results and Sharper Algorithmic Guarantees
- 2023
-
Ingår i: Journal of machine learning research. - : MICROTOME PUBL. - 1532-4435 .- 1533-7928. ; 24
-
Tidskriftsartikel (refereegranskat)abstract
- We introduce novel convergence results for asynchronous iterations that appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply and give explicit estimates for how the degree of asynchrony impacts the convergence rates of the iterates. Our results shorten, streamline and strengthen existing convergence proofs for several asynchronous optimization methods and allow us to establish convergence guarantees for popular algorithms that were thus far lacking a complete theoretical under-standing. Specifically, we use our results to derive better iteration complexity bounds for proximal incremental aggregated gradient methods, to obtain tighter guarantees depending on the average rather than maximum delay for the asynchronous stochastic gradient descent method, to provide less conservative analyses of the speedup conditions for asynchronous block-co ordinate implementations of Krasnosel'skii-Mann iterations, and to quantify the convergence rates for totally asynchronous iterations under various assumptions on communication delays and update rates.
|
|
