| 1. |
- Alistarh, Dan, et al.
(författare)
-
The Convergence of Sparsified Gradient Methods
- 2018
-
Ingår i: Advances in Neural Information Processing Systems 31 (NIPS 2018). - : Neural Information Processing Systems (NIPS).
-
Konferensbidrag (refereegranskat)abstract
- Stochastic Gradient Descent (SGD) has become the standard tool for distributed training of massive machine learning models, in particular deep neural networks. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed to reduce the overheads of distribution. To date, gradient sparsification methods-where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally-are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to three orders of magnitude, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis also reveals that these methods do require analytical conditions to converge well, justifying and complementing existing heuristics.
|
|
| 2. |
- Halloran, John T., et al.
(författare)
-
Speeding Up Percolator
- 2019
-
Ingår i: Journal of Proteome Research. - : AMER CHEMICAL SOC. - 1535-3893 .- 1535-3907. ; 18:9, s. 3353-3359
-
Tidskriftsartikel (refereegranskat)abstract
- The processing of peptide tandem mass spectrometry data involves matching observed spectra against a sequence database. The ranking and calibration of these peptide-spectrum matches can be improved substantially using a machine learning postprocessor. Here, we describe our efforts to speed up one widely used postprocessor, Percolator. The improved software is dramatically faster than the previous version of Percolator, even when using relatively few processors. We tested the new version of Percolator on a data set containing over 215 million spectra and recorded an overall reduction to 23% of the running time as compared to the unoptimized code. We also show that the memory footprint required by these speedups is modest relative to that of the original version of Percolator.
|
|
