Search: id:"swepub:oai:research.chalmers.se:9650ba97-db2c-4fb4-b6a4-bfccf3c8faa1" >
DSAG: A Mixed Synch...
DSAG: A Mixed Synchronous-Asynchronous Iterative Method for Straggler-Resilient Learning
-
- Severinson, Albin (author)
- Universitetet i Bergen,University of Bergen,Simula UiB
-
- Rosnes, Eirik (author)
- Simula UiB
-
Rouayheb, Salim El (author)
-
show more...
-
- Graell I Amat, Alexandre, 1976 (author)
- Chalmers tekniska högskola,Chalmers University of Technology,Simula UiB
-
show less...
-
(creator_code:org_t)
- 2023
- 2023
- English.
-
In: IEEE Transactions on Communications. - 0090-6778 .- 1558-0857. ; 71:2, s. 808-822
- Related links:
-
https://research.cha...
-
show more...
-
https://doi.org/10.1...
-
show less...
Abstract
Subject headings
Close
- We consider straggler-resilient learning. In many previous works, e.g., in the coded computing literature, straggling is modeled as random delays that are independent and identically distributed between workers. However, in many practical scenarios, a given worker may straggle over an extended period of time. We propose a latency model that captures this behavior and is substantiated by traces collected on Microsoft Azure, Amazon Web Services (AWS), and a small local cluster. Building on this model, we propose DSAG, a mixed synchronous-asynchronous iterative optimization method, based on the stochastic average gradient (SAG) method, that combines timely and stale results. We also propose a dynamic load-balancing strategy to further reduce the impact of straggling workers. We evaluate DSAG for principal component analysis, cast as a finite-sum optimization problem, of a large genomics dataset, and for logistic regression on a cluster composed of 100 workers on AWS, and find that DSAG is up to about 50% faster than SAG, and more than twice as fast as coded computing methods, for the particular scenario that we consider.
Subject headings
- NATURVETENSKAP -- Data- och informationsvetenskap -- Annan data- och informationsvetenskap (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences -- Other Computer and Information Science (hsv//eng)
- NATURVETENSKAP -- Matematik -- Beräkningsmatematik (hsv//swe)
- NATURAL SCIENCES -- Mathematics -- Computational Mathematics (hsv//eng)
- NATURVETENSKAP -- Matematik -- Sannolikhetsteori och statistik (hsv//swe)
- NATURAL SCIENCES -- Mathematics -- Probability Theory and Statistics (hsv//eng)
Keyword
- stochastic average gradient (SAG)
- Coded computing
- load-balancing
- variance reduction
- straggler mitigation
- principal component analysis (PCA)
- iterative optimization
Publication and Content Type
- art (subject category)
- ref (subject category)
Find in a library
To the university's database