Search: onr:"swepub:oai:DiVA.org:kth-216289" >
Scaling HDFS to mor...
Scaling HDFS to more than 1 million operations per second with HopsFS
-
- Ismail, Mahmoud (author)
- KTH,Programvaruteknik och Datorsystem, SCS
-
- Niazi, Salman, 1982- (author)
- KTH,Programvaruteknik och Datorsystem, SCS
-
Ronstrom, M. (author)
-
show more...
-
Haridi, S. (author)
-
- Dowling, Jim (author)
- KTH,Programvaruteknik och Datorsystem, SCS
-
show less...
-
(creator_code:org_t)
- Institute of Electrical and Electronics Engineers Inc. 2017
- 2017
- English.
-
In: Proceedings - 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2017. - : Institute of Electrical and Electronics Engineers Inc.. - 9781509066100 ; , s. 683-688
- Related links:
-
https://urn.kb.se/re...
-
show more...
-
https://doi.org/10.1...
-
show less...
Abstract
Subject headings
Close
- HopsFS is an open-source, next generation distribution of the Apache Hadoop Distributed File System(HDFS) that replaces the main scalability bottleneck in HDFS, single node in-memory metadata service, with a no-sharedstate distributed system built on a NewSQL database. By removing the metadata bottleneck in Apache HDFS, HopsFS enables significantly larger cluster sizes, more than an order of magnitude higher throughput, and significantly lower clientlatencies for large clusters. In this paper, we detail the techniques and optimizations that enable HopsFS to surpass 1 million file system operations per second-at least 16 times higher throughput than HDFS. In particular, we discuss how we exploit recent high performance features from NewSQL databases, such as application defined partitioning, partition-pruned index scans, and distribution aware transactions. Together with more traditional techniques, such as batching and write-Ahead caches, we show how many incremental optimizations have enabled a revolution in distributed hierarchical file system performance.
Subject headings
- NATURVETENSKAP -- Data- och informationsvetenskap (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences (hsv//eng)
Keyword
- Distributed File System
- File System Design
- High-performance file systems
- NewSQL
- Cluster computing
- Computer software
- Distributed database systems
- File organization
- Grid computing
- Metadata
- Open systems
- Distributed file systems
- Distributed systems
- File systems
- Hadoop distributed file system (HDFS)
- Incremental optimization
- Metadata services
- Traditional techniques
- Distributed computer systems
Publication and Content Type
- ref (subject category)
- kon (subject category)
Find in a library
To the university's database