Scaling HDFS to more than 1 million operations per second with HopsFS

• HopsFS is an open-source, next generation distribution of the Apache Hadoop Distributed File System(HDFS) that replaces the main scalability bottleneck in HDFS, single node in-memory metadata service, with a no-sharedstate distributed system built on a NewSQL database. By removing the metadata bottleneck in Apache HDFS, HopsFS enables significantly larger cluster sizes, more than an order of magnitude higher throughput, and significantly lower clientlatencies for large clusters. In this paper, we detail the techniques and optimizations that enable HopsFS to surpass 1 million file system operations per second-at least 16 times higher throughput than HDFS. In particular, we discuss how we exploit recent high performance features from NewSQL databases, such as application defined partitioning, partition-pruned index scans, and distribution aware transactions. Together with more traditional techniques, such as batching and write-Ahead caches, we show how many incremental optimizations have enabled a revolution in distributed hierarchical file system performance.

Ämnesord

NATURVETENSKAP  -- Data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences (hsv//eng)

Nyckelord

Distributed File System

File System Design

High-performance file systems

NewSQL

Cluster computing

Computer software

Distributed database systems

File organization

Grid computing

Metadata

Open systems

Distributed file systems

Distributed systems

File systems

Hadoop distributed file system (HDFS)

Incremental optimization

Metadata services

Traditional techniques

Distributed computer systems

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)