BiobankCloud : A platform for the secure storage, sharing, and processing of large biomedical data sets

• Biobanks store and catalog human biological material that is increasingly being digitized using next-generation sequencing (NGS). There is, however, a computational bottleneck, as existing software systems are not scalable and secure enough to store and process the incoming wave of genomic data from NGS machines. In the BiobankCloud project, we are building a Hadoop-based platform for the secure storage, sharing, and parallel processing of genomic data. We extended Hadoop to include support for multi-tenant studies, reduced storage requirements with erasure coding, and added support for extensible and consistent metadata. On top of Hadoop, we built a scalable scientific workflow engine featuring a proper workflow definition language focusing on simple integration and chaining of existing tools, adaptive scheduling on Apache Yarn, and support for iterative dataflows. Our platform also supports the secure sharing of data across different, distributed Hadoop clusters. The software is easily installed and comes with a user-friendly web interface for running, managing, and accessing data sets behind a secure 2-factor authentication. Initial tests have shown that the engine scales well to dozens of nodes. The entire system is open-source and includes pre-defined workflows for popular tasks in biomedical data analysis, such as variant identification, differential transcriptome analysis using RNA-Seq, and analysis of miRNA-Seq and ChIP-Seq data.

Subject headings

TEKNIK OCH TEKNOLOGIER  -- Industriell bioteknik -- Biomaterial (hsv//swe)

ENGINEERING AND TECHNOLOGY  -- Industrial Biotechnology -- Bio Materials (hsv//eng)

NATURVETENSKAP  -- Data- och informationsvetenskap -- Annan data- och informationsvetenskap (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Other Computer and Information Science (hsv//eng)

MEDICIN OCH HÄLSOVETENSKAP  -- Hälsovetenskap -- Medicinsk etik (hsv//swe)

MEDICAL AND HEALTH SCIENCES  -- Health Sciences -- Medical Ethics (hsv//eng)

Keyword

Biological materials

Data handling

Engines

Genes

Information management

Open source software

Open systems

RNA

Storage (materials)

Adaptive scheduling

Biomedical data analysis

Computational bottlenecks

Next-generation sequencing

Parallel processing

Scientific workflow engines

Storage requirements

Transcriptome analysis

Digital storage

Publication and Content Type

ref (subject category)

kon (subject category)