BiobankCloud: A platform for the secure storage, sharing, and processing of large biomedical data sets

BiobankCloud : A platform for the secure storage, sharing, and processing of large biomedical data sets

QC 20161110
Biobanks store and catalog human biological material that is increasingly being digitized using next-generation sequencing (NGS). There is, however, a computational bottleneck, as existing software systems are not scalable and secure enough to store and process the incoming wave of genomic data from NGS machines. In the BiobankCloud project, we are building a Hadoop-based platform for the secure storage, sharing, and parallel processing of genomic data. We extended Hadoop to include support for multi-tenant studies, reduced storage requirements with erasure coding, and added support for extensible and consistent metadata. On top of Hadoop, we built a scalable scientific workflow engine featuring a proper workflow definition language focusing on simple integration and chaining of existing tools, adaptive scheduling on Apache Yarn, and support for iterative dataflows. Our platform also supports the secure sharing of data across different, distributed Hadoop clusters. The software is easily installed and comes with a user-friendly web interface for running, managing, and accessing data sets behind a secure 2-factor authentication. Initial tests have shown that the engine scales well to dozens of nodes. The entire system is open-source and includes pre-defined workflows for popular tasks in biomedical data analysis, such as variant identification, differential transcriptome analysis using RNA-Seq, and analysis of miRNA-Seq and ChIP-Seq data.

TEKNIK OCH TEKNOLOGIER Industriell bioteknik Biomaterial hsv//swe
ENGINEERING AND TECHNOLOGY Industrial Biotechnology Bio Materials hsv//eng
NATURVETENSKAP Data- och informationsvetenskap Annan data- och informationsvetenskap hsv//swe
NATURAL SCIENCES Computer and Information Sciences Other Computer and Information Science hsv//eng
MEDICIN OCH HÄLSOVETENSKAP Hälsovetenskap Medicinsk etik hsv//swe
MEDICAL AND HEALTH SCIENCES Health Sciences Medical Ethics hsv//eng
Biological materials
Data handling
Engines
Genes
Information management
Open source software
Open systems
RNA
Storage (materials)
Adaptive scheduling
Biomedical data analysis
Computational bottlenecks
Next-generation sequencing
Parallel processing
Scientific workflow engines
Storage requirements
Transcriptome analysis
Digital storage

Brandt, J.Humboldt Univ, Berlin, Germany. (author)
Bux, M.Humboldt Univ, Berlin, Germany. (author)
Cogo, V.Univ Lisbon, Fac Ciencias, LaSIGE, Lisbon, Portugal. (author)
Dimitrova, L.Charite, Berlin, Germany. (author)
Dowling, JimKTH,Programvaruteknik och Datorsystem, SCS,KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u1ajem4z (author)
Gholami, AliKTH,KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u1c8thwr (author)
Hakimzadeh, KamalKTH,Programvaruteknik och Datorsystem, SCS,KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u1g0naq4 (author)
Hummel, M.Charite, Berlin, Germany. (author)
Ismail, MahmoudKTH,Programvaruteknik och Datorsystem, SCS,KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u18jmfwc (author)
Laure, ErwinKTH,Parallelldatorcentrum, PDC,High Performance Computing and Visualization (HPCViz),KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u1n9d88r (author)
Leser, U.Humboldt Univ, Berlin, Germany. (author)
Litton, J. -EKarolinska Inst, Solna, Sweden. (author)
Martinez, R.Karolinska Inst, Solna, Sweden. (author)
Niazi, SalmanKTH,Programvaruteknik och Datorsystem, SCS,KTH Royal Inst Technol, Stockholm, Sweden.(Swepub:kth)u1egcptg (author)
Reichel, JaneUppsala universitet,Centrum för forsknings- och bioetik(Swepub:uu)janre863 (author)
Zimmermann, K.Charite, Berlin, Germany. (author)
Univ Lisbon, Fac Ciencias, LaSIGE, Lisbon, Portugal.Humboldt Univ, Berlin, Germany. (creator_code:org_t)

In:1st International Workshop on Data Management and Analytics for Medicine and Healthcare, DMAH 2015 and Workshop on Big-Graphs Online Querying, Big-O(Q) 2015 held in conjunction with 41st International Conference on Very Large Data Bases, VLDB 2015Cham : Springer, s. 89-10597833194157589783319415765

By the author/editor: Bessani, A.; Brandt, J.; Bux, M.; Cogo, V.; Dimitrova, L.; Dowling, Jim; show more...; Gholami, Ali; Hakimzadeh, Kama ...; Hummel, M.; Ismail, Mahmoud; Laure, Erwin; Leser, U.; Litton, J. -E; Martinez, R.; Niazi, Salman; Reichel, Jane; Zimmermann, K.; show less...

About the subject

ENGINEERING AND TECHNOLOGY: ENGINEERING AND ...; and Industrial Biote ...; and Bio Materials

NATURAL SCIENCES: NATURAL SCIENCES; and Computer and Inf ...; and Other Computer a ...

MEDICAL AND HEALTH SCIENCES: MEDICAL AND HEAL ...; and Health Sciences; and Medical Ethics

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.