SwePub
Sök i LIBRIS databas

  Extended search

WFRF:(Wählby Carolina 1974 )
 

Search: WFRF:(Wählby Carolina 1974 ) > (2020-2024) > Rapid development o...

  • Blamey, BenUppsala universitet,Avdelningen för beräkningsvetenskap,Tillämpad beräkningsvetenskap,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden (author)

Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit

  • Article/chapterEnglish2021

Publisher, publication year, extent ...

  • 2021-03-19
  • Oxford University Press,2021
  • electronicrdacarrier

Numbers

  • LIBRIS-ID:oai:DiVA.org:uu-439701
  • https://urn.kb.se/resolve?urn=urn:nbn:se:uu:diva-439701URI
  • https://doi.org/10.1093/gigascience/giab018DOI
  • https://urn.kb.se/resolve?urn=urn:nbn:se:mau:diva-66098URI

Supplementary language notes

  • Language:English
  • Summary in:English

Part of subdatabase

Classification

  • Subject category:ref swepub-contenttype
  • Subject category:art swepub-publicationtype

Notes

  • Spjuth and Hellander shared senior authorship
  • BACKGROUND: Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered "data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources.FINDINGS: In our pipeline model, an "interestingness function" assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a "policy" guides decisions on how to prioritize computational resource use for a given object. The HASTE Toolkit is a collection of tools to adopt this approach. We evaluate with 2 microscopy imaging case studies. The first is a high content screening experiment, where images are analyzed in an on-premise container cloud to prioritize storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for real-time control of a transmission electron microscope.CONCLUSIONS: Through our evaluation, we created smart data pipelines capable of effective use of storage, compute, and network resources, enabling more efficient data-intensive experiments. We note a beneficial separation between scientific concerns of data priority, and the implementation of this behaviour for different resources in different deployment contexts. The toolkit allows intelligent prioritization to be `bolted on' to new and existing systems - and is intended for use with a range of technologies in different deployment scenarios.

Subject headings and genre

Added entries (persons, corporate bodies, meetings, titles ...)

  • Toor, SalmanUppsala universitet,Tillämpad beräkningsvetenskap,Avdelningen för beräkningsvetenskap,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden(Swepub:uu)salto690 (author)
  • Dahlö, MartinUppsala universitet,Science for Life Laboratory, SciLifeLab,Institutionen för farmaceutisk biovetenskap,Ola Spjuth,Department of Pharmaceutical Biosciences, Uppsala University, Husargatan 3, 75237, Uppsala, Sweden;Science for Life Laboratory, Uppsala University, Husargatan 3, 75237 Uppsala, Sweden(Swepub:uu)marda709 (author)
  • Wieslander, HåkanUppsala universitet,Avdelningen för visuell information och interaktion,Bildanalys och människa-datorinteraktion,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden(Swepub:uu)hakwi402 (author)
  • Harrison, Philip J.Uppsala universitet,Institutionen för farmaceutisk biovetenskap,Science for Life Laboratory, SciLifeLab,Spjuth,Department of Pharmaceutical Biosciences, Uppsala University, Husargatan 3, 75237, Uppsala, Sweden;Science for Life Laboratory, Uppsala University, Husargatan 3, 75237 Uppsala, Sweden(Swepub:uu)phiha876 (author)
  • Sintorn, Ida-Maria,1976-Uppsala universitet,Bildanalys och människa-datorinteraktion,Avdelningen för visuell information och interaktion,Science for Life Laboratory, SciLifeLab,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden;Science for Life Laboratory, Uppsala University, Husargatan 3, 75237 Uppsala, Sweden;Vironova AB, Gävlegatan 22, 11330 Stockholm, Sweden(Swepub:uu)idsin102 (author)
  • Sabirsh, AlanAdvanced Drug Delivery, Pharmaceutical Sciences, R&D, AstraZeneca, Pepparedsleden 1, 43183 Mölndal, Sweden (author)
  • Wählby, Carolina,professor,1974-Uppsala universitet,Bildanalys och människa-datorinteraktion,Science for Life Laboratory, SciLifeLab,Avdelningen för visuell information och interaktion,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden;Science for Life Laboratory, Uppsala University, Husargatan 3, 75237 Uppsala, Sweden(Swepub:uu)cli05194 (author)
  • Spjuth, Ola,Professor,1977-Uppsala universitet,Institutionen för farmaceutisk biovetenskap,Science for Life Laboratory, SciLifeLab,Spjuth,Department of Pharmaceutical Biosciences, Uppsala University, Husargatan 3, 75237, Uppsala, Sweden;Science for Life Laboratory, Uppsala University, Husargatan 3, 75237 Uppsala, Sweden(Swepub:uu)olspj499 (author)
  • Hellander, AndreasUppsala universitet,Avdelningen för beräkningsvetenskap,Tillämpad beräkningsvetenskap,Department of Information Technology, Uppsala University, Lägerhyddsvägen 2, 75237 Uppsala, Sweden(Swepub:uu)andhe896 (author)
  • Uppsala universitetAvdelningen för beräkningsvetenskap (creator_code:org_t)

Related titles

  • In:GigaScience: Oxford University Press10:3, s. 1-142047-217X

Internet link

Find in a library

To the university's database

Kungliga biblioteket hanterar dina personuppgifter i enlighet med EU:s dataskyddsförordning (2018), GDPR. Läs mer om hur det funkar här.
Så här hanterar KB dina uppgifter vid användning av denna tjänst.

 
pil uppåt Close

Copy and save the link in order to return to this view