| 1. |
|
|
| 2. |
|
|
| 3. |
|
|
| 4. |
|
|
| 5. |
|
|
| 6. |
- Blamey, Ben, et al.
(författare)
-
Rapid development of cloud-native intelligent data pipelines for scientific data streams using the HASTE Toolkit
- 2021
-
Ingår i: GigaScience. - : Oxford University Press. - 2047-217X. ; 10:3, s. 1-14
-
Tidskriftsartikel (refereegranskat)abstract
- BACKGROUND: Large streamed datasets, characteristic of life science applications, are often resource-intensive to process, transport and store. We propose a pipeline model, a design pattern for scientific pipelines, where an incoming stream of scientific data is organized into a tiered or ordered "data hierarchy". We introduce the HASTE Toolkit, a proof-of-concept cloud-native software toolkit based on this pipeline model, to partition and prioritize data streams to optimize use of limited computing resources.FINDINGS: In our pipeline model, an "interestingness function" assigns an interestingness score to data objects in the stream, inducing a data hierarchy. From this score, a "policy" guides decisions on how to prioritize computational resource use for a given object. The HASTE Toolkit is a collection of tools to adopt this approach. We evaluate with 2 microscopy imaging case studies. The first is a high content screening experiment, where images are analyzed in an on-premise container cloud to prioritize storage and subsequent computation. The second considers edge processing of images for upload into the public cloud for real-time control of a transmission electron microscope.CONCLUSIONS: Through our evaluation, we created smart data pipelines capable of effective use of storage, compute, and network resources, enabling more efficient data-intensive experiments. We note a beneficial separation between scientific concerns of data priority, and the implementation of this behaviour for different resources in different deployment contexts. The toolkit allows intelligent prioritization to be `bolted on' to new and existing systems - and is intended for use with a range of technologies in different deployment scenarios.
|
|
| 7. |
- Nelson, Greg L., et al.
(författare)
-
Differentiated Assessments for Advanced Courses that Reveal Issues with Prerequisite Skills : A Design Investigation
- 2020
-
Ingår i: ITICSE-WGR'20. - New York, NY, USA : Association for Computing Machinery (ACM). - 9781450382939 ; , s. 75-129
-
Konferensbidrag (refereegranskat)abstract
- Computing learners may not master basic concepts, or forget them between courses or from infrequent use. Learners also often struggle with advanced computing courses, perhaps from weakness with prerequisite concepts. One underlying challenge for researchers and instructors is determining the reason why a learner gets an advanced question wrong. Was the wrong answer because the learner lacked prerequisite skills, has not mastered the advanced skill, or some combination of the two? We contribute a design investigation into how to create differentiated questions which diagnose prerequisite and advanced skills at the same time. We focused on tracing and related skills as prerequisites, and on advanced object-oriented programming, concurrency, algorithm and data structures as the advanced skills. We conducted an inductive qualitative analysis of existing assessment questions from instructors and from a concept inventory with a validity argument (the Basic Data Structures Inventory). We found dependencies on a variety of prerequisite knowledge and mixed potential for diagnosing difficulties with prerequisites. Inspired by this analysis, we developed examples of differentiated assessments and reflected on design principles for creating/modifying assessments to better assess both advanced and prerequisite skills. Our example differentiated assessment questions and methods help enable research into how prerequisites skills affect learning of advanced concepts. They also may help instructors better understand and help learners with varying prerequisite knowledge, which may improve equity of learning outcomes. Our work also raises theoretical questions about what assessments really assess and how separate advanced topics and prerequisite skills are.
|
|
| 8. |
- Stein, Oliver, et al.
(författare)
-
Smart Resource Management for Data Streaming using an Online Bin-packing Strategy
- 2020
-
Ingår i: 2020 IEEE International Conference on Big Data (Big Data). - 9781728162515 - 9781728162522 ; , s. 2207-2216
-
Konferensbidrag (refereegranskat)abstract
- Data stream processing frameworks provide reliable and efficient mechanisms for executing complex workflows over large datasets. A common challenge for the majority of currently available streaming frameworks is efficient utilization of resources. Most frameworks use static or semi-static settings for resource utilization that work well for established use cases but lead to marginal improvements for unseen scenarios. Another pressing issue is the efficient processing of large individual objects such as images and matrices typical for scientific datasets. HarmonicIO has proven to be a good solution for streams of relatively large individual objects, as demonstrated in a benchmark comparison with the Apache Spark and Kafka streaming frameworks. We here present an extension of the HarmonicIO framework based on the online bin-packing algorithm. The main focus is to compare different strategies adapted in streaming frameworks for efficient resource utilization. Based on a real world use case from large-scale microscopy pipelines, we compare two different strategies of auto-scaling implemented in the HarmonicIO and Spark Streaming frameworks.
|
|
| 9. |
|
|
