| 1. |
- Kroll, Lars, et al.
(author)
-
Fast and Flexible Networking for Message-Oriented Middleware
- 2017
-
In: Proceedings - International Conference on Distributed Computing Systems. - : Institute of Electrical and Electronics Engineers (IEEE). - 9781538617915 ; , s. 1453-1464
-
Conference paper (peer-reviewed)abstract
- Distributed applications deployed in multi-datacenter environments need to deal with network connections of varying quality, including high bandwidth and low latency within a datacenter and, more recently, high bandwidth and high latency between datacentres. In principle, for a given network connection, each message should be sent over the best available network protocol, but existing middlewares do not provide this functionality. In this paper, we present KompicsMessaging, a messaging middleware that allows for fine-grained control of the network protocol used on a per-message basis. Rather than always requiring application developers to specify the appropriate protocol for each message, we also provide an online reinforcement learner that optimises the selection of the network protocol for the current network environment. In experiments, we show how connection properties, such as the varying round-trip time, influence the performance of the application and we show how throughput and latency can be improved by picking the right protocol at the right time.
|
|
| 2. |
- Ormenisan, Alexandru-Adrian, et al.
(author)
-
Dela-Sharing Large Datasets between Hadoop Clusters
- 2017
-
In: Proceedings - International Conference on Distributed Computing Systems. - : Institute of Electrical and Electronics Engineers (IEEE). - 9781538617915 ; , s. 2533-2536
-
Conference paper (peer-reviewed)abstract
- Big data has, in recent years, revolutionised an evergrowing number of fields, from machine learning to climate science to genomics. The current state-of-the-art for storing large datasets is either object stores or distributed filesystems, with Hadoop being the dominant open-source platform for managing 'Big Data'. Existing large-scale storage platforms, however, lack support for the efficient sharing of large datasets over the Internet. Those systems that are widely used for the dissemination of large files, like BitTorrent, need to be adapted to handle challenges such as network links with both high latency and high bandwidth, and scalable storage backends that are optimised for streaming and not random access. In this paper, we introduce Dela, a peer-to-peer data-sharing service integrated into the Hops Hadoop platform that provides an end-to-end solution for dataset sharing. Dela is designed for large-scale storage backends and data transfers that are both non-intrusive to existing TCP network traffic and provide higher network throughput than TCP on high latency, high bandwidth network links, such as transatlantic network links. Dela provides a pluggable storage layer, implementing two alternative ways for clients to access shared data: stream processing of data as it arrives with Kafka, and traditional offline access to data using the Hadoop Distributed Filesystem. Dela is the first step for the Hadoop platform towards creating an open dataset ecosystem that supports user-friendly publishing, searching, and downloading of large datasets.
|
|
| 3. |
- Ormenisan, Alexandru-Adrian, et al.
(author)
-
Time travel and provenance for machine learning pipelines
- 2020
-
In: OpML 2020 - 2020 USENIX Conference on Operational Machine Learning. - : USENIX Association.
-
Conference paper (peer-reviewed)abstract
- Machine learning pipelines have become the defacto paradigm for productionizing machine learning applications as they clearly abstract the processing steps involved in transforming raw data into engineered features that are then used to train models. In this paper, we use a bottom-up method for capturing provenance information regarding the processing steps and artifacts produced in ML pipelines. Our approach is based on replacing traditional intrusive hooks in application code (to capture ML pipeline events) with standardized change-data-capture support in the systems involved in ML pipelines: the distributed file system, feature store, resource manager, and applications themselves. In particular, we leverage data versioning and time-travel capabilities in our feature store to show how provenance can enable model reproducibility and debugging.
|
|
