| 1. |
- Fredriksson, Teodor, 1992, et al.
(författare)
-
Assessing the Suitability of Semi-Supervised Learning Datasets using Item Response Theory
- 2021
-
Ingår i: Proceedings - 2021 47th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2021. - : IEEE. - 9781665427050 ; , s. 326-333
-
Konferensbidrag (refereegranskat)abstract
- In practice, supervised learning algorithms require fully labeled datasets to achieve the high accuracy demanded by current modern applications. However, in industrial settings supervised learning algorithms can perform poorly because of few labeled instances. Semi-supervised learning (SSL) is an automatic labeling approach that utilizes complete labels to infer missing labels in partially complete datasets. The high number of available SSL algorithms and the lack of systematic comparison between them leaves practitioners without guidelines to select the appropriate one for their application. Moreover, each SSL algorithm is often validated and evaluated in a small number of common datasets. However, there is no research that examines what datasets are suitable for comparing different SSL algorihtms. The purpose of this paper is to empirically evaluate the suitability of the datasets commonly used to evaluate and compare different SSL algorithms. We performed a simulation study using twelve datasets of three different datatypes (numerical, text, image) on thirteen different SSL algorithms. The contributions of this paper are two-fold. First, we propose the use of Bayesian congeneric item response theory model to assess the suitability of commonly used datasets. Second, we compare the different SSL algorithms using these datasets. The results show that with except of three datasets, the others have very low discrimination factors and are easily solved by the current algorithms. Additionally, the SSL algorithms have overlapping 90% credible intervals, indicating uncertainty in the difference between the accuracy of these SSL models. The paper concludes suggesting that researchers and practitioners should better consider the choice of datasets used for comparing SSL algorithms.
|
|
| 2. |
- John, Meenu Mary, et al.
(författare)
-
Towards MLOps : A Framework and Maturity Model
- 2021
-
Ingår i: Proceedings - 2021 47th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2021. - : IEEE. - 9781665427050 ; , s. 334-341
-
Konferensbidrag (refereegranskat)abstract
- The adoption of continuous software engineering practices such as DevOps (Development and Operations) in business operations has contributed to significantly shorter software development and deployment cycles. Recently, the term MLOps (Machine Learning Operations) has gained increasing interest as a practice that brings together data scientists and operations teams. However, the adoption of MLOps in practice is still in its infancy and there are few common guidelines on how to effectively integrate it into existing software development practices. In this paper, we conduct a systematic literature review and a grey literature review to derive a framework that identifies the activities involved in the adoption of MLOps and the stages in which companies evolve as they become more mature and advanced. We validate this framework in three case companies and show how they have managed to adopt and integrate MLOps in their large-scale software development companies. The contribution of this paper is threefold. First, we review contemporary literature to provide an overview of the state-of-the-art in MLOps. Based on this review, we derive an MLOps framework that details the activities involved in the continuous development of machine learning models. Second, we present a maturity model in which we outline the different stages that companies go through in evolving their MLOps practices. Third, we validate our framework in three embedded systems case companies and map the companies to the stages in the maturity model.
|
|
| 3. |
- Liu, Yuchu, 1992, et al.
(författare)
-
Size matters? Or not : A/B testing with limited sample in automotive embedded software
- 2021
-
Ingår i: 2021 47TH EUROMICRO CONFERENCE ON SOFTWARE ENGINEERING AND ADVANCED APPLICATIONS (SEAA 2021). - : IEEE. - 9781665427050 ; , s. 300-307
-
Konferensbidrag (refereegranskat)abstract
- A/B testing is gaining attention in the automotive sector as a promising tool to measure casual effects from software changes. Different from the web-facing businesses, where A/B testing has been well-established, the automotive domain often suffers from limited eligible users to participate in online experiments. To address this shortcoming, we present a method for designing balanced control and treatment groups so that sound conclusions can be drawn from experiments with considerably small sample sizes. While the Balance Match Weighted method has been used in other domains such as medicine, this is the first paper to apply and evaluate it in the context of software development. Furthermore, we describe the Balance Match Weighted method in detail and we conduct a case study together with an automotive manufacturer to apply the group design method in a fleet of vehicles. Finally, we present our case study in the automotive software engineering domain, as well as a discussion on the benefits and limitations of the A/B group design method.
|
|
| 4. |
- Zhang, Hongyi, 1996, et al.
(författare)
-
AF-DNDF : Asynchronous Federated Learning of Deep Neural Decision Forests
- 2021
-
Ingår i: Proceedings - 2021 47th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2021. - : IEEE. - 9781665427050 ; , s. 308-315
-
Konferensbidrag (refereegranskat)abstract
- In recent years, with more edge devices being put into use, the amount of data that is created, transmitted and stored is increasing exponentially. Moreover, due to the development of machine learning algorithms, modern software-intensive systems are able to take advantage of the data to further improve their service quality. However, it is expensive and inefficient to transmit large amounts of data to a central location for the purpose of training and deploying machine learning models. Data transfer from edge devices across the globe to central locations may also raise privacy and concerns related to local data regulations. As a distributed learning approach, Federated Learning has been introduced to tackle those challenges. Since Federated Learning simply exchanges locally trained machine learning models rather than the entire data set throughout the training process, the method not only protects user data privacy but also improves model training efficiency. In this paper, we have investigated an advanced machine learning algorithm, Deep Neural Decision Forests (DNDF), which unites classification trees with the representation learning functionality from deep convolutional neural networks. In this paper, we propose a novel algorithm, AF-DNDF which extends DNDF with an asynchronous federated aggregation protocol. Based on the local quality of each classification tree, our architecture can select and combine the optimal groups of decision trees from multiple local devices. The introduction of the asynchronous protocol enables the algorithm to be deployed in the industrial context with heterogeneous hardware settings. Our AF-DNDF architecture is validated in an automotive industrial use case focusing on road objects recognition and demonstrated by an empirical experiment with two different data sets. The experimental results show that our AF-DNDF algorithm significantly reduces the communication overhead and accelerates model training speed without sacrificing model classification performance. The algorithm can reach the same classification accuracy as the commonly used centralized machine learning methods but also greatly improve local edge model quality.
|
|
