| 1. |
- Borg, Markus, et al.
(författare)
-
Quality Assurance of Generative Dialog Models in an Evolving Conversational Agent Used for Swedish Language Practice
- 2022
-
Ingår i: Proceedings - 1st International Conference on AI Engineering - Software Engineering for AI, CAIN 2022. - New York, NY, USA : Institute of Electrical and Electronics Engineers Inc.. - 9781450392754 ; , s. 22-32
-
Konferensbidrag (refereegranskat)abstract
- Due to the migration megatrend, efficient and effective second-language acquisition is vital. One proposed solution involves AI-enabled conversational agents for person-centered interactive language practice. We present results from ongoing action research targeting quality assurance of proprietary generative dialog models trained for virtual job interviews. The action team elicited a set of 38 requirements for which we designed corresponding automated test cases for 15 of particular interest to the evolving solution. Our results show that six of the test case designs can detect meaningful differences between candidate models. While quality assurance of natural language processing applications is complex, we provide initial steps toward an automated framework for machine learning model selection in the context of an evolving conversational agent. Future work will focus on model selection in an MLOps setting.
|
|
| 2. |
- Foidl, Harald, et al.
(författare)
-
Data Smells : Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems
- 2022
-
Ingår i: Proceedings - 1st International Conference on AI Engineering - Software Engineering for AI, CAIN 2022. - New York, NY, USA : Institute of Electrical and Electronics Engineers (IEEE). - 9781450392754 ; , s. 229-239
-
Konferensbidrag (refereegranskat)abstract
- High data quality is fundamental for today's AI-based systems. However, although data quality has been an object of research for decades, there is a clear lack of research on potential data quality issues (e.g., ambiguous, extraneous values). These kinds of issues are latent in nature and thus often not obvious. Nevertheless, they can be associated with an increased risk of future problems in AI-based systems (e.g., technical debt, data-induced faults). As a counterpart to code smells in software engineering, we refer to such issues as Data Smells. This article conceptualizes data smells and elaborates on their causes, consequences, detection, and use in the context of AI-based systems. In addition, a catalogue of 36 data smells divided into three categories (i.e., Believability Smells, Understandability Smells, Consistency Smells) is presented. Moreover, the article outlines tool support for detecting data smells and presents the result of an initial smell detection on more than 240 real-world datasets.
|
|
| 3. |
- Song, Qunying, et al.
(författare)
-
Exploring ML testing in practice - Lessons learned from an interactive rapid review with Axis Communications
- 2022
-
Ingår i: Proceedings - 1st International Conference on AI Engineering - Software Engineering for AI, CAIN 2022. - New York, NY, USA : Institute of Electrical and Electronics Engineers Inc.. - 9781450392754 - 9781665452069 ; , s. 10-21
-
Konferensbidrag (refereegranskat)abstract
- There is a growing interest in industry and academia in machine learning (ML) testing. We believe that industry and academia need to learn together to produce rigorous and relevant knowledge. In this study, we initiate a collaboration between stakeholders from one case company, one research institute, and one university. To establish a common view of the problem domain, we applied an interactive rapid review of the state of the art. Four researchers from Lund University and RISE Research Institutes and four practitioners from Axis Communications reviewed a set of 180 primary studies on ML testing. We developed a taxonomy for the communication around ML testing challenges and results and identified a list of 12 review questions relevant for Axis Communications. The three most important questions (data testing, metrics for assessment, and test generation) were mapped to the literature, and an in-depth analysis of the 35 primary studies matching the most important question (data testing) was made. A final set of the five best matches were analysed and we reflect on the criteria for applicability and relevance for the industry. The taxonomies are helpful for communication but not final. Furthermore, there was no perfect match to the case company's investigated review question (data testing). However, we extracted relevant approaches from the five studies on a conceptual level to support later context-specific improvements. We found the interactive rapid review approach useful for triggering and aligning communication between the different stakeholders.
|
|
