| 1. |
- Buljan, Maja, et al.
(author)
-
A Tale of Four Parsers : Methodological Reflections on Diagnostic Evaluation and In-Depth Error Analysis for Meaning Representation Parsing
- 2022
-
In: Language Resources and Evaluation. - : Springer Science and Business Media LLC. - 1574-020X .- 1574-0218. ; 56:4, s. 1075-1102
-
Journal article (peer-reviewed)abstract
- We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena—by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.
|
|
| 2. |
- Buljan, Maja, et al.
(author)
-
A Tale of Three Parsers : Towards Diagnostic Evaluation for Meaning Representation Parsing
- 2020
-
In: Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020). - Paris : European Language Resources Association (ELRA). - 9791095546344 ; , s. 1902-1909
-
Conference paper (peer-reviewed)abstract
- We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and two of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.
|
|
| 3. |
|
|
| 4. |
- Fares, Murhaf, et al.
(author)
-
The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers
- 2018
-
In: Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. - : Association for Computational Linguistics.
-
Conference paper (peer-reviewed)abstract
- We summarize empirical results and tentative conclusions from the Second Extrinsic Parser Evaluation Initiative (EPE 2018). We review the basic task setup, downstream applications involved, and end-to-end results for seventeen participating parsers. Based on both quantitative and qualitative analysis, we correlate intrinsic evaluation results at different layers of morph-syntactic analysis with observed downstream behavior.
|
|
| 5. |
- Kuhlmann, Marco, 1977-, et al.
(author)
-
Towards a Catalogue of Linguistic Graph Banks
- 2016
-
In: Computational linguistics - Association for Computational Linguistics (Print). - : MIT Press. - 0891-2017 .- 1530-9312. ; 42:4, s. 819-827
-
Journal article (peer-reviewed)abstract
- Graphs exceeding the formal complexity of rooted trees are of growing relevance to much NLP research. Although formally well understood in graph theory, there is substantial variation in the types of linguistic graphs, as well as in the interpretation of various structural properties. To provide a common terminology and transparent statistics across different collections of graphs in NLP, we propose to establish a shared community resource with an open-source reference implementation for common statistics.
|
|
| 6. |
- Kurtz, Robin, 1988-, et al.
(author)
-
End-to-End Negation Resolution as Graph Parsing
- 2020
-
In: Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781952148118 ; , s. 14-24
-
Conference paper (peer-reviewed)abstract
- We present a neural end-to-end architecture for negation resolution based on a formulation of the task as a graph parsing problem. Our approach allows for the straightforward inclusion of many types of graph-structured features without the need for representation-specific heuristics. In our experiments, we specifically gauge the usefulness of syntactic information for negation resolution. Despite the conceptual simplicity of our architecture, we achieve state-of-the-art results on the Conan Doyle benchmark dataset, including a new top result for our best model.
|
|
| 7. |
- Oepen, Stephan, et al.
(author)
-
MRP 2019: Cross-Framework Meaning Representation Parsing
- 2019
-
In: Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning. - : Association for Computational Linguistics. - 9781950737604 ; , s. 1-27
-
Conference paper (other academic/artistic)abstract
- The 2019 Shared Task at the Conference for Computational Language Learning (CoNLL) was devoted to Meaning Representation Parsing (MRP) across frameworks. Five distinct approaches to the representation of sentence meaning in the form of directed graph were represented in the training and evaluation data for the task, packaged in a uniform abstract graph representation and serialization. The task received submissions from eighteen teams, of which five do not participate in the official ranking because they arrived after the closing deadline, made use of additional training data, or involved one of the task co-organizers. All technical information regarding the task, including system submissions, official results, and links to supporting resources and software are available from the task web site at: http://mrp.nlpl.eu
|
|
| 8. |
- Oepen, Stephan, et al.
(author)
-
SemEval 2014 Task 8: Broad-Coverage Semantic Dependency Parsing
- 2014
-
In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014). - : Association for Computational Linguistics. - 9781941643242 ; , s. 63-72
-
Conference paper (peer-reviewed)abstract
- Task 8 at SemEval 2014 defines Broad-Coverage Semantic Dependency Parsing (SDP) as the problem of recovering sentence-internal predicate–argument relationships for all content words, i.e. the semantic structure constituting the relational core of sentence meaning. In this task description, we position the problem in comparison to other sub-tasks in computational language analysis, introduce the semantic dependency target representations used, reflect on high-level commonalities and differences between these representations, and summarize the task setup, participating systems, and main results.
|
|
| 9. |
- Oepen, Stephan, et al.
(author)
-
SemEval 2015 Task 18: Broad-Coverage Semantic Dependency Parsing
- 2015
-
In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). - : Association for Computational Linguistics. - 9781941643396 ; , s. 915-926
-
Conference paper (peer-reviewed)abstract
- Task 18 at SemEval 2015 defines Broad-Coverage Semantic Dependency Parsing (SDP) as the problem of recovering sentence-internal predicate–argument relationships for all content words, i.e. the semantic structure constituting the relational core of sentence meaning. In this task description, we position the problem in comparison to other language analysis sub-tasks, introduce and compare the semantic dependency target representations used, and summarize the task setup, participating systems, and main results.
|
|
| 10. |
- Oepen, Stephan, et al.
(author)
-
The 2017 Shared Task on Extrinsic Parser Evaluation. Towards a Reusable Community Infrastructure
- 2017
-
In: Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. - Stroudsburg, USA : Association for Computational Linguistics (ACL). - 9781945626746
-
Conference paper (peer-reviewed)abstract
- The 2017 Shared Task on Extrinsic Parser Evaluation (EPE 2017) seeks to provide better estimates of the relative utility of different types of dependency representa- tions for a variety of downstream applica- tions that depend centrally on the analysis of grammatical structure. EPE 2017 de- fi nes a generalized notion of lexicalized syntactico-semantic dependency represen- tations and provides a common interchange format to three state-of-the-art downstream applications, viz. biomedical event extrac- tion, negation resolution, and fi ne-grained opinion analysis. As a fi rst step towards building a generic and extensible infras- tructure for extrinsic parser evaluation, the downstream applications have been gener- alized to support a broad range of diverese dependency representations (including di- vergent sentence and token boundaries) and to allow fully automated re-training and evaluation for a speci fi c collection of parser outputs. Nine teams participated in EPE 2017, submitting 49 distinct runs that encompass many different families of dependency representations, distinct ap- proaches to preprocessing and parsing, and various types and volumes of training data.
|
|
