SwePub - search: WFRF:(Nivre Joakim 1962 )

Enumeration	Reference	Cover	Find
1.	Baldwin, Timothy, et al. (author) Universals of Linguistic Idiosyncrasy in Multilingual Computational Linguistics 2021 In: Dagstuhl Reports. - Dagstuhl. - 2192-5283. ; 11:7, s. 89-138 Journal article (peer-reviewed)abstract Computational linguistics builds models that can usefully process and produce language and thatcan increase our understanding of linguistic phenomena. From the computational perspective,language data are particularly challenging notably due to their variable degree of idiosyncrasy(unexpected properties shared by few peer objects), and the pervasiveness of non-compositionalphenomena such as multiword expressions (whose meaning cannot be straightforwardly deducedfrom the meanings of their components, e.g. red tape, by and large, to pay a visit and to pullone’s leg) and constructions (conventional associations of forms and meanings). Additionally, ifmodels and methods are to be consistent and valid across languages, they have to face specificitiesinherent either to particular languages, or to various linguistic traditions.These challenges were addressed by the Dagstuhl Seminar 21351 entitled “Universals ofLinguistic Idiosyncrasy in Multilingual Computational Linguistics”, which took place on 30-31 August 2021. Its main goal was to create synergies between three distinct though partlyoverlapping communities: experts in typology, in cross-lingual morphosyntactic annotation and inmultiword expressions. This report documents the program and the outcomes of the seminar. Wepresent the executive summary of the event, reports from the 3 Working Groups and abstracts ofindividual talks and open problems presented by the participants.
2.	Basirat, Ali, 1982-, et al. (author) Real-valued syntactic word vectors 2020 In: Journal of experimental and theoretical artificial intelligence (Print). - 0952-813X .- 1362-3079. ; 32:4, s. 557-579 Journal article (peer-reviewed)abstract We introduce a word embedding method that generates a set of real-valued word vectors from a distributional semantic space. The semantic space is built with a set of context units (words) which are selected by an entropy-based feature selection approach with respect to the certainty involved in their contextual environments. We show that the most predictive context of a target word is its preceding word. An adaptive transformation function is also introduced that reshapes the data distribution to make it suitable for dimensionality reduction techniques. The final low-dimensional word vectors are formed by the singular vectors of a matrix of transformed data. We show that the resulting word vectors are as good as other sets of word vectors generated with popular word embedding methods.
3.	Basirat, Ali, Postdoctoral Researcher, 1982-, et al. (author) Syntactic Nuclei in Dependency Parsing – : A Multilingual Exploration 2021 In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781954085022 ; , s. 1376-1387 Conference paper (peer-reviewed)abstract Standard models for syntactic dependency parsing take words to be the elementary units that enter into dependency relations. In this paper, we investigate whether there are any benefits from enriching these models with the more abstract notion of nucleus proposed by Tesniere. We do this by showing how the concept of nucleus can be defined in the framework of Universal Dependencies and how we can use composition functions to make a transition-based dependency parser aware of this concept. Experiments on 12 languages show that nucleus composition gives small but significant improvements in parsing accuracy. Further analysis reveals that the improvement mainly concerns a small number of dependency relations, including relations of coordination, direct objects, nominal modifiers, and main predicates.
4.	Buljan, Maja, et al. (author) A Tale of Four Parsers : Methodological Reflections on Diagnostic Evaluation and In-Depth Error Analysis for Meaning Representation Parsing 2022 In: Language Resources and Evaluation. - : Springer Science and Business Media LLC. - 1574-020X .- 1574-0218. ; 56:4, s. 1075-1102 Journal article (peer-reviewed)abstract We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena—by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.
5.	Buljan, Maja, et al. (author) A Tale of Three Parsers : Towards Diagnostic Evaluation for Meaning Representation Parsing 2020 In: Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020). - Paris : European Language Resources Association (ELRA). - 9791095546344 ; , s. 1902-1909 Conference paper (peer-reviewed)abstract We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and two of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.
6.	Carlsson, Fredrik, et al. (author) Fine-Grained Controllable Text Generation Using Non-Residual Prompting 2022 In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781955917216 ; , s. 6837-6857 Conference paper (peer-reviewed)abstract The introduction of immensely large Causal Language Models (CLMs) has rejuvenated the interest in open-ended text generation. However, controlling the generative process for these Transformer-based models is at large an unsolved problem. Earlier work has explored either plug-and-play decoding strategies, or more powerful but blunt approaches such as prompting. There hence currently exists a trade-off between fine-grained control, and the capability for more expressive high-level instructions. To alleviate this trade-off, we propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. We propose a resource-efficient method for converting a pre-trained CLM into this architecture, and demonstrate its potential on various experiments, including the novel task of contextualized word inclusion. Our method provides strong results on multiple experimental settings, proving itself to be both expressive and versatile.
7.	de Lhoneux, Miryam, 1990-, et al. (author) What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions? 2020 In: Computational linguistics - Association for Computational Linguistics (Print). - : MIT Press. - 0891-2017 .- 1530-9312. ; 46:4, s. 763-784 Journal article (peer-reviewed)abstract There is a growing interest in investigating what neural NLP models learn about language. A prominent open question is the question of whether or not it is necessary to model hierarchical structure. We present a linguistic investigation of a neural parser adding insights to this question. We look at transitivity and agreement information of auxiliary verb constructions (AVCs) in comparison to finite main verbs (FMVs). This comparison is motivated by theoretical work in dependency grammar and in particular the work of Tesnière (1959), where AVCs and FMVs are both instances of a nucleus, the basic unit of syntax. An AVC is a dissociated nucleus; it consists of at least two words, and an FMV is its non-dissociated counterpart, consisting of exactly one word. We suggest that the representation of AVCs and FMVs should capture similar information. We use diagnostic classifiers to probe agreement and transitivity information in vectors learned by a transition-based neural parser in four typologically different languages. We find that the parser learns different information about AVCs and FMVs if only sequential models (BiLSTMs) are used in the architecture but similar information when a recursive layer is used. We find explanations for why this is the case by looking closely at how information is learned in the network and looking at what happens with different dependency representations of AVCs. We conclude that there may be benefits to using a recursive layer in dependency parsing and that we have not yet found the best way to integrate it in our parsers.
8.	De Marneffe, Marie-Catherine, et al. (author) Universal Dependencies 2021 In: Computational Linguistics. - : MIT Press. - 0891-2017 .- 1530-9312. ; 47, s. 255-308 Journal article (peer-reviewed)abstract Universal dependencies (UD) is a framework for morphosyntactic annotation of human language, which to date has been used to create treebanks for more than 100 languages. In this article, we outline the linguistic theory of the UD framework, which draws on a long tradition of typologically oriented grammatical theories. Grammatical relations between words are centrally used to explain how predicate–argument structures are encoded morphosyntactically in different languages while morphological features and part-of-speech classes give the properties of words. We argue that this theory is a good basis for crosslinguistically consistent annotation of typologically diverse languages in a way that supports computational natural language understanding as well as broader linguistic studies.
9.	Dürlich, Luise, et al. (author) On the Concept of Resource-Efficiency in NLP 2023 In: Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). ; , s. 135-145 Conference paper (peer-reviewed)abstract Resource-efficiency is a growing concern in the NLP community. But what are the resources we care about and why? How do we measure efficiency in a way that is reliable and relevant? And how do we balance efficiency and other important concerns? Based on a review of the emerging literature on the subject, we discuss different ways of conceptualizing efficiency in terms of product and cost, using a simple case study on fine-tuning and knowledge distillation for illustration. We propose a novel metric of amortized efficiency that is better suited for life-cycle analysis than existing metrics.
10.	Dürlich, Luise, et al. (author) On the Concept of Resource-Efficiency in NLP 2023 In: Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). ; , s. 135-145 Conference paper (peer-reviewed)
11.	Dürlich, Luise, et al. (author) What Causes Unemployment? : Unsupervised Causality Mining from Swedish Governmental Reports 2023 In: Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). - : Association for Computational Linguistics. - 9781959429739 ; , s. 25-29 Conference paper (peer-reviewed)abstract Extracting statements about causality from text documents is a challenging task in the absence of annotated training data. We create a search system for causal statements about user-specified concepts by combining pattern matching of causal connectives with semantic similarity ranking, using a language model fine-tuned for semantic textual similarity. Preliminary experiments on a small test set from Swedish governmental reports show promising results in comparison to two simple baselines.
12.	Gogoulou, Evangelia, et al. (author) A Study of Continual Learning Under Language Shift 2023 Other publication (pop. science, debate, etc.)abstract The recent increase in data and model scale for language model pre-training has led to huge training costs. In scenarios where new data become available over time, updating a model instead of fully retraining it would therefore provide significant gains. In this paper, we study the benefits and downsides of updating a language model when new data comes from new languages - the case of continual learning under language shift. Starting from a monolingual English language model, we incrementally add data from Norwegian and Icelandic to investigate how forward and backward transfer effects depend on the pre-training order and characteristics of languages, for different model sizes and learning rate schedulers. Our results show that, while forward transfer is largely positive and independent of language order, backward transfer can be either positive or negative depending on the order and characteristics of new languages. To explain these patterns we explore several language similarity metrics and find that syntactic similarity appears to have the best correlation with our results.
13.	Hershcovich, Daniel, et al. (author) Kopsala : Transition-Based Graph Parsing via Efficient Training and Effective Encoding 2020 In: 16th International Conference on Parsing Technologies and IWPT 2020 Shared Task on Parsing Into Enhanced Universal Dependencies. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781952148118 ; , s. 236-244 Conference paper (peer-reviewed)abstract We present Kopsala, the Copenhagen-Uppsala system for the Enhanced Universal Dependencies Shared Task at IWPT 2020. Our system is a pipeline consisting of off-the-shelf models for everything but enhanced graph parsing, and for the latter, a transition-based graph parser adapted from Che et al. (2019). We train a single enhanced parser model per language, using gold sentence splitting and tokenization for training, and rely only on tokenized surface forms and multilingual BERT for encoding. While a bug introduced just before submission resulted in a severe drop in precision, its post-submission fix would bring us to 4th place in the official ranking, according to average ELAS. Our parser demonstrates that a unified pipeline is elective for both Meaning Representation Parsing and Enhanced Universal Dependencies.
14.	Karlgren, Jussi, et al. (author) ELOQUENT CLEF Shared Tasks for Evaluation of Generative Language Model Quality 2024 In: Lecture Notes in Computer Science. - : Springer Science and Business Media Deutschland GmbH. - 0302-9743 .- 1611-3349. ; 14612 LNCS, s. 459-465 Journal article (peer-reviewed)abstract ELOQUENT is a set of shared tasks for evaluating the quality and usefulness of generative language models. ELOQUENT aims to bring together some high-level quality criteria, grounded in experiences from deploying models in real-life tasks, and to formulate tests for those criteria, preferably implemented to require minimal human assessment effort and in a multilingual setting. The selected tasks for this first year of ELOQUENT are (1) probing a language model for topical competence; (2) assessing the ability of models to generate and detect hallucinations; (3) assessing the robustness of a model output given variation in the input prompts; and (4) establishing the possibility to distinguish human-generated text from machine-generated text.
15.	Kulmizev, Artur, et al. (author) Do Neural Language Models Show Preferences for Syntactic Formalisms? 2020 In: 58Th Annual Meeting Of The Association For Computational Linguistics (Acl 2020). - : ASSOC COMPUTATIONAL LINGUISTICS-ACL. - 9781952148255 ; , s. 4077-4091 Conference paper (peer-reviewed)abstract Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces. However, such studies often suffer from limited scope by focusing on a single language and a single linguistic formalism. In this study, we aim to investigate the extent to which the semblance of syntactic structure captured by language models adheres to a surface-syntactic or deep syntactic style of analysis, and whether the patterns are consistent across different languages. We apply a probe for extracting directed dependency trees to BERT and ELMo models trained on 13 different languages, probing for two different syntactic annotation styles: Universal Dependencies (UD), prioritizing deep syntactic relations, and Surface-Syntactic Universal Dependencies (SUD), focusing on surface structure. We find that both models exhibit a preference for UD over SUD - with interesting variations across languages and layers - and that the strength of this preference is correlated with differences in tree shape.
16.	Kulmizev, Artur, 1989-, et al. (author) Investigating UD Treebanks via Dataset Difficulty Measures 2023 In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. - Dubrovnik, Croatia : Association for Computational Linguistics. ; , s. 1076-1089 Conference paper (peer-reviewed)abstract Treebanks annotated with Universal Dependencies (UD) are currently available for over 100 languages and are widely utilized by the community. However, their inherent characteristics are hard to measure and are only partially reflected in parser evaluations via accuracy metrics like LAS. In this study, we analyze a large subset of the UD treebanks using three recently proposed accuracy-free dataset analysis methods: dataset cartography, ?-information, and minimum description length. Each method provides insights about UD treebanks that would remain undetected if only LAS was considered. Specifically, we identify a number of treebanks that, despite yielding high LAS, contain very little information that is usable by a parser to surpass what can be achieved by simple heuristics. Furthermore, we make note of several treebanks that score consistently low across numerous metrics, indicating a high degree of noise or annotation inconsistency present therein.
17.	Kulmizev, Artur, et al. (author) Schrödinger's tree : On syntax and neural language models 2022 In: Frontiers in Artificial Intelligence. - : Frontiers Media S.A.. - 2624-8212. ; 5 Journal article (peer-reviewed)abstract In the last half-decade, the field of natural language processing (NLP) hasundergone two major transitions: the switch to neural networks as the primarymodeling paradigm and the homogenization of the training regime (pre-train, then fine-tune). Amidst this process, language models have emergedas NLP’s workhorse, displaying increasingly fluent generation capabilities andproving to be an indispensable means of knowledge transfer downstream.Due to the otherwise opaque, black-box nature of such models, researchershave employed aspects of linguistic theory in order to characterize theirbehavior. Questions central to syntax—the study of the hierarchical structureof language—have factored heavily into such work, shedding invaluableinsights about models’ inherent biases and their ability to make human-likegeneralizations. In this paper, we attempt to take stock of this growing body ofliterature. In doing so, we observe a lack of clarity across numerous dimensions,which influences the hypotheses that researchers form, as well as theconclusions they draw from their findings. To remedy this, we urge researchersto make careful considerations when investigating coding properties, selectingrepresentations, and evaluating via downstream tasks. Furthermore, we outlinethe implications of the different types of research questions exhibited in studieson syntax, as well as the inherent pitfalls of aggregate metrics. Ultimately, wehope that our discussion adds nuance to the prospect of studying languagemodels and paves the way for a less monolithic perspective on syntax in thiscontext.
18.	Kulmizev, Artur (author) The Search for Syntax : Investigating the Syntactic Knowledge of Neural Language Models Through the Lens of Dependency Parsing 2023 Doctoral thesis (other academic/artistic)abstract Syntax — the study of the hierarchical structure of language — has long featured as a prominent research topic in the field of natural language processing (NLP). Traditionally, its role in NLP was confined towards developing parsers: supervised algorithms tasked with predicting the structure of utterances (often for use in downstream applications). More recently, however, syntax (and syntactic theory) has factored much less into the development of NLP models, and much more into their analysis. This has been particularly true with the nascent relevance of language models: semi-supervised algorithms trained to predict (or infill) strings given a provided context. In this dissertation, I describe four separate studies that seek to explore the interplay between syntactic parsers and language models upon the backdrop of dependency syntax. In the first study, I investigate the error profiles of neural transition-based and graph-based dependency parsers, showing that they are effectively homogenized when leveraging representations from pre-trained language models. Following this, I report the results of two additional studies which show that dependency tree structure can be partially decoded from the internal components of neural language models — specifically, hidden state representations and self-attention distributions. I then expand on these findings by exploring a set of additional results, which serve to highlight the influence of experimental factors, such as the choice of annotation framework or learning objective, in decoding syntactic structure from model components. In the final study, I describe efforts to quantify the overall learnability of a large set of multilingual dependency treebanks — the data upon which the previous experiments were based — and how it may be affected by factors such as annotation quality or tokenization decisions. Finally, I conclude the thesis with a conceptual analysis that relates the aforementioned studies to a broader body of work concerning the syntactic knowledge of language models.
19.	Lindqvist, Ellinor, et al. (author) Low-Resource Techniques for Analysing the Rhetorical Structure of Swedish Historical Petitions 2023 In: <em>RESOURCEFUL 2023 - Workshop on Resources and Representations for Under-Resourced Languages and Domains, Proceedings of the 2nd</em>. - : Association for Computational Linguistics. ; , s. 132-139 Conference paper (peer-reviewed)abstract Natural language processing techniques can be valuable for improving and facilitating historical research. This is also true for the analysis of petitions, a source which has been relatively little used in historical research. However, limited data resources pose challenges for mainstream natural language processing approaches based on machine learning. In this paper, we explore methods for automatically segmenting petitions according to their rhetorical structure. We find that the use of rules, word embeddings, and especially keywords can give promising results for this task.
20.	Lindqvist, Ellinor, et al. (author) Low-Resource Techniques for Analysing the Rhetorical Structure of Swedish Historical Petitions 2023 In: Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). ; , s. 132-139 Conference paper (peer-reviewed)
21.	Lindqvist, Ellinor, et al. (author) To the Most Gracious Highness, from Your Humble Servant : Analysing Swedish 18th Century Petitions Using Text Classification 2022 In: Proceedings of the 6th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature. ; , s. 53-64 Conference paper (peer-reviewed)abstract Petitions are a rich historical source, yet they have been relatively little used in historical research. In this paper, we aim to analyse Swedish texts from around the 18th century, and petitions in particular, using automatic means of text classification. We also test how text pre-processing and different feature representations affect the result, and we examine feature importance for our main class of interest – petitions. Our experiments show that the statistical algorithms NB, RF, SVM, and kNN are indeed very able to classify different genres of historical text. Further, we find that normalisation has a positive impact on classification, and that content words are particularly informative for the traditional models. A fine-tuned BERT model, fed with normalised data, outperforms all other classification experiments with a macro average F1 score at 98.8. However, using less computationally expensive methods, including feature representation with word2vec, fastText embeddings or even TF-IDF values, with a SVM classifier also show good results for both unnormalised and normalised data. In the feature importance analysis, where we obtain the features most decisive for the classification models, we find highly relevant characteristics of the petitions, namely words expressing signs of someone inferior addressing someone superior.
22.	Lundin, Emil, 1985- (author) Media Arabic Grammar and Semantics. Clauses and non-core elements : A corpus investigation of print hard news 2021 Doctoral thesis (other academic/artistic)abstract ”Media Arabic” is taught on universities all over the world and its understanding ranks among the top-reasons for students to pursue Arabic studies. The coursebooks on ”Media Arabic” focus on print hard news and tacitly assume the existence of an Arabic journalese. Previous research on Arabic newspaper language is scarce.Focusing on the morphosyntax and semantics of subordinate and peripheral syntagms within the sentence of a Media Arabic corpus, this thesis aims to quantitatively answer the question of what characterizes Media Arabic from a linguistic perspective.This study is a corpus investigation of print hard news based on a corpus of 35,000 words, or 1,144 sentences, taken from the Egyptian newspaper al-Ahram, September 2014. The core focus is on the morphosyntax, and logico-semantic relations of peripheral nominal and clausal non-core elements to their matrices or heads. In laymen terms: the grammar and meaning of adverbs, prepositional phrases, and subordinate clauses.Following a description of the corpus, journalism and journalese, Media Arabic, their contextualization, and relevant theoretical preliminaries, peculiarities, and the methodology, the analysis centres around the morphosyntax and semantics of the attested 1,144 main clauses, 891 complement main clauses and 828 complementizers, 869 non-nominalized non-main clauses, 642 non-deverbalized relative clauses, 473 adverbs (including peripheral accusative nouns), 461 peripheral and predicative participles (including nominalized relative clauses), 40 peripheral accusative verbal nouns, and 6391 prepositional phrases. Backed by detailed quantitative data, the study concludes that the language of Arabic news is characterized linguistically by a high degree of formalization.
23.	Löwenmark, Karl, 1994-, et al. (author) Labelling of Annotated Condition Monitoring Data Through Technical Language Processing 2023 In: Proceedings of the Annual Conference of the PHM Society 2023. - : The Prognostics and Health Management Society. Conference paper (peer-reviewed)abstract We propose a novel approach to facilitate supervised fault diagnosis on unlabelled but annotated industry datasets using human-centric technical language processing and weak supervision. Fault diagnosis through Condition Monitoring (CM) is vital for high safety and resource efficiency in the green transition and digital transformation of the process industry. Learning-based Intelligent Fault Diagnosis (IFD) methods are required to automate maintenance decisions and improve decision support for analysts. A major challenge is the lack of labelled industry datasets, limiting supervised IFD research to lab datasets. However, features learned from lab environments generalise poorly to field environments due to different signal distributions, artificial induction or acceleration of lab faults, and lab set-up properties such as average frequency profiles affecting learned features. In this study, we investigate how the unstructured free text fault annotations and maintenance work orders that are present in many industrial CM systems can be used for IFD through technical language processing, based on recent advances in natural language supervision. We introduce two distinct pipelines, one based on contrastive pre-training on large datasets, and one based on a small-data human-centric approach with unsupervised clustering methods. Finally, we showcase one example of the small-data fault classification implementation on a CM industry dataset with a SentenceBERT language model, kMeans clustering, and conventional signal processing methods. Fault class imbalance and time-shift uncertainty is overcome with weak supervision through aggregates of features, and human-centric clustering is used to integrate technical knowledge with the annotation-based fault classes. We show that our model can separate cable and sensor fault recordings from bearing-related fault recordings with an F1-score of 93. To our knowledge, this is the first system to classify faults in field industry CM data based only on associated unstructured fault annotations.
24.	Löwenmark, Karl, et al. (author) Processing of Condition Monitoring Annotations with BERT and Technical Language Substitution: A Case Study 2022 In: Proceedings of the 7th European Conference of the Prognostics and Health Management Society 2022. - : PHM Society. - 9781936263363 ; , s. 306-314 Conference paper (peer-reviewed)abstract Annotations in condition monitoring systems contain information regarding asset history and fault characteristics in the form of unstructured text that could, if unlocked, be used for intelligent fault diagnosis. However, processing these annotations with pre-trained natural language models such as BERT is problematic due to out-of-vocabulary (OOV) technical terms, resulting in inaccurate language embeddings. Here we investigate the effect of OOV technical terms on BERT and SentenceBERT embeddings by substituting technical terms with natural language descriptions. The embeddings were computed for each annotation in a pre-processed corpus, with and without substitution. The K-Means clustering score was calculated on sentence embeddings, and a Long Short-Term Memory (LSTM) network was trained on word embeddings with the objective to recreate the output from a keyword-based annotation classifier. The K-Means score for SentenceBERT annotation embeddings improved by 40% at seven clusters by technical language substitution, and the labelling capacityof the BERT-LSTM model was improved from 88.3 to 94.2%. These results indicate that the substitution of OOV technical terms can improve the representation accuracy of the embeddings of the pre-trained BERT and SentenceBERT models, and that pre-trained language models can be used to process technical language.
25.	Nivre, Joakim, 1962- (author) Multilingual Dependency Parsing from Universal Dependencies to Sesame Street 2020 In: Text, Speech, and Dialogue (TSD 2020). - Cham : SPRINGER INTERNATIONAL PUBLISHING AG. - 9783030583231 - 9783030583224 ; , s. 11-29 Conference paper (peer-reviewed)abstract Research on dependency parsing has always had a strong multilingual orientation, but the lack of standardized annotations for a long time made it difficult both to meaningfully compare results across languages and to develop truly multilingual systems. The Universal Dependencies project has during the last five years tried to overcome this obstacle by developing cross-linguistically consistent morphosyntactic annotation for many languages. During the same period, dependency parsing (like the rest of NLP) has been transformed by the adoption of continuous vector representations and neural network techniques. In this paper, I will introduce the framework and resources of Universal Dependencies, and discuss advances in dependency parsing enabled by these resources in combination with deep learning techniques, ranging from traditional word and character embeddings to deep contextualized word representations like ELMo and BERT.
26.	Nivre, Joakim, 1962-, et al. (author) Nucleus Composition in Transition-based Dependency Parsing 2022 In: Computational linguistics - Association for Computational Linguistics (Print). - : MIT Press Journals. - 0891-2017 .- 1530-9312. ; 48:4, s. 849-886 Journal article (peer-reviewed)abstract Dependency-based approaches to syntactic analysis assume that syntactic structure can be analyzed in terms of binary asymmetric dependency relations holding between elementary syntactic units. Computational models for dependency parsing almost universally assume that an elementary syntactic unit is a word, while the influential theory of Lucien Tesnière instead posits a more abstract notion of nucleus, which may be realized as one or more words. In this article, we investigate the effect of enriching computational parsing models with a concept of nucleus inspired by Tesnière. We begin by reviewing how the concept of nucleus can be defined in the framework of Universal Dependencies, which has become the de facto standard for training and evaluating supervised dependency parsers, and explaining how composition functions can be used to make neural transition-based dependency parsers aware of the nuclei thus defined. We then perform an extensive experimental study, using data from 20 languages to assess the impact of nucleus composition across languages with different typological characteristics, and utilizing a variety of analytical tools including ablation, linear mixed-effects models, diagnostic classifiers, and dimensionality reduction. The analysis reveals that nucleus composition gives small but consistent improvements in parsing accuracy for most languages, and that the improvement mainly concerns the analysis of main predicates, nominal dependents, clausal dependents, and coordination structures. Significant factors explaining the rate of improvement across languages include entropy in coordination structures and frequency of certain function words, in particular determiners. Analysis using dimensionality reduction and diagnostic classifiers suggests that nucleus composition increases the similarity of vectors representing nuclei of the same syntactic type.
27.	Nivre, Joakim, 1962-, et al. (author) Universal Dependencies v2 : An Evergrowing Multilingual Treebank Collection 2020 In: Proceedings of the 12th Language Resources and Evaluation Conference. - 9791095546344 ; , s. 4034-4043 Conference paper (peer-reviewed)abstract Universal Dependencies is an open community effort to create cross-linguistically consistent treebank annotation for many languages within a dependency-based lexicalist framework. The annotation consists in a linguistically motivated word segmentation; a morphological layer comprising lemmas, universal part-of-speech tags, and standardized morphological features; and a syntactic layer focusing on syntactic relations between predicates, arguments and modifiers. In this paper, we describe version 2 of the guidelines (UD v2), discuss the major changes from UD v1 to UD v2, and give an overview of the currently available treebanks for 90 languages.
28.	Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023) 2023 Editorial proceedings (peer-reviewed)
29.	Ravishankar, Vinit, et al. (author) Attention Can Reflect Syntactic Structure (If You Let It) 2021 In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. - Stroudsburg, PA, USA : Association for Computational Linguistics. - 9781954085022 ; , s. 3031-3045 Conference paper (peer-reviewed)abstract Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism. However, much of such work focused almost exclusively on English - a language with rigid word order and a lack of inflectional morphology. In this study, we present decoding experiments for multilingual BERT across 18 languages in order to test the generalizability of the claim that dependency syntax is reflected in attention patterns. We show that full trees can be decoded above baseline accuracy from single attention heads, and that individual relations are often tracked by the same heads across languages. Furthermore, in an attempt to address recent debates about the status of attention as an explanatory mechanism, we experiment with fine-tuning mBERT on a supervised parsing objective while freezing different series of parameters. Interestingly, in steering the objective to learn explicit linguistic structure, we find much of the same structure represented in the resulting attention patterns, with interesting differences with respect to which parameters are frozen.
30.	Savary, Agata, et al. (author) PARSEME Meets Universal Dependencies : Getting on the Same Page in Representing Multiword Expressions 2023 In: Northern European Journal of Language Technology (NEJLT). - : Linköping University Electronic Press. - 2000-1533. ; 9:1 Journal article (peer-reviewed)abstract Multiword expressions (MWEs) are challenging and pervasive phenomena whose idiosyncratic properties show notably at the levels of lexicon, morphology, and syntax. Thus, they should best be annotated jointly with morphosyntax. In this position paper we discuss two multilingual initiatives, Universal Dependencies and PARSEME, addressing these annotation layers in cross-lingually unified ways. We compare the annotation principles of these initiatives with respect to MWEs, and we put forward a roadmap towards their gradual unification. The expected outcomes are more consistent treebanking and higher universality in modeling idiosyncrasy.
31.	Tang, Gongbo, et al. (author) Revisiting Negation in Neural Machine Translation 2021 In: Transactions of the Association for Computational Linguistics. - : MIT Press. - 2307-387X. ; 9, s. 740-755 Journal article (peer-reviewed)abstract In this paper, we evaluate the translation of negation both automatically and manually, in English–German (EN–DE) and English– Chinese (EN–ZH). We show that the ability of neural machine translation (NMT) models to translate negation has improved with deeper and more advanced networks, although the performance varies between language pairs and translation directions. The accuracy of manual evaluation in EN→DE, DE→EN, EN→ZH, and ZH→EN is 95.7%, 94.8%, 93.4%, and 91.7%, respectively. In addition, we show that under-translation is the most significant error type in NMT, which contrasts with the more diverse error profile previously observed for statistical machine translation. To better understand the root of the under-translation of negation, we study the model’s information flow and training data. While our information flow analysis does not reveal any deficiencies that could be used to detect or fix the under-translation of negation, we find that negation is often rephrased during training, which could make it more difficult for the model to learn a reliable link between source and target negation. We finally conduct intrinsic analysis and extrinsic probing tasks on negation, showing that NMT models can distinguish negation and non-negation tokens very well and encode a lot of information about negation in hidden states but nevertheless leave room for improvement.

Skapa referenser, mejla, bekava och länka

Permalink

Träfflista för sökning "WFRF:(Nivre Joakim 1962 ) srt2:(2020-2024)"

Refine your search

Year