Sökning: onr:"swepub:oai:DiVA.org:uu-518286" >
Using Wikidata for ...
Using Wikidata for Enhancing Compositionality in Pre-trained Language Models
-
- Beloucif, Meriem (författare)
- Uppsala universitet,Institutionen för lingvistik och filologi,Computational Linguistics
-
- Bansal, Mihir (författare)
- Carnegie Mellon University
-
- Biemann, Chris (författare)
- Hamburg University,Language Technology
-
(creator_code:org_t)
- INCOMA, 2023
- 2023
- Engelska.
-
Ingår i: Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing. - : INCOMA. - 9789544520922 ; , s. 170-178
- Relaterad länk:
-
https://aclanthology...
-
visa fler...
-
https://urn.kb.se/re...
-
visa färre...
Abstract
Ämnesord
Stäng
- One of the many advantages of pre-trained language models (PLMs) such as BERT and RoBERTa is their flexibility and contextual nature. These features give PLMs strong capabilities for representing lexical semantics. However, PLMs seem incapable of capturing high-level semantics in terms of compositionally. We show that when augmented with the relevant semantic knowledge, PMLs learn to capture a higher degree of lexical compositionality. We annotate a large dataset from Wikidata highlighting a type of semantic inference that is easy for humans to understand but difficult for PLMs, like the correlation between age and date of birth. We use this resource for finetuning DistilBERT, BERT large and RoBERTa. Our results show that the performance of PLMs against the test data continuously improves when augmented with such a rich resource. Our results are corroborated by a consistent improvement over most GLUE benchmark natural language understanding tasks.
Ämnesord
- NATURVETENSKAP -- Data- och informationsvetenskap -- Språkteknologi (hsv//swe)
- NATURAL SCIENCES -- Computer and Information Sciences -- Language Technology (hsv//eng)
Nyckelord
- BERT
- Pretrained Language Models
- Computational Linguistics
- Semantics in LLM
- Datorlingvistik
- Computational Linguistics
Publikations- och innehållstyp
- ref (ämneskategori)
- kon (ämneskategori)
Hitta via bibliotek
Till lärosätets databas