All that glitters . . . : Interannotator agreement in natural language processing

• Evaluation has emerged as a central concern in natural language processing (NLP) over the last few decades. Evaluation is done against a gold standard, a manually linguistically annotated dataset, which is assumed to provide the ground truth against which the accuracy of the NLP system can be assessed automatically. In this article, some methodological questions in connection with the creation of gold standard datasets are discussed, in particular (non-)expectations of linguistic expertise in annotators and the interannotator agreement measure standardly but unreflectedly used as a kind of quality index of NLP gold standards.

Ämnesord

NATURVETENSKAP  -- Data- och informationsvetenskap -- Språkteknologi (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Language Technology (hsv//eng)

HUMANIORA  -- Språk och litteratur -- Jämförande språkvetenskap och allmän lingvistik (hsv//swe)

HUMANITIES  -- Languages and Literature -- General Language Studies and Linguistics (hsv//eng)

Nyckelord

Evaluation

natural language processing

interannotator agreement

annotation

gold standard

Publikations- och innehållstyp

ref (ämneskategori)

art (ämneskategori)