Automatic Evaluation of Robustness and Degradation in Tagging and Parsing

• We address the topic of automatic evaluationof robustness and performance degradation inparsing systems. We focus on one aspect of robustness, namely ill-formed sentences and the impact of spelling errors on the different componentsof a parsing system. We propose anautomated framework to evaluate robustness, where ill-formed and noisy data is introducedusing an automatic tool and fed to the parsingsystem. With increasing levels of noise, the performanceof a system will inevitably degrade, and the question is to what extent? The experiments show a graceful degradation in performancefor both state-of-the-art taggers usedand a Swedish shallow parser. The automatednature of the evaluation allows easy and reproducibleevaluation of the individual componentsof a parsing system.

Nyckelord

automatic evaluation

robustness

spelling errors

tagging

shallow parsing

TECHNOLOGY

TEKNIKVETENSKAP

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)