Arabicized and Romanized Berber Automatic Identification

• We present an automatic language identification tool for both Arabicized Berber (Berber written in the Arabic script) and Romanized Berber (Berber written in the Latin script). The focus is on short texts (social media content). We use supervised machine learning method with character and word-based n-gram models as features. We also describe the corpora used in this paper. For both Arabicized and Romanized Berber, character-based 5-grams score the best giving an F-score of 99.50%.

Ämnesord

NATURVETENSKAP  -- Data- och informationsvetenskap -- Språkteknologi (hsv//swe)

NATURAL SCIENCES  -- Computer and Information Sciences -- Language Technology (hsv//eng)

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)