Looking beyond scores - a study of raters and ratings of speaking

• A challenge with the paired speaking test format is scoring reliability. In order to interpret test scores from this kind of performance assessment, it is important to explore how raters reach their decisions. The present study aims to examine the rating of oral proficiency in a paired speaking test, part of a Swedish national test of English. The first group of raters are Swedish teachers of English (n = 17), who made individual ratings of six audio-recorded paired conversations in relation to national standards. In addition, two groups of external European raters (n = 14) rated the same conversations in relation to corresponding CEFR scales from the Manual, the latter with the additional aim of making a small-scale and tentative comparison between Swedish performance standards and the CEFR levels. The data consist of rater notes, summary comments and scores. Notes and summary comments are analysed to possibly identify features of the performances salient to the raters. Furthermore, scores are analysed to examine rater profiles and issues of consistency. Finally, the relationship between comments and scores is focused upon. Initial analyses of scores show that the rank ordering of performances as well as the degree of variability of ratings are very similar between the Swedish and CEFR raters. The presentation will briefly discuss the design of the study. It will then present findings from analyses of rater comments and scores. Moreover, some attention will be paid to the comparison between the Swedish national standards and the CEFR scales.

Ämnesord

SAMHÄLLSVETENSKAP  -- Utbildningsvetenskap -- Pedagogik (hsv//swe)

SOCIAL SCIENCES  -- Educational Sciences -- Pedagogy (hsv//eng)

HUMANIORA  -- Språk och litteratur -- Studier av enskilda språk (hsv//swe)

HUMANITIES  -- Languages and Literature -- Specific Languages (hsv//eng)

SAMHÄLLSVETENSKAP  -- Utbildningsvetenskap -- Didaktik (hsv//swe)

SOCIAL SCIENCES  -- Educational Sciences -- Didactics (hsv//eng)

Publikations- och innehållstyp

vet (ämneskategori)

kon (ämneskategori)