Quality Agreement on Learnersourced Multiple Choice Questions

• Learnersourcing presents an efficient and economical pathway to producing more learning content, whilst engaging students more actively and deeply in their learning. However, it also presents new challenges to solve. Most of all, how best can we manage the variance in quality of content produced. In this work, we focus on the extent to which students and teachers agree on quality of learnersourced multiple choice questions. Students (n=30) were tasked with producing six questions over three weeks of an introductory programming course as part of their assessment. They also had to review 12 questions authored by their peers over the same period using a set of principles for good questions. After this period, four teaching staff involved with the course reviewed the student questions using the same process and principles. Inter-rater reliability statistics found overall positive agreement across principles, however this dropped to weaker agreement for principles aimed at more subjective and higher order concerns of question quality and quality of question feedback.

Subject headings

SAMHÄLLSVETENSKAP  -- Utbildningsvetenskap -- Lärande (hsv//swe)

SOCIAL SCIENCES  -- Educational Sciences -- Learning (hsv//eng)

Keyword

Inter-rater Reliability

Learnersourcing

Peerwise

Publication and Content Type

ref (subject category)

kon (subject category)