Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions : A Comparison of First-Year Students and Educators' Judgments

• This Complete Research paper investigates the holistic assessment of creativity in design solutions in engineering education. Design is a key element in contemporary engineering education, given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure through traditional assessment rubrics and holistic assessment approaches may be more suitable to assess creativity of design solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the item of work that they deem to have demonstrated the greatest level of a specific criterion or set of criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all items in the sample. The use of this assessment approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of first-year engineering students design solutions. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and the faculty who regularly teach first-year students. Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher to meet specific criteria. In both ACJ sessions each item of work was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student ACJ session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p < .001, and their absolute rank position, s = .553, p < .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks of students and faculty, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability which could increase the alignment of teaching and learning.

Ämnesord

SAMHÄLLSVETENSKAP  -- Utbildningsvetenskap -- Lärande (hsv//swe)

SOCIAL SCIENCES  -- Educational Sciences -- Learning (hsv//eng)

Nyckelord

Adaptive Comparative Judgement

Assessment

Design

Engineering Education

Publikations- och innehållstyp

ref (ämneskategori)

kon (ämneskategori)