A Comparative Study Between Human Raters and ChatGPT in L2 Speaking Performance Assessment

• 영문명: A Comparative Study Between Human Raters and ChatGPT in L2 Speaking Performance Assessment
• 발행기관: 팬코리아영어교육학회
• 저자명: 박소윤(So Yun Park) 신유선(Yousun Shin)
• 간행물 정보: 『영어교육연구』제35권 4호, 63~83쪽, 전체 21쪽
• 주제분류: 사회과학 > 교육학
• 파일형태: PDF
• 발행일자: 2023.12.31

영문 초록

This study aimed to compare L2 learners’ oral performance scores by human raters with ChatGPT in multiple scoring categories. It also examined score consistency within each group. A total of 39 university students participated in this study, completing various speaking tasks including a self- introduction and two picture descriptions. Statistical analyses using ANOVA and the Multi-Facets Rasch Model (MFRM) were used to analyze the scoring data from both human raters and ChatGPT. The ANOVA results revealed significant mean differences between the two rater groups across all three speaking tasks and all scoring categories (content/organization, vocabulary, collocation use, grammar), except for the category of collocation use in self-introduction. While participants’ vocabulary and collocation use showed gradual improvement in speaking tasks, substantial gains in content/organization and grammar were not observed due to time limitations. Furthermore, the MFRM results indicated that ChatGPT demonstrated higher sensitivity in discerning score patterns across different task types and scoring categories compared to the human raters. Overall, human raters consistently assigned higher scores across all task types. Although both groups scored differently, they maintained consistency in scoring, indicating agreement in their evaluation standards. Pedagogical implications pertaining to the potential advantages of integrating AI-based technology into performance-based assessments were further discussed. (201 words)

목차

I. INTRODUCTION

II. LITERATURE REVIEW

III. METHODS

IV. RESULTS

V. DISCUSSION AND CONCLUSION

REFERENCES

키워드

speaking performance assessment human raters ChatGPT ANOVA MFRM