The Influence of Raters’ Topic Familiarity on Rater Severity in a Teaching Simulation Test for International Teaching Assistants

• 발행기관: 한국영어평가학회
• 저자명: 원용국(Yongkook Won)
• 간행물 정보: 『English Language Assessment』Vol.15 No.2, 235~254쪽, 전체 20쪽
• 주제분류: 인문학 > 언어학
• 파일형태: PDF
• 발행일자: 2020.12.30

영문 초록

Test score variability associated with characteristics of raters is considered as a construct-irrelevant variance in test scores. Several studies have tried to identify the effect of rater characteristics on test score variances in performance-based oral communication tests. However, little attention has been paid to the effect of raters’ familiarity with speech topics in an oral communication test. This study investigates how raters’ prior knowledge of test takers’ presentation topics affects rater severity in a teaching simulation test for international teaching assistants. Rating data of 162 test takers by 15 raters at a U.S. university were analysed by using a many-facet Rasch measurement (MFRM) analysis. A three-facet (i.e., examinee, rater, and topic familiarity) rating scale model with MFRM framework was fitted to estimate the magnitude of topic familiarity on rating severity. It was found that raters were harsher when they were less familiar with test takers’ speech topic. The difficulty difference between high (or mid) and low topic familiarity was 0.56 (or 0.53) in logits. There was found to be no interaction between raters’ topic familiarity and their background (NS vs NNS). The results of this study suggest that solid rater training is needed to reduce construct-irrelevant variances in test scores.

목차

I. INTRODUCTION

II. LITERATURE REVIEW

III. RESEARCH METHOD

IV. RESULTS

V. DISCUSSION & CONCLUSION

REFERENCES

키워드

topic familiarity rater severity oral communication test MFRM