Investigating ChatGPT’s phonology problem-solving abilities through reasoning with varying custom instructions

• 발행기관: 경희대학교 언어정보연구소
• 저자명: Hyesun Cho Sunwoo Park Sanghoun Song Eunjin Oh
• 간행물 정보: 『언어연구』제42권 제1호, 53~93쪽, 전체 41쪽
• 주제분류: 인문학 > 언어학
• 파일형태: PDF
• 발행일자: 2025.03.31

국문 초록

In this paper, we investigate ChatGPT’s ability to solve phonological problems with varying custom instructions. It is known that ChatGPT has difficulty solving unfamiliar novel problems, and phonology provides a good test case for evaluating its reasoning abilities when faced with such challenges. We designed phonological problems using data or rules that are not observed in the phonology literature. We varied model versions (GPT-4 and GPT-4o) and custom instructions by varying the levels of knowledge: no custom instructions, beginner(‘Student’), and expert (‘Professor’) levels. Four novel questions were created by modifying the following phonological data: (1) Korean fricative palatalization, (2)Spanish intervocalic lenition, (3) Rule ordering in Canadian English, and (4) English syllabification. ChatGPT exhibits some level of reasoning capabilities, with test scores ranging from 27% to 75%. GPT-4o performed better than GPT-4. However, it made errors when dealing with novel patterns or rules. The effects of varying custom instructions were present but less clear and inconsistent. Our results suggest that,depending on the model version and customization, GPT can learn phonologically unnatural processes through reasoning.

목차

1. Introduction

2. Experiment

3. Results

4. Summary and discussion

5. Conclusion

References

키워드

ChatGPT-4 ChatGPT-4o phonology reasoning custom instructions