LLM 기반 RAG 시스템 성능 평가 및 실증 분석:리크루팅 상담서비스를 중심으로

• 영문명: A Study on Performance Evaluation and Empirical Analysis of LLM-based RAG Systems: Focusing on Recruitment Counseling Services
• 발행기관: 한국IT서비스학회
• 저자명: 민윤정(Yunjeung Min) 안재경(Jaekyoung Ahn) 임종덕(Jongduck Lim)
• 간행물 정보: 『한국IT서비스학회지』제24권 제4호, 175~188쪽, 전체 14쪽
• 주제분류: 경제경영 > 경영학
• 파일형태: PDF
• 발행일자: 2025.08.31

국문 초록

This study comprehensively evaluated and empirically analyzed the performance of Retrieval-Augmented Generation (RAG) systems based on Large Language Models (LLMs), with a specific focus on recruitment counseling services. Specifically, it systematically compared the performance of LLaMA 3.1 70B and GPT-4o models using the comprehensive RAGAS evaluation framework. The research methodology involved conducting extensive experiments based on various employment-related data sources, including job application documents, vocational psychological tests, detailed job care reports, vocational training courses, and National Tomorrow Learning Card operating regulations. Performance evaluation utilized five key indicators: Context Precision, Faithfulness, Answer Similarity, Context Recall, and Answer Relevancy. The experimental results showed that both models demonstrated excellent performance in Context Precision (0.97), but LLaMA 3.1 significantly outperformed GPT-4o in terms of Faithfulness metrics. Conversely, GPT-4o showed superior performance in Answer Similarity and Answer Relevancy compared to LLaMA 3.1. Data type-specific analysis revealed that Context Precision was perfect (1.0) for both standard text and meta data, but notable differences existed in other performance indicators. This study emphasizes the critical importance of model selection in designing LLM-based RAG systems and suggests appropriate model selection according to specific service objectives. The research findings can contribute to improving the quality of AI-based recruitment counseling services and serve as valuable reference material for designing and implementing RAG systems across various industries in the future.

목차

1. 서론

2. 이론적 배경

3. 연구 방법

4. 분석 결과

5. 결론 및 제언

참고문헌

키워드

sLLM(small Large Languge Model) RAGAS Retrieval Vector Database Embedding Recruitment Counseling Services Chunking RAG