LLM-as-a-Judge 기반 지각 지표의 활용에 대한 탐색적 분석: 반려동물 CQA 플랫폼에서의 답변 채택 여부를 중심으로

초록

Answer quality evaluation in online community question answering (CQA) platforms has traditionally relied on indicators evaluating answer quality. In particular, both text-based indicators, which capture the objective quality of text, and perceptual indicators, which reflect perceived subjective quality, have been utilized. However, perceptual indicators are costly to collect and prone to quality issues. This study examines the applicability of using LLM-as-a-Judge–based perceptual indicators as an alternative in a Pet CQA platform. Machine learning results show that models incorporating both text-based and LLM-as-a-Judge–based perceptual indicators outperform the baseline in classifying answer acceptance. SHAP, LIME, and counterfactual analyses reveal that perceptual indicators play a central role, demonstrating that understandability and question relevance are key factors influencing answer acceptance, while text-based indicators have limited impact. Comparative analyses further show that accepted answers score higher across all text-based and perceptual indicators. This study contributes to the literature by confirming the applicability of LLM-as-a-Judge–based perceptual indicators in Pet CQA and identifying understandability and question relevance as key determinants of answer acceptance.

키워드