Abstract
Ensuring high-quality career guidance in higher education is challenging due to variability in teachers’ experience and lack of feedback mechanisms. This study develops an AI-powered quality inspection system for teachers’ responses in student career planning scenarios. We followed a five-step closed-loop design: (1) constructing typical dialogue scenarios (academic learning, professional development, and employment guidance), (2) collecting simulated teacher–student dialogues (8 recorded role-play sessions, avg. 3.61 minutes), (3) designing multi-level quality check rules for compliant and non-compliant responses, (4) employing a large language model on the Yunzhisheng platform to automatically evaluate each teacher response against the rules, and (5) analyzing results to refine the rules. Initial results showed large disparities: the professional development scenario achieved a 71.88% compliance execution rate and highest score, whereas the learning scenario had only 7.5% execution and scored zero. Overall, only 25% of dialogues met quality standards. After iterative rule optimization (adding specialized criteria and adjusting thresholds), the pass rate doubled to 50%, compliance detection accuracy rose from 65% to 90%, and violation recall from 70% to 95%. The system effectively flagged generic or misleading advice and recognized improved, targeted guidance. This study demonstrates a feasible approach to standardize career guidance dialogues using AI, providing data-driven feedback to educators. It offers an innovative model for intelligent quality assurance in education, transforming implicit expert knowledge into explicit rules and closing the loop for continuous improvement in teacher training.

This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2025 Feng Ju, Yingliang Wan, Xia Fu, Xinrong Zhu, Yiyi Yin (Author)