Evaluating the Educational Impact of an LLM Virtual Assistant in Video-Based Learning

This study presents a controlled trial with 191 upper secondary and vocational students examining whether an LLM-based pedagogical agent, combining a consultative chat, a proactive evaluative chat and automatically generated study materials, improves learning from instructional video. Access to the agent alone did not raise average learning gains; instead, outcomes depended on the cognitive depth of student-AI interaction, measured through a novel Weighted Semantic Depth Index derived from chat logs. The agent narrowed rather than widened prior-attainment gaps. Status: submitted for publication.

Recommended citation: Scielzo-Ortiz, F., et al. (2026). "Evaluating the educational impact of an LLM virtual assistant in video-based learning." (Submitted to Journal of Computer Assisted Learning).