Evaluating the Effectiveness of a Japanese Language Learning Management System with an AI-Powered Tutor Utilizing LLaMA 3.1 8B

Abstract

This research aims to develop a web-based platform that integrates artificial intelligence to make Japanese language learning more accessible and interactive. Designed for travelers, overseas workers, and self-learners, Japanese Learning Management system with Artificial Intelligence Powered Tutor utilizing Llama-3.1-8B provides JLPT N5–N4 lessons with an AI tutor that offers pronunciation feedback and captions displaying Japanese scripts with romaji (the Romanized representation of Japanese words) and English translations. The system was developed using Flask, Alpine.js, Tailwind CSS, and MySQL, while the AI feature was trained with Hugging Face Transformers and PyTorch for voice recognition. To achieve these objectives, the system was developed using the Agile methodology following the Scrum framework, ensuring iterative design, testing, and integration of features. Evaluation results from 219 students, three IT professionals, and one Japanese language instructor showed high satisfaction, with overall ratings of 3.85, 4.61, and 5.00 respectively. These results confirm that the system effectively enhances pronunciation, comprehension, and learner engagement through AI integration. Future improvements include expanding datasets, training the AI on stronger hardware, refining usability, and adding higher JLPT levels and other languages to make the platform more versatile and scalable. This study presents the development and evaluation of a Japanese Language Learning Management System (LMS) integrated with an AI-powered tutor utilizing LLaMA 3.1 8B, designed to support learners preparing for JLPT N5–N4 levels. The system combines structured lessons with interactive features, including real-time pronunciation feedback, voice recognition, and captioned scripts displaying Japanese characters with romaji and English translations. Built using Flask, Alpine.js, Tailwind CSS, and MySQL, the AI component was fine-tuned with Hugging Face Transformers and PyTorch for speech processing. A quantitative descriptive design guided the evaluation, involving 219 students, three IT professionals, and one certified Japanese language instructor. Assessment instruments were based on ISO/IEC 25010 software quality standards and language learning objectives. Results indicate high user satisfaction: end users rated the system 3.85 (Agree), IT professionals scored 4.61 (Agree), and the instructor gave a perfect 5.00 (Strongly Agree). Findings confirm that the platform effectively enhances pronunciation, comprehension, and learner engagement while meeting international software quality benchmarks. Future improvements include expanding datasets, optimizing AI performance, and adding higher JLPT levels and multilingual support to ensure scalability and broader applicability.