Harnessing AI for CEFR-Aligned Writing Assessment: A Study of a Customised GPT

Abstract

Artificial intelligence (AI) tools, specifically ChatGPT, has garnered considerable interest in the field of English as a Second Language (ESL) as demand grows in recent times. The introduction of ChatGPT has transformed the assessment landscape, thus marking the beginning of a new era in AI-assisted assessment. While existing research has largely addressed the comparison of ChatGPT and human raters with regards to assessment, limited study has focused on customised GPTs and the context of Common European Framework of References for Languages (CEFR). To address this gap, this paper investigates the effectiveness of a customised GPT as a formative and summative assessment tool in a CEFR-aligned written task pitched at B2 CEFR level. Adopting a quantitative research design, the respondents’ attitudes towards using the GPT for formative assessment were examined through a questionnaire administered to 31 English teachers. In parallel, the scores assigned by the GPT and the teachers for the same writing tasks were compared via inter-rater reliability analysis. Findings revealed that the respondents hold a generally positive view regarding the effectiveness of GPT in providing formative feedback. However, the results also indicated that the GPT demonstrates a moderate level of agreement with the teacher scores in most assessment constructs. The data further emphasized the need of prompt engineering in developing the GPT to be an effective formative and summative assessment assistant for teachers. This paper concludes by discussing the practical implications of employing the customised GPT in assessment, thereby contributing to the discourse on AI-assisted assessment.