Testing and Evaluation College and University Education Research-Oriented Presentation
Assessing the validity of essay marking rubrics
As English high school curricula becomes increasingly communication-oriented, it is becoming more necessary to develop university entrance tests which assess students’ ability to produce target language based on communicative goals rather than to translate between languages or select correct answers. A potential problem with these more communication-oriented test questions is they may risk sacrificing reliability for validity; however, the use of rubrics can ensure that both reliability and validity remain high (Jonsson and Svingby, 2007). This presentation looks at the results of a preliminary study to determine if a university entrance exam rubric results in high inter-rater reliability. The study looks at the test scores of three types of markers: 1) those trained to apply the rubric; 2) those who have seen the rubric but have not been trained to apply it; and 3) those who have not seen the rubric. It aims to answer the following questions: Does the rubric achieve a Cohen's kappa value greater than 0.7 for inter-rater reliability 1. between trained markers? 2. between trained and untrained markers? 3. between trained markers and markers who have not seen the rubric?
The findings of this study will interest educators involved in test and assessment design.
-
Paul is an associate professor in the Department of Clinical English at Nara Medical University, where he has been working since 2016. Paul has a law degree and an arts degree from the University of Auckland (New Zealand), and a master’s degree in applied linguistics and TESOL from the University of Leicester (UK). He is currently a PhD candidate at the Kyoto University Graduate School of Human and Environmental Studies. Paul's primary research interests are in second language vocabulary acquisition, content and language integrated learning (CLIL), English for specific purposes (ESP), and learner motivation. When he is not teaching or researching, he can be found jogging or cycling around the Kita-ku area in Osaka, eating out at one of Fukushima’s fabulous restaurants, or rocking with his band ‘The Ballbreakers’ somewhere in Osaka. Paul also loves travelling, and isn’t far away from achieving his goal of visiting all 47 Japanese prefectures (38 so far!).
-
Francesco Bolstad holds degrees in both liberal arts and STEM subjects and as a New Zealand-trained high school teacher has taught in classrooms of various kinds for over 27 years. In his current role as professor and head of Nara Medical University's Department of Clinical (DOC) English, he is involved in pedagogical and research projects which allow him to further his interests in teacher efficacy, content and language integrated learning (CLIL), collaborative approaches to teaching and learning, and vocabulary acquisition. Email: bolstad@naramed-u.ac.jp
