Handwriting recognition and automatic scoring for descriptive answers in Japanese language tests

This paper presents an experiment of automatically scoring handwritten descriptive answers in the trial tests for the new Japanese university entrance examination, which were made for about 120,000 examinees in 2017 and 2018. There are about 400,000 answers with more than 20 million characters. Although all answers have been scored by human examiners, handwritten characters are not labeled. We present our attempt to adapt deep neural network-based handwriting recognizers trained on a labeled handwriting dataset into this unlabeled answer set. Our proposed method combines different training strategies, ensembles multiple recognizers, and uses a language model built from a large general corpus to avoid overfitting into specific data. In our experiment, the proposed method records character accuracy of over 97% using about 2,000 verified labeled answers that account for less than 0.5% of the dataset. Then, the recognized answers are fed into a pre-trained automatic scoring system based on the BERT model without correcting misrecognized characters and providing rubric annotations. The automatic scoring system achieves from 0.84 to 0.98 of Quadratic Weighted Kappa (QWK). As QWK is over 0.8, it represents an acceptable similarity of scoring between the automatic scoring system and the human examiners. These results are promising for further research on end-to-end automatic scoring of descriptive answers.Comment: Keywords: handwritten Japanese answers, handwriting recognition, automatic scoring, ensemble recognition, deep neural networks; Reported in IEICE technical report, PRMU2021-32, pp.45-50 (2021.12) Published after peer review and Presented in ICFHR2022, Lecture Notes in Computer Science, vol. 13639, pp. 274-284 (2022.11

Automatic assessment of Java code

[EN] Assessment is an integral part of education often used to evaluate students, but also to provide them with feedback. It is essential to ensure that assessment is fair, objective, and equally applied to all students. This holds, for instance, in multiple-choice tests, but, unfortunately, it is not ensured in the assessment of source code, which is still a manual and error-prone task. In this paper, we present JavAssess, a Java library with an API composed of around 200 methods to automatically inspect, test, mark, and correct Java code. It can be used to produce both black-box (based on output comparison) and white-box (based on the internal properties of the code) assessment tools. This means that it allows for marking the code even if it is only partially correct. We describe the library, how to use it, and we provide a complete example to automatically mark and correct a student's code. We also report the use of this system in a real university context to compare manual and automatic assessment in university courses. The study reports the average error in the marks produced by teachers when assessing source code manually, and it shows that the system automatically assesses around 50% of the work. (C) 2018 Elsevier Ltd. All rights reserved.This work has been partially supported by MINECO/AEI/FEDER (EU) under grants TIN2013-44742-C4-1-R and TIN2016-76843-C4-1-R, by the Generalitat Valenciana under grant PROMETEO-II/2015/013 (SmartLogic), and by the Universitat Politecnica de Valencia under grant PIME B18 and EICE HEGEA.Insa Cabrera, D.; Silva, J. (2018). Automatic assessment of Java code. Computer Languages Systems & Structures. 53:59-72. https://doi.org/10.1016/j.cl.2018.01.004S59725

Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises

© ACM 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in http://dx.doi.org/10.1145/2729094.2742615Automated marking of multiple-choice exams is of great interest in university courses with a large number of students. For this reason, it has been systematically implanted in almost all universities. Automatic assessment of source code is however less extended. There are several reasons for that. One reason is that almost all existing systems are based on output comparison with a gold standard. If the output is the expected, the code is correct. Otherwise, it is reported as wrong, even if there is only one typo in the code. Moreover, why it is wrong remains a mystery. In general, assessment tools treat the code as a black box, and they only assess the externally observable behavior. In this work we introduce a new code assessment method that also verifies properties of the code, thus allowing to mark the code even if it is only partially correct. We also report about the use of this system in a real university context, showing that the system automatically assesses around 50% of the work.This work has been partially supported by the EU (FEDER) and the Spanish Ministerio de Economíay Competitividad (Secretaría de Estado de Investigación, Desarrollo e Innovación) under grant TIN2013-44742-C4-1-R and by the Generalitat Valenciana under grant PROMETEOII2015/013. David Insa was partially supported by the Spanish Ministerio de Educación under FPU grant AP2010-4415.Insa Cabrera, D.; Silva, J. (2015). Semi-automatic assessment of unrestrained Java code: a Library, a DSL, and a workbench to assess exams and exercises. ACM. https://doi.org/10.1145/2729094.2742615SK. A Rahman and M. Jan Nordin. A review on the static analysis approach in the automated programming assessment systems. In National Conference on Programming 07, 2007.K. Ala-Mutka. A survey of automated assessment approaches for programming assignments. In Computer Science Education, volume 15, pages 83--102, 2005.C. Beierle, M. Kula, and M. Widera. Automatic analysis of programming assignments. In Proc. der 1. E-Learning Fachtagung Informatik (DeLFI '03), volume P-37, pages 144--153, 2003.J. Biggs and C. Tang. Teaching for Quality Learning at University : What the Student Does (3rd Edition). In Open University Press, 2007.P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. CodeWrite: Supporting student-driven practice of java. In Proceedings of the 42nd ACM technical symposium on Computer science education, pages 09--12, 2011.R. Hendriks. Automatic exam correction. 2012.P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppala. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, pages 86--93, 2010.H. Kitaya and U. Inoue. An online automated scoring system for Java programming assignments. In International Journal of Information and Education Technology, volume 6, pages 275--279, 2014.M.-J. Laakso, T. Salakoski, A. Korhonen, and L. Malmi. Automatic assessment of exercises for algorithms and data structures - a case study with TRAKLA2. In Proceedings of Kolin Kolistelut/Koli Calling - Fourth Finnish/Baltic Sea Conference on Computer Science Education, pages 28--36, 2004.Y. Liang, Q. Liu, J. Xu, and D. Wang. The recent development of automated programming assessment. In Computational Intelligence and Software Engineering, pages 1--5, 2009.K. A. Naudé, J. H. Greyling, and D. Vogts. Marking student programs using graph similarity. In Computers & Education, volume 54, pages 545--561, 2010.A. Pears, S. Seidman, C. Eney, P. Kinnunen, and L. Malmi. Constructing a core literature for computing education research. In SIGCSE Bulletin, volume 37, pages 152--161, 2005.F. Prados, I. Boada, J. Soler, and J. Poch. Automatic generation and correction of technical exercices. In International Conference on Engineering and Computer Education (ICECE 2005), 2005.M. Supic, K. Brkic, T. Hrkac, Z. Mihajlovic, and Z. Kalafatic. Automatic recognition of handwritten corrections for multiple-choice exam answer sheets. In Information and Communication Technology, Electronics and Microelectronics (MIPRO), pages 1136--1141, 2014.S. Tung, T. Lin, and Y. Lin. An exercise management system for teaching programming. In Journal of Software, 2013.T. Wang, X. Su, Y. Wang, and P. Ma. Semantic similarity-based grading of student programs. In Information and Software Technology, volume 49, pages 99--107, 2007

An investigation of handwriting legibility and pencil use tasks in healthy older adults

This project explores handwriting legibility and pencil use tasks in 120 healthy older Australian adults, aged 60 to 99 years. A cross sectional study design was used. The aim of these studies was to explore if handwriting legibility or pencil use performance deteriorated as people aged. This is important to help therapists determine if handwriting difficulties following stroke, or other medical conditions, are more likely a consequence of condition-related impairments or due to ‘normal ageing’. Tasks performed under standardised test conditions included writing copied and self-composed sentences, shopping lists, transcribing a telephone message and completing the ‘lines’ and ‘dots’ pencil use Motor Assessment Scale (MAS) subtests. Handwriting legibility was scored using the Modified Four Point Scale-version 2. The first study explored the distribution of handwriting legibility scores in healthy older adults, relationships between handwriting legibility, age and writing task and reliability of rating procedures. Results indicated that handwriting generally remained legible in older adults, regardless of increasing age. The second study explored the performance of older adults without stroke on the ‘lines’ and ‘dots’ tasks, the relationship between age and task performance, and the relationship between writing speed and performance on the ‘lines’ task. Results indicated that many older adults failed the ‘lines’ task and many over 90 years of age failed the ‘dots’ task. Results suggest that impaired handwriting legibility in older adults who have had a stroke (or other medical condition) is likely due to the effects of the medical condition (or the complexity of the task) rather than ‘normal ageing’. However, failure to pass the ‘lines’ and ‘dots’ tasks is likely related to a combination of age and individual skill level and not solely due to condition-related impairment. A revised method for rating performance on the ‘lines’ and ‘dots’ tasks is also proposed

Diverse Contributions to Implicit Human-Computer Interaction

Cuando las personas interactúan con los ordenadores, hay mucha información que no se proporciona a propósito. Mediante el estudio de estas interacciones implícitas es posible entender qué características de la interfaz de usuario son beneficiosas (o no), derivando así en implicaciones para el diseño de futuros sistemas interactivos. La principal ventaja de aprovechar datos implícitos del usuario en aplicaciones informáticas es que cualquier interacción con el sistema puede contribuir a mejorar su utilidad. Además, dichos datos eliminan el coste de tener que interrumpir al usuario para que envíe información explícitamente sobre un tema que en principio no tiene por qué guardar relación con la intención de utilizar el sistema. Por el contrario, en ocasiones las interacciones implícitas no proporcionan datos claros y concretos. Por ello, hay que prestar especial atención a la manera de gestionar esta fuente de información. El propósito de esta investigación es doble: 1) aplicar una nueva visión tanto al diseño como al desarrollo de aplicaciones que puedan reaccionar consecuentemente a las interacciones implícitas del usuario, y 2) proporcionar una serie de metodologías para la evaluación de dichos sistemas interactivos. Cinco escenarios sirven para ilustrar la viabilidad y la adecuación del marco de trabajo de la tesis. Resultados empíricos con usuarios reales demuestran que aprovechar la interacción implícita es un medio tanto adecuado como conveniente para mejorar de múltiples maneras los sistemas interactivos.Leiva Torres, LA. (2012). Diverse Contributions to Implicit Human-Computer Interaction [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/17803Palanci

Archives, Access and Artificial Intelligence: Working with Born-Digital and Digitized Archival Collections

Digital archives are transforming the Humanities and the Sciences. Digitized collections of newspapers and books have pushed scholars to develop new, data-rich methods. Born-digital archives are now better preserved and managed thanks to the development of open-access and commercial software. Digital Humanities have moved from the fringe to the center of academia. Yet, the path from the appraisal of records to their analysis is far from smooth. This book explores crossovers between various disciplines to improve the discoverability, accessibility, and use of born-digital archives and other cultural assets

Archives, Access and Artificial Intelligence

Digital archives are transforming the Humanities and the Sciences. Digitized collections of newspapers and books have pushed scholars to develop new, data-rich methods. Born-digital archives are now better preserved and managed thanks to the development of open-access and commercial software. Digital Humanities have moved from the fringe to the center of academia. Yet, the path from the appraisal of records to their analysis is far from smooth. This book explores crossovers between various disciplines to improve the discoverability, accessibility, and use of born-digital archives and other cultural assets

Archives, Access and Artificial Intelligence

Digital archives are transforming the Humanities and the Sciences. Digitized collections of newspapers and books have pushed scholars to develop new, data-rich methods. Born-digital archives are now better preserved and managed thanks to the development of open-access and commercial software. Digital Humanities have moved from the fringe to the center of academia. Yet, the path from the appraisal of records to their analysis is far from smooth. This book explores crossovers between various disciplines to improve the discoverability, accessibility, and use of born-digital archives and other cultural assets

On the Evolution to PAPA

A narrative account of the origins and evolution of PAPA