Identifying evidences of computer programming skills through automatic source code evaluation

Orientador: Roberto PereiraCoorientador: Eleandro MaschioTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa : Curitiba, 27/03/2020Inclui referências: p. 98-106Área de concentração: Ciência da ComputaçãoResumo: Esta tese e contextualizada no ensino de programacao de computadores em cursos de Computacao e investiga aspectos e estrategias para avaliacao automatica e continua de codigos fonte desenvolvidos pelos alunos. O estado da arte foi identificado por meio de revisao sistematica de literatura e revelou que as pesquisas anteriores tendem a realizar avaliacoes baseadas em aspectos tecnicos de codigos fonte, como a avaliacao de corretude funcional e a deteccao de erros. Avaliacoes baseadas em habilidades, por outro lado, sao pouco exploradas e possuem potencial para fornecer detalhes a respeito de habilidades representadas por conceitos de alto nivel, como desvios condicionais e estruturas de repeticao. Um metodo de identificacao automatica de evidencias de aprendizado e entao proposto como uma abordagem baseada em habilidades para a avaliacao automatica de codigos fonte de programacao. O metodo e caracterizado pela implementacao de diferentes estrategias para avaliacao de codigos fonte, identificacao de evidencias de habilidades de programacao, e representacao destas habilidades em um modelo do aluno. Experimentos realizados em ambientes controlados (bases de dados artificiais) mostraram que estrategias automaticas de avaliacao de codigo fonte sao viaveis. Experimentos conduzidos em ambientes reais (codigos fonte produzidos por alunos) produziram resultados semelhantes aos ambientes controlados, entretanto revelaram limitacoes relacionadas a implementacao das estrategias, como vulnerabilidades a sintaxes inesperadas e falhas em expressoes regulares. Um conjunto de habilidades foi selecionado para compor o modelo do aluno, representado por uma rede bayesiana dinamica. Por meio de experimentos foi demonstrado que a alimentacao do modelo com evidencias resultantes da avaliacao automatica de codigos fonte permite o acompanhamento do progresso das habilidades dos alunos. Finalmente, as estrategias automaticas em conjunto com os recursos do modelo do aluno permitiram a demonstracao da avaliacao baseada em habilidades, que se mostrou um recurso valioso para identificacao de solucoes funcionalmente corretas, porem conceitualmente incorretas; quando o programa e funcionalmente correto, retornando resultados esperados a determinadas entradas, porem foi construido com recursos e conceitos incorretos. Palavras-chave: Programacao de Computadores, Avaliacao Automatica, Avaliacao Baseada em HabilidadesAbstract: This thesis is contextualized in the teaching of computer programming in Computing courses and investigates aspects and strategies for automatic and continuous evaluation of student developed source codes. The state of the art was identified through systematic literature review and revealed previous research tends to perform evaluations based on source codes technical aspects, such as functional correctness assessment and error detection. Skills-based assessments, in turn, are less explored although having potential to provide details of skills represented by high-level concepts, such as conditionals and repetition structures. A method for automatic identification of learning evidences is then proposed as a skills-based approach to automatic evaluation of programming source codes. The method is characterized by implementing different strategies for source code evaluation, identifying evidences of programming skills, and representing these skills in a student model. Experiments conducted in controlled scenarios (testing datasets) have shown automatic source code evaluation strategies are viable. Experiments conducted in real scenarios (student-made source codes) produced results similar to controlled scenarios, however, implementation-related limitations were revealed for some strategies, such as vulnerabilities to unexpected syntax and flaws in regular expressions. A skill set was selected to compose our student model, represented by a Dynamic Bayesian Network. Experiments have shown feeding the model with evidences resulting from source codes automatic evaluation allows monitoring students' skills progress. Finally, automatic strategies coupled with student model capabilities enabled demonstrating skills-based assessment, which showed a valuable resource for identifying functionally correct source codes, but conceptually incorrect; when a program is correct functionally, returning expected results to specific inputs, but it was built with erroneous concepts and resources. Keywords: Computer Programming, Automatic Evaluation, Skills-Based Assessmen

Improving learning and teaching through automated short-answer marking

Automated short-answer marking cannot 'guarantee' 100% agreement between the marks generated by a software system and the marks produced separately by a human. This problem has prevented automated marking systems from being used in high-stake short-answer marking. Given this limitation, can an automated short-answer marking system have any practical application? This thesis describes how an automated short-answer marking system, called IndusMarker, can be effectively used to improve learning and teaching.The design and evaluation of IndusMarker are also presented in the thesis. IndusMarker is designed for factual answers where there is a clear criterion for answers being right or wrong. The system is based on structure matching, i.e. matching a pre-specified structure, developed via a purpose-built structure editor, with the content of the student's answer text. An examiner specifies the required structure of an answer in a simple purpose-designed language called Question Answer Markup Language (QAML). The structure editor ensures that users construct correct required structures (with respect to QAML's syntax and informal semantics) in a form that is suitable for accurate automated marking.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Experiences with Online Programming Examinations

An online programming examination was used to assess undergraduates who had learnt Java for two terms. The advantages are that the students are assessed on what they have to do in practice, rather than on their theoretical knowledge, and that the marking load on the examiner is drastically reduced. The style of question used and the results of the examination are analysed.