sk_p: a neural program corrector for MOOCs

We present a novel technique for automatic program correction in MOOCs, capable of fixing both syntactic and semantic errors without manual, problem specific correction strategies. Given an incorrect student program, it generates candidate programs from a distribution of likely corrections, and checks each candidate for correctness against a test suite. The key observation is that in MOOCs many programs share similar code fragments, and the seq2seq neural network model, used in the natural-language processing task of machine translation, can be modified and trained to recover these fragments. Experiment shows our scheme can correct 29% of all incorrect submissions and out-performs state of the art approach which requires manual, problem specific correction strategies

Automatically Repairing Programs Using Both Tests and Bug Reports

The success of automated program repair (APR) depends significantly on its ability to localize the defects it is repairing. For fault localization (FL), APR tools typically use either spectrum-based (SBFL) techniques that use test executions or information-retrieval-based (IRFL) techniques that use bug reports. These two approaches often complement each other, patching different defects. No existing repair tool uses both SBFL and IRFL. We develop RAFL (Rank-Aggregation-Based Fault Localization), a novel FL approach that combines multiple FL techniques. We also develop Blues, a new IRFL technique that uses bug reports, and an unsupervised approach to localize defects. On a dataset of 818 real-world defects, SBIR (combined SBFL and Blues) consistently localizes more bugs and ranks buggy statements higher than the two underlying techniques. For example, SBIR correctly identifies a buggy statement as the most suspicious for 18.1% of the defects, while SBFL does so for 10.9% and Blues for 3.1%. We extend SimFix, a state-of-the-art APR tool, to use SBIR, SBFL, and Blues. SimFix using SBIR patches 112 out of the 818 defects; 110 when using SBFL, and 55 when using Blues. The 112 patched defects include 55 defects patched exclusively using SBFL, 7 patched exclusively using IRFL, 47 patched using both SBFL and IRFL and 3 new defects. SimFix using Blues significantly outperforms iFixR, the state-of-the-art IRFL-based APR tool. Overall, SimFix using our FL techniques patches ten defects no prior tools could patch. By evaluating on a benchmark of 818 defects, 442 previously unused in APR evaluations, we find that prior evaluations on the overused Defects4J benchmark have led to overly generous findings. Our paper is the first to (1) use combined FL for APR, (2) apply a more rigorous methodology for measuring patch correctness, and (3) evaluate on the new, substantially larger version of Defects4J.Comment: working pape

Verifix: Verified Repair of Programming Assignments

Automated feedback generation for introductory programming assignments is useful for programming education. Most works try to generate feedback to correct a student program by comparing its behavior with an instructor's reference program on selected tests. In this work, our aim is to generate verifiably correct program repairs as student feedback. The student assignment is aligned and composed with a reference solution in terms of control flow, and differences in data variables are automatically summarized via predicates to relate the variable names. Failed verification attempts for the equivalence of the two programs are exploited to obtain a collection of maxSMT queries, whose solutions point to repairs of the student assignment. We have conducted experiments on student assignments curated from a widely deployed intelligent tutoring system. Our results indicate that we can generate verified feedback in up to 58% of the assignments. More importantly, our system indicates when it is able to generate a verified feedback, which is then usable by novice students with high confidence

Autoavaliação na aprendizagem de programação introdutória: uma revisão sistemática da literatura / Self-assessment in introductory programming learning: a systematic literature review

O processo de aprendizagem de programação em cursos introdutórios é desafiador para alunos, professores e instituições de ensino, já que, devido a diversas dificuldades, os alunos tendem a evadir dos cursos por não se acharem capazes de aprender os conceitos de programação ou sentirem dificuldades durante seu aprendizado que não são supridas pelos professores. Diante desse contexto, este artigo busca, por meio de uma revisão sistemática da literatura, fazer um levantamento sobre o estado atual e as principais características da autoavaliação na aprendizagem de programação introdutória, para que os resultados levantados possam ser utilizados em contextos reais de sala de aula, com a finalidade de prover uma melhoria no processo de ensino e aprendizagem. O protocolo seguido, permitiu a busca de artigos em bases conhecidas na área de informática para educação, onde foram aplicados os critérios de inclusão e exclusão para posterior análise de qualidade e seleção final para leitura na íntegra desses estudos, para que assim, pudesse ser feita a extração dos dados que responderam às questões de pesquisa da obra. Os resultados apontam que nos últimos 5 anos: (i) a principal abordagem de autoavaliação utilizada foi o uso de feedback personalizado; (ii) a principal técnica utilizada foi o feedback; (iii) as principais limitações reportadas pelos autores foram limitações relacionadas à abordagem utilizada; (iv) dentre os benefícios, o que obteve mais relevância foi a melhoria na compreensão do conteúdo; (v) a abordagem de medição mais adotada foi a análise de performance; e (vi) as tecnologias que mais auxiliam no processo de autoavaliação, foram as ferramentas