Waterproof: educational software for learning how to write mathematical proofs

In order to help students learn how to write mathematical proofs, we developed the educational software called Waterproof (https://github.com/impermeable/waterproof). Waterproof is based on the Coq proof assistant. As students type out their proofs in the program, it checks the logical soundness of each proof step and provides additional guiding feedback. Contrary to Coq proofs, proofs written in Waterproof are similar in style to handwritten ones: proof steps are denoted using controlled natural language, the structure of proofs is made explicit by enforced signposting, and chains of inequalities can be used to prove larger estimates. To achieve this, we developed the Coq library coq-waterproof. The library extends Coq's default tactics using the Ltac2 tactic language. We include many code snippets in this article to increase the number of available Ltac2 examples. Waterproof has been used to supplement teaching the course Analysis 1 at the TU/e for a couple of years. Students started using Waterproof's controlled formulations of proof steps in their handwritten proofs as well; the explicit phrasing of these sentences helps to clarify the logical structure of their arguments.Comment: The Waterproof software can be found at https://github.com/impermeable/waterproof . This article pertains to Waterproof version 0.6.1. The Coq library coq-waterproof can be found at https://github.com/impermeable/coq-waterproof . This article pertains to coq-waterproof version 1.2.

Analysis of classifiers' robustness to adversarial perturbations

The goal of this paper is to analyze an intriguing phenomenon recently discovered in deep networks, namely their instability to adversarial perturbations (Szegedy et. al., 2014). We provide a theoretical framework for analyzing the robustness of classifiers to adversarial perturbations, and show fundamental upper bounds on the robustness of classifiers. Specifically, we establish a general upper bound on the robustness of classifiers to adversarial perturbations, and then illustrate the obtained upper bound on the families of linear and quadratic classifiers. In both cases, our upper bound depends on a distinguishability measure that captures the notion of difficulty of the classification task. Our results for both classes imply that in tasks involving small distinguishability, no classifier in the considered set will be robust to adversarial perturbations, even if a good accuracy is achieved. Our theoretical framework moreover suggests that the phenomenon of adversarial instability is due to the low flexibility of classifiers, compared to the difficulty of the classification task (captured by the distinguishability). Moreover, we show the existence of a clear distinction between the robustness of a classifier to random noise and its robustness to adversarial perturbations. Specifically, the former is shown to be larger than the latter by a factor that is proportional to \sqrt{d} (with d being the signal dimension) for linear classifiers. This result gives a theoretical explanation for the discrepancy between the two robustness properties in high dimensional problems, which was empirically observed in the context of neural networks. To the best of our knowledge, our results provide the first theoretical work that addresses the phenomenon of adversarial instability recently observed for deep networks. Our analysis is complemented by experimental results on controlled and real-world data

Tools for Tutoring Theoretical Computer Science Topics

This thesis introduces COMPLEXITY TUTOR, a tutoring system to assist in learning abstract proof-based topics, which has been specifically targeted towards the population of computer science students studying theoretical computer science. Existing literature has shown tremendous educational benefits produced by active learning techniques, student-centered pedagogy, gamification and intelligent tutoring systems. However, previously, there had been almost no research on adapting these ideas to the domain of theoretical computer science. As a population, computer science students receive immediate feedback from compilers and debuggers, but receive no similar level of guidance for theoretical coursework. One hypothesis of this thesis is that immediate feedback while working on theoretical problems would be particularly well-received by students, and this hypothesis has been supported by the feedback of students who used the system. This thesis makes several contributions to the field. It provides assistance for teaching proof construction in theoretical computer science. A second contribution is a framework that can be readily adapted to many other domains with abstract mathematical content. Exercises can be constructed in natural language and instructors with limited programming knowledge can quickly develop new subject material for COMPLEXITY TUTOR. A third contribution is a platform for writing algorithms in Python code that has been integrated into this framework, for constructive proofs in computer science. A fourth contribution is development of an interactive environment that uses a novel graphical puzzle-like platform and gamification ideas to teach proof concepts. The learning curve for students is reduced, in comparison to other systems that use a formal language or complex interface. A multi-semester evaluation of 101 computer science students using COMPLEXITY TUTOR was conducted. An additional 98 students participated in the study as part of control groups. COMPLEXITY TUTOR was used to help students learn the topics of NP-completeness in algorithms classes and prepositional logic proofs in discrete math classes. Since this is the first significant study of using a computerized tutoring system in theoretical computer science, results from the study not only provide evidence to support the suitability of using tutoring systems in theoretical computer science, but also provide insights for future research directions

Emergences and affordances as opportunities to develop teachers’ mathematical content knowledge

Teachers’ mathematical content knowledge has been under scrutiny for some time. This development is in the wake of learners’ unsatisfactory performance in national examinations and international achievement tests. A widely held belief is that one, if not the most important, of the efforts to improve and enhance the performance and achievement in mathematics of learners is addressing teachers’ mathematical content and pedagogical content knowledge through continuous professional development initiatives. The focus of this article is on the former. It describes how emergent and affording opportunities are brought to the fore from classroom observations and interactions in workshops and institutes with practising teachers. It concludes that this in situ dealing with mathematical content knowledge holds much promise for buy-in by teachers because it addresses an immediate need related to their practice

Mathematical Language Models: A Survey

In recent years, there has been remarkable progress in leveraging Language Models (LMs), encompassing Pre-trained Language Models (PLMs) and Large-scale Language Models (LLMs), within the domain of mathematics. This paper conducts a comprehensive survey of mathematical LMs, systematically categorizing pivotal research endeavors from two distinct perspectives: tasks and methodologies. The landscape reveals a large number of proposed mathematical LLMs, which are further delineated into instruction learning, tool-based methods, fundamental CoT techniques, and advanced CoT methodologies. In addition, our survey entails the compilation of over 60 mathematical datasets, including training datasets, benchmark datasets, and augmented datasets. Addressing the primary challenges and delineating future trajectories within the field of mathematical LMs, this survey is positioned as a valuable resource, poised to facilitate and inspire future innovation among researchers invested in advancing this domain.Comment: arXiv admin note: text overlap with arXiv:1705.04146, arXiv:2304.10977, arXiv:2112.00114, arXiv:1905.13319, arXiv:2304.12244, arXiv:2206.01347, arXiv:2006.09265 by other author