A Framework for Evaluating Statistical Models in Physics Education Research

Across the field of education research there has been an increased focus on the development, critique, and evaluation of statistical methods and data usage due to recently created, very large data sets and machine learning techniques. In physics education research (PER), this increased focus has recently been shown through the 2019 Physical Review PER Focused Collection examining quantitative methods in PER. Quantitative PER has provided strong arguments for reforming courses by including interactive engagement, demonstrated that students often move away from scientist-like views due to science education, and has injected robust assessment into the physics classroom via concept inventories. The work presented here examines the impact that machine learning may have on physics education research, presents a framework for the entire process including data management, model evaluation, and results communication, and demonstrates the utility of this framework through the analysis of two types of survey data

Estimating the quality of arguments in argument-based machine learning

Argument-based machine learning (ABML) knowledge refinement loop enables an interaction between a machine learning algorithm and a domain expert. It represents a powerful knowledge elicitation tool, suitable for obtaining expert knowledge in difficult domains. The loop enables the expert to focus on the most critical parts of the current knowledge base, and helps him or her to argue about automatically chosen relevant examples. The expert only needs to explain a single example at the time, which facilitates articulating arguments. It also helps the expert to improve the explanations by providing (automatically chosen) relevant counter examples. It has been shown recently that ABML knowledge refinement loop also enables design of argumentation-based interactive teaching tool. However, so far the machine was not able to provide neither the teachers (that designed such a tool) nor the students (that used it for learning) with concrete estimations about the quality of their arguments. In this thesis, we have designed three approaches for giving immediate feedback about the quality of arguments used in the ABML knowledge refinement loop. The chosen experimental domain was financial statement analysis, more concretely estimating credit scores of companies (enterprises). Our goal was twofold: to obtain a successful classification model for predicting the credit scores, and to enable the students to learn about this rather difficult domain. In the experimental sessions, both the teacher and the students were involved in the process of knowledge elicitation with the ABML knowledge refinement loop, receiving the feedback about their arguments. The goal of the learning session with the teacher was in particular to obtain advanced concepts (attributes) that describe the domain well, are suitable for teaching, and also enable successful predictions. This was done with the help of a financial expert. In the “tutoring" sessions, the students learned about the intricacies of the domain and strived for the best predictive model as possible, also by using the teacher's advanced concepts in their arguments. The main contributions of this work are: - the design of three approaches for estimating the quality of arguments used in the argument-based machine learning (ABML) knowledge refinement loop, - implementation of argumentation-based interactive teaching tool for estimating credit scores of companies (enterprises), using real data, - a detailed description of the learning session, where the student received three types of feedback about the arguments used

Dataset of Shell Commands Used by Participants of Hands-on Cybersecurity Training

We present a dataset of 13446 shell commands from 175 participants who attended cybersecurity training and solved assignments in the Linux terminal. Each acquired data record contains a command with its arguments and metadata, such as a timestamp, working directory, and host identification in the emulated training infrastructure. The commands were captured in Bash, ZSH, and Metasploit shells. The data are stored as JSON records, enabling vast possibilities for their further use in research and development. These include educational data mining, learning analytics, student modeling, and evaluating machine learning models for intrusion detection. The data were collected from 27 cybersecurity training sessions using an open-source logging toolset and two open-source interactive learning environments. Researchers and developers may use the dataset or deploy the learning environments with the logging toolset to generate their own data in the same format. Moreover, we provide a set of common analytical queries to facilitate the exploratory analysis of the dataset

Proof-Pattern Recognition and Lemma Discovery in ACL2

We present a novel technique for combining statistical machine learning for proof-pattern recognition with symbolic methods for lemma discovery. The resulting tool, ACL2(ml), gathers proof statistics and uses statistical pattern-recognition to pre-processes data from libraries, and then suggests auxiliary lemmas in new proofs by analogy with already seen examples. This paper presents the implementation of ACL2(ml) alongside theoretical descriptions of the proof-pattern recognition and lemma discovery methods involved in it

Learning to Prove Theorems via Interacting with Proof Assistants

Humans prove theorems by relying on substantial high-level reasoning and problem-specific insights. Proof assistants offer a formalism that resembles human mathematical reasoning, representing theorems in higher-order logic and proofs as high-level tactics. However, human experts have to construct proofs manually by entering tactics into the proof assistant. In this paper, we study the problem of using machine learning to automate the interaction with proof assistants. We construct CoqGym, a large-scale dataset and learning environment containing 71K human-written proofs from 123 projects developed with the Coq proof assistant. We develop ASTactic, a deep learning-based model that generates tactics as programs in the form of abstract syntax trees (ASTs). Experiments show that ASTactic trained on CoqGym can generate effective tactics and can be used to prove new theorems not previously provable by automated methods. Code is available at https://github.com/princeton-vl/CoqGym.Comment: Accepted to ICML 201

Premise Selection and External Provers for HOL4

Learning-assisted automated reasoning has recently gained popularity among the users of Isabelle/HOL, HOL Light, and Mizar. In this paper, we present an add-on to the HOL4 proof assistant and an adaptation of the HOLyHammer system that provides machine learning-based premise selection and automated reasoning also for HOL4. We efficiently record the HOL4 dependencies and extract features from the theorem statements, which form a basis for premise selection. HOLyHammer transforms the HOL4 statements in the various TPTP-ATP proof formats, which are then processed by the ATPs. We discuss the different evaluation settings: ATPs, accessible lemmas, and premise numbers. We measure the performance of HOLyHammer on the HOL4 standard library. The results are combined accordingly and compared with the HOL Light experiments, showing a comparably high quality of predictions. The system directly benefits HOL4 users by automatically finding proofs dependencies that can be reconstructed by Metis