30 research outputs found
Deduplicating and Ranking Solution Programs for Suggesting Reference Solutions
Referring to solution programs written by other users is helpful for learners
in programming education. However, current online judge systems just list all
solution programs submitted by users for references, and the programs are
sorted based on the submission date and time, execution time, or user rating,
ignoring to what extent the programs can be helpful to be referenced. In
addition, users struggle to refer to a variety of solution approaches since
there are too many duplicated and near-duplicated programs. To motivate
learners to refer to various solutions to learn better solution approaches, in
this paper, we propose an approach to deduplicate and rank common solution
programs in each programming problem. Inspired by the nature that the
many-duplicated program adopts a more common approach and can be a general
reference, we remove the near-duplicated solution programs and rank the unique
programs based on the duplicate count. The experiments on the solution programs
submitted to a real-world online judge system demonstrate that the number of
programs is reduced by 60.20%, whereas the baseline only reduces by 29.59%
after the deduplication, meaning that users only need to refer to 39.80% of
programs on average. Furthermore, our analysis shows that top-10 ranked
programs cover 29.95% of programs on average, indicating that users can grasp
29.95% of solution approaches by referring to only 10 programs. The proposed
approach shows the potential of reducing the learners' burden of referring to
too many solutions and motivating them to learn a variety of solution
approaches.Comment: 7 pages, 5 figures, accepted to ASSE 202
Composing control flow and formula rules for computing on grids
We define computation on grids as the composition, through pushout constructions, of control flows, carried across adjacency relations between grid cells, with formulas updating the value of some attribute. The approach is based on the identification of a subcategory of attributed typed graphs suitable to the definition of pushouts on grids, and is illustrated in the context of the Cyberfilm visual language
Exploring Automated Code Evaluation Systems and Resources for Code Analysis: A Comprehensive Survey
The automated code evaluation system (AES) is mainly designed to reliably
assess user-submitted code. Due to their extensive range of applications and
the accumulation of valuable resources, AESs are becoming increasingly popular.
Research on the application of AES and their real-world resource exploration
for diverse coding tasks is still lacking. In this study, we conducted a
comprehensive survey on AESs and their resources. This survey explores the
application areas of AESs, available resources, and resource utilization for
coding tasks. AESs are categorized into programming contests, programming
learning and education, recruitment, online compilers, and additional modules,
depending on their application. We explore the available datasets and other
resources of these systems for research, analysis, and coding tasks. Moreover,
we provide an overview of machine learning-driven coding tasks, such as bug
detection, code review, comprehension, refactoring, search, representation, and
repair. These tasks are performed using real-life datasets. In addition, we
briefly discuss the Aizu Online Judge platform as a real example of an AES from
the perspectives of system design (hardware and software), operation
(competition and education), and research. This is due to the scalability of
the AOJ platform (programming education, competitions, and practice), open
internal features (hardware and software), attention from the research
community, open source data (e.g., solution codes and submission documents),
and transparency. We also analyze the overall performance of this system and
the perceived challenges over the years
Program Repair with Minimal Edits Using CodeT5
Programmers often struggle to identify and fix bugs in their programs. In
recent years, many language models (LMs) have been proposed to fix erroneous
programs and support error recovery. However, the LMs tend to generate
solutions that differ from the original input programs. This leads to potential
comprehension difficulties for users. In this paper, we propose an approach to
suggest a correct program with minimal repair edits using CodeT5. We fine-tune
a pre-trained CodeT5 on code pairs of wrong and correct programs and evaluate
its performance with several baseline models. The experimental results show
that the fine-tuned CodeT5 achieves a pass@100 of 91.95% and an average edit
distance of the most similar correct program of 6.84, which indicates that at
least one correct program can be suggested by generating 100 candidate
programs. We demonstrate the effectiveness of LMs in suggesting program repair
with minimal edits for solving introductory programming problems.Comment: 7 pages, 6 figures, accepted to iCAST 202
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
In recent years, the rise of advanced artificial intelligence technologies has had a profound impact on many fields, including education and research. One such technology is ChatGPT, a powerful large language model developed by OpenAI. This technology offers exciting opportunities for students and educators, including personalized feedback, increased accessibility, interactive conversations, lesson preparation, evaluation, and new ways to teach complex concepts. However, ChatGPT poses different threats to the traditional education and research system, including the possibility of cheating on online exams, human-like text generation, diminished critical thinking skills, and difficulties in evaluating information generated by ChatGPT. This study explores the potential opportunities and threats that ChatGPT poses to overall education from the perspective of students and educators. Furthermore, for programming learning, we explore how ChatGPT helps students improve their programming skills. To demonstrate this, we conducted different coding-related experiments with ChatGPT, including code generation from problem descriptions, pseudocode generation of algorithms from texts, and code correction. The generated codes are validated with an online judge system to evaluate their accuracy. In addition, we conducted several surveys with students and teachers to find out how ChatGPT supports programming learning and teaching. Finally, we present the survey results and analysis
A Model with Iterative Trials for Correcting Logic Errors in Source Code
It is difficult for students and teachers to detect and correct logic errors in source code. Compilers and integrated development environments (IDEs) have the ability to detect and correct syntax errors but it is also difficult for them to detect and correct logic errors. Although many machine learning approaches have been proposed that can show correction candidates for logic errors, they do not provide guidance concerning how the user should fix them. In this paper, we propose a model for correcting logic errors in a given source code. The proposed model realizes debugging of multiple logic errors in the source code by iterative trials of identifying the errors, correcting the errors, and testing the source code. In this model, in the first stage, a list of correction candidates is provided by a deep learning model, and then the list is given to an editing operation predictor that predicts the editing operation for the correction candidate. To learn the internal parameters of the proposed model, we use a set of solution codes created to solve the corresponding programming tasks in a real e-learning system. To verify the usefulness of the proposed model, we apply it to 32 programming tasks. Experimental results show that the correction accuracy is, on average, 58.64% higher than that of the conventional model without iterative trials
Efficient visualisation of the relative distribution of keyword search results in a corpus data cube
Most keyword searches target precision for finding the most relevant document. However some target recall, finding all relevant documents. Our system supports high recall searches that return hundreds or thousands of relevant results. In particular, it provides a visualization that shows the distribution of search results relative to the distribution of items for the entire corpus. Such relative distributional features include over and under representation, clusters and outliers. The contribution of this paper is efficient visualisation, that is, how to provide the best relative distribution view for a given data cube size. This requirement is translated to: for which limited size meta-data summary cube are search results disambiguated the most in our relative distribution view. We identify metrics and several algorithms for such a summary cube selection
Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education
The rate of software development has increased dramatically. Conventional compilers cannot assess and detect all source code errors. Software may thus contain errors, negatively affecting end-users. It is also difficult to assess and detect source code logic errors using traditional compilers, resulting in software that contains errors. A method that utilizes artificial intelligence for assessing and detecting errors and classifying source code as correct (error-free) or incorrect is thus required. Here, we propose a sequential language model that uses an attention-mechanism-based long short-term memory (LSTM) neural network to assess and classify source code based on the estimated error probability. The attentive mechanism enhances the accuracy of the proposed language model for error assessment and classification. We trained the proposed model using correct source code and then evaluated its performance. The experimental results show that the proposed model has logic and syntax error detection accuracies of 92.2% and 94.8%, respectively, outperforming state-of-the-art models. We also applied the proposed model to the classification of source code with logic and syntax errors. The average precision, recall, and F-measure values for such classification are much better than those of benchmark models. To strengthen the proposed model, we combined the attention mechanism with LSTM to enhance the results of error assessment and detection as well as source code classification. Finally, our proposed model can be effective in programming education and software engineering by improving code writing, debugging, error-correction, and reasoning
A Neural Network Based Intelligent Support Model for Program Code Completion
In recent years, millions of source codes are generated in different languages on a daily basis all over the world. A deep neural network-based intelligent support model for source code completion would be a great advantage in software engineering and programming education fields. Vast numbers of syntax, logical, and other critical errors that cannot be detected by normal compilers continue to exist in source codes, and the development of an intelligent evaluation methodology that does not rely on manual compilation has become essential. Even experienced programmers often find it necessary to analyze an entire program in order to find a single error and are thus being forced to waste valuable time debugging their source codes. With this point in mind, we proposed an intelligent model that is based on long short-term memory (LSTM) and combined it with an attention mechanism for source code completion. Thus, the proposed model can detect source code errors with locations and then predict the correct words. In addition, the proposed model can classify the source codes as to whether they are erroneous or not. We trained our proposed model using the source code and then evaluated the performance. All of the data used in our experiments were extracted from Aizu Online Judge (AOJ) system. The experimental results obtained show that the accuracy in terms of error detection and prediction of our proposed model approximately is 62% and source code classification accuracy is approximately 96% which outperformed a standard LSTM and other state-of-the-art models. Moreover, in comparison to state-of-the-art models, our proposed model achieved an interesting level of success in terms of error detection, prediction, and classification when applied to long source code sequences. Overall, these experimental results indicate the usefulness of our proposed model in software engineering and programming education arena