101,134 research outputs found
Mutation testing on an object-oriented framework: An experience report
This is the preprint version of the article - Copyright @ 2011 ElsevierContext
The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners.
Objective
This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective.
Method
In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT.
Results
Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field.
Conclusion
We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT Project SETI (TIN2009-07366) and the Andalusian Government Projects ISABEL (TIC-2533) and THEOS (TIC-5906)
Software Verification and Graph Similarity for Automated Evaluation of Students' Assignments
In this paper we promote introducing software verification and control flow
graph similarity measurement in automated evaluation of students' programs. We
present a new grading framework that merges results obtained by combination of
these two approaches with results obtained by automated testing, leading to
improved quality and precision of automated grading. These two approaches are
also useful in providing a comprehensible feedback that can help students to
improve the quality of their programs We also present our corresponding tools
that are publicly available and open source. The tools are based on LLVM
low-level intermediate code representation, so they could be applied to a
number of programming languages. Experimental evaluation of the proposed
grading framework is performed on a corpus of university students' programs
written in programming language C. Results of the experiments show that
automatically generated grades are highly correlated with manually determined
grades suggesting that the presented tools can find real-world applications in
studying and grading
Intelligent Tutoring System: Experience of Linking Software Engineering and Programming Teaching
The increasing number of computer science students pushes lecturers and
tutors of first-year programming courses to their limits to provide
high-quality feedback to the students. Existing systems that handle automated
grading primarily focus on the automation of test case executions in the
context of programming assignments. However, they cannot provide customized
feedback about the students' errors, and hence, cannot replace the help of
tutors. While recent research works in the area of automated grading and
feedback generation address this issue by using automated repair techniques, so
far, to the best of our knowledge, there has been no real-world deployment of
such techniques. Based on the research advances in recent years, we have built
an intelligent tutoring system that has the capability of providing automated
feedback and grading. Furthermore, we designed a Software Engineering course
that guides third-year undergraduate students in incrementally developing such
a system over the coming years. Each year, students will make contributions
that improve the current implementation, while at the same time, we can deploy
the current system for usage by first year students. This paper describes our
teaching concept, the intelligent tutoring system architecture, and our
experience with the stakeholders. This software engineering project for the
students has the key advantage that the users of the system are available
in-house (i.e., students, tutors, and lecturers from the first-year programming
courses). This helps organize requirements engineering sessions and builds
awareness about their contribution to a "to be deployed" software project. In
this multi-year teaching effort, we have incrementally built a tutoring system
that can be used in first-year programming courses. Further, it represents a
platform that can integrate the latest research results in APR for education
Evaluation of a tool for Java structural specification checking
Although a number of tools for evaluating Java code functionality and style exist, little work has been done in a distance learning context on automated marking of Java programs with respect to structural specifications. Such automated checks support human markers in assessing students’ work and evaluating their own marking; online automated marking; students checking code before submitting it for marking; and question setters evaluating the completeness of questions set. This project developed and evaluated a prototype tool that performs an automated check of a Java program’s correctness with respect to a structural specification. Questionnaires and interviews were used to gather feedback on the usefulness of the tool as a marking aid to humans, and on its potential usefulness to students for self-assessment when working on their assignments. Markers were asked to compare the usefulness of structural specification testing as compared to other kinds of support, including syntax error assistance, style checking and functionality testing. Initial results suggest that most markers using the structural specification checking tool found it to be useful, and some reported that it increased their accuracy in marking. Reasons for not using the tool included lack of time and the simplicity of the assignment it was trialled on. Some reservations were expressed about reliance on tools for assessment, both for markers and for students. The need for advice on incorporating tools in marking workflow is suggested
Recommended from our members
Keywords of written reflection - a comparison between reflective and descriptive datasets
This study investigates reflection keywords by contrasting two datasets, one of reflective sentences and another of descriptive sentences. The log-likelihood statistic reveals several reflection keywords that are discussed in the context of a model for reflective writing. These keywords are seen as a useful building block for tools that can automatically analyse reflection in texts
Automata Tutor v3
Computer science class enrollments have rapidly risen in the past decade.
With current class sizes, standard approaches to grading and providing
personalized feedback are no longer possible and new techniques become both
feasible and necessary. In this paper, we present the third version of Automata
Tutor, a tool for helping teachers and students in large courses on automata
and formal languages. The second version of Automata Tutor supported automatic
grading and feedback for finite-automata constructions and has already been
used by thousands of users in dozens of countries. This new version of Automata
Tutor supports automated grading and feedback generation for a greatly extended
variety of new problems, including problems that ask students to create regular
expressions, context-free grammars, pushdown automata and Turing machines
corresponding to a given description, and problems about converting between
equivalent models - e.g., from regular expressions to nondeterministic finite
automata. Moreover, for several problems, this new version also enables
teachers and students to automatically generate new problem instances. We also
present the results of a survey run on a class of 950 students, which shows
very positive results about the usability and usefulness of the tool
- …