DeepEval: An Integrated Framework for the Evaluation of Student Responses in Dialogue Based Intelligent Tutoring Systems

The automatic assessment of student answers is one of the critical components of an Intelligent Tutoring System (ITS) because accurate assessment of student input is needed in order to provide effective feedback that leads to learning. But this is a very challenging task because it requires natural language understanding capabilities. The process requires various components, concepts identification, co-reference resolution, ellipsis handling etc. As part of this thesis, we thoroughly analyzed a set of student responses obtained from an experiment with the intelligent tutoring system DeepTutor in which college students interacted with the tutor to solve conceptual physics problems, designed an automatic answer assessment framework (DeepEval), and evaluated the framework after implementing several important components. To evaluate our system, we annotated 618 responses from 41 students for correctness. Our system performs better as compared to the typical similarity calculation method. We also discuss various issues in automatic answer evaluation

Deeper Understanding of Tutorial Dialogues and Student Assessment

Bloom (1984) reported two standard deviation improvement with human tutoring which inspired many researchers to develop Intelligent Tutoring Systems (ITSs) that are as effective as human tutoring. However, recent studies suggest that the 2-sigma result was misleading and that current ITSs are as good as human tutors. Nevertheless, we can think of 2 standard deviations as the benchmark for tutoring effectiveness of ideal expert tutors. In the case of ITSs, there is still the possibility that ITSs could be better than humans.One way to improve the ITSs would be identifying, understanding, and then successfully implementing effective tutorial strategies that lead to learning gains. Another step towards improving the effectiveness of ITSs is an accurate assessment of student responses. However, evaluating student answers in tutorial dialogues is challenging. The student answers often refer to the entities in the previous dialogue turns and problem description. Therefore, the student answers should be evaluated by taking dialogue context into account. Moreover, the system should explain which parts of the student answer are correct and which are incorrect. Such explanation capability allows the ITSs to provide targeted feedback to help students reflect upon and correct their knowledge deficits. Furthermore, targeted feedback increases learners\u27 engagement, enabling them to persist in solving the instructional task at hand on their own. In this dissertation, we describe our approach to discover and understand effective tutorial strategies employed by effective human tutors while interacting with learners. We also present various approaches to automatically assess students\u27 contributions using general methods that we developed for semantic analysis of short texts. We explain our work using generic semantic similarity approaches to evaluate the semantic similarity between individual learner contributions and ideal answers provided by experts for target instructional tasks. We also describe our method to assess student performance based on tutorial dialogue context, accounting for linguistic phenomena such as ellipsis and pronouns. We then propose an approach to provide an explanatory capability for assessing student responses. Finally, we recommend a novel method based on concept maps for jointly evaluating and interpreting the correctness of student responses

Game-Based Learning, Gamification in Education and Serious Games

The aim of this book is to present and discuss new advances in serious games to show how they could enhance the effectiveness and outreach of education, advertising, social awareness, health, policies, etc. We present their use in structured learning activities, not only with a focus on game-based learning, but also on the use of game elements and game design techniques to gamify the learning process. The published contributions really demonstrate the wide scope of application of game-based approaches in terms of purpose, target groups, technologies and domains and one aspect they have in common is that they provide evidence of how effective serious games, game-based learning and gamification can be

An integrated environment for problem solving and program development

A framework for an integrated problem solving and program development environment that addresses the needs of students learning programming is proposed. Several objectives have been accomplished: defining the tasks required for program development and a literature review to determine the actual difficulties involved in learning those tasks. A comprehensive Study of environments and tools developed to support the learning of problem solving and programming was then performed, covering programming environments, debugging aids, intelligent tutoring systems, and intelligent programming environments. This was followed by a careful analysis and critique of these systems, which uncovered the limitations that have prevented them from accomplishing their goals. Next, an extensive study of problem solving methodologies developed in this century was carried out and a common model for problem solving was produced. The tasks of program development were then integrated with the common model for problem solving. Then, the cognitive activities required for problem solving and program development were identified and also integrated with the common model to form a Dual Common Model for problem Solving and Program Development. This dual common model was then used to define the functional specifications for a problem solving and program development environment which was designed, implemented, tested, and integrated into the curriculum. The development of the new environment for learning problem solving and programming was followed by the planning of a cognitively oriented assessment method and the development of related instruments to evaluate the process and the product of problem solving. A detailed statistical experiment to study the effect of this environment on students\u27 problem solving and program development skills, including system testing by protocol analysis, and performance evaluation of students based on research hypotheses and questions, was also designed, implemented and the result reported

Action-based Learning Assessment Method (ALAM) in Virtual Training Environments

Specialised and high priced simulators for surgical training, chemical labs, and flight training can provide real-world simulation in a safe and risk-free environment, but they are not accessible for the broader community due to costs for technology and availability of experts. Thus, training scenarios shifted to virtual worlds providing access for everyone interested in acquiring skills and knowledge at educational or professional institutions. Even in this context, we still expect a detailed formative feedback as would have been provided by a human trainer during the face to face process. Whilst the literature is focusing on goal-oriented assessment, it neglects the performed actions. In this paper, we present the Action-based Learning Assessment Method (ALAM) that analyses the action-sequences of the learners according to reference solutions by experts and automated formative feedback

Management: A bibliography for NASA managers

This bibliography lists 731 reports, articles and other documents introduced into the NASA Scientific and Technical Information System in 1990. Items are selected and grouped according to their usefulness to the manager as manager. Citations are grouped into ten subject categories: human factors and personnel issues; management theory and techniques; industrial management and manufacturing; robotics and expert systems; computers and information management; research and development; economics, costs and markets; logistics and operations management; reliability and quality control; and legality, legislation, and policy

Continual improvement: A bibliography with indexes, 1992-1993

This bibliography lists 606 references to reports and journal articles entered into the NASA Scientific and Technical Information Database during 1992 to 1993. Topics cover the philosophy and history of Continual Improvement (CI), basic approaches and strategies for implementation, and lessons learned from public and private sector models. Entries are arranged according to the following categories: Leadership for Quality, Information and Analysis, Strategic Planning for CI, Human Resources Utilization, Management of Process Quality, Supplier Quality, Assessing Results, Customer Focus and Satisfaction, TQM Tools and Philosophies, and Applications. Indexes include subject, personal author, corporate source, contract number, report number, and accession number

Redbird Scholar, Volume 8 Number 1

https://ir.library.illinoisstate.edu/rsp_redscholar/1015/thumbnail.jp