IRT-Based Adaptive Hints to Scaffold Learning in Programming

Over the past few decades, many studies conducted in the field of learning science have described that scaffolding plays an important role in human learning. To scaffold a learner efficiently, a teacher should predict how much support a learner must have to complete tasks and then decide the optimal degree of assistance to support the learner\u27s development. Nevertheless, it is difficult to ascertain the optimal degree of assistance for learner development. For this study, it is assumed that optimal scaffolding is based on a probabilistic decision rule: Given a teacher\u27s assistance to facilitate the learner development, an optimal probability exists for a learner to solve a task. To ascertain that optimal probability, we developed a scaffolding system that provides adaptive hints to adjust the predictive probability of the learner\u27s successful performance to the previously determined certain value, using a probabilistic model, i.e., item response theory (IRT). Furthermore, using the scaffolding system, we compared learning performances by changing the predictive probability. Results show that scaffolding to achieve 0.5 learner success probability provides the best performance. Additionally, results demonstrate that a scaffolding system providing 0.5 probability decreases the number of hints (amount of support) automatically as a fading function according to the learner\u27s growth capability

Technology Enabled Assessments: An Investigation of Scoring Models for Scaffolded Tasks

While significant progress has been made in recent years on technology enabled assessments (TEAs), including assessment systems that incorporate scaffolding into the assessment process, there is a dearth of research regarding psychometric scoring models that can be used to fully capture students' knowledge, skills and abilities as measured by TEAs. This investigation provides a comparison of seven scoring models applied to an operational assessment system that incorporates scaffolding into the assessment process and evaluates student ability estimates derived from those models from a validity perspective. A sequential procedure for fitting and evaluating increasingly complex models was conducted. Specifically, a baseline model that did not account for any scaffolding features in the assessment system was established and compared to three additional models that each accounted for scaffolding features using a dichotomous, a polytomous and a testlet model approach. Models were compared and evaluated against several criteria including model convergence, the amount of information each model provided and the statistical relationships between scaled scores and a criterion measure of student ability. Based on these criteria, the dichotomous model that accounted for all of the scaffold items but ignored local dependence was determined to be the optimal scoring model for the assessment system used in this study. However, if the violation against the local independence assumption is deemed unacceptable, it was also concluded that the polytomous model for scoring these assessments is a worthwhile and viable alternative. In any case, the scoring models that accounted for the scaffolding features in the assessment system were determined to be better overall models than the baseline model that did not account for these features. It was also determined that the testlet model approach was not a practical or useful scoring option for this assessment system. Given the purpose of the assessment system used in this study, which is a formative tool that also provides instructional opportunities to students during the assessment process, the advantages of applying any of these scoring models from a measurement perspective may not justify the practical disadvantages. For instance, a basic percent correct score may be completely dependent on the specific items that a student took but it is relatively simple to understand and compute. On the other hand, scaled scores from these scoring models are independent of the items from which they were calibrated from, but ability estimates are more complex to understand and derive. As the assessment system used in this study is a low stakes environment that is mostly geared towards learning, the benefits of the scoring models presented in this study need to be weighed against the practical constraints within an operational context with respect to time, cost and resources

アダプティブラーニングのためのヒントを組み込んだDeep-IRT

近年、教育の現場では、機械学習を用いて学習履歴データから学習者の能力成長を把握し、個々の学習者に適切なヒントを提供するアダプティブラーニングが注目されている。学習者に最適なヒントを提供するためには、学習者が誤答した際に、各ヒントを提供した場合の正答確率を正確に推定する必要がある。最新の研究では、深層学習モデルと項目反応理論を組み合わせたDeep-IRT手法が開発されており、学習履歴データから課題の難易度と学習者の多次元のスキルに対する能力変化を推定できるようになってきた。しかし、既存のDeep-IRT手法では、課題に依存したヒントの提供しか想定しておらず、能力変化を考慮した最適ヒントは提案されていない。本論文ではDeep-IRTモデルをアダプティブラーニングに適用できるようにするために、学習者が項目に正答するまでに必要とする最適なヒントを予測する新たなモデルを提案する。評価実験では実データを用いて学習者が課題に正答するために必要とするヒントを予測し、実際のデータと比較して提案手法の有効性を示す。電気通信大学202

New measurement paradigms

This collection of New Measurement Paradigms papers represents a snapshot of the variety of measurement methods in use at the time of writing across several projects funded by the National Science Foundation (US) through its REESE and DR K–12 programs. All of the projects are developing and testing intelligent learning environments that seek to carefully measure and promote student learning, and the purpose of this collection of papers is to describe and illustrate the use of several measurement methods employed to achieve this. The papers are deliberately short because they are designed to introduce the methods in use and not to be a textbook chapter on each method. The New Measurement Paradigms collection is designed to serve as a reference point for researchers who are working in projects that are creating e-learning environments in which there is a need to make judgments about students’ levels of knowledge and skills, or for those interested in this but who have not yet delved into these methods

Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

Investigating Learning in an Intelligent Tutoring System through Randomized Controlled Experiments

In the United States, many students are doing poorly on new high-stakes standards-based tests that are required by the No Child Left Behind Act of 2002. Teachers are expected to cover more material to address all of the topics covered in standardized tests, and instructional time is more precious than ever. Educators want to know that the interventions that they are using in their classrooms are effective for students of varying abilities. Many educational technologies rely on tutored problem solving, which requires students to work through problems step-by-step while the system provides hints and feedback, to improve student learning. Intelligent tutoring researchers, education scientists and cognitive scientists are interested in knowing whether tutored problem solving is effective and for whom. Intelligent tutoring systems have the ability to adapt to individual students but need to know what types of feedback to present to individual students for the best and most efficient learning results. This dissertation presents an evaluation of the ASSISTment System, an intelligent tutoring system for the domain of middle school mathematics. In general, students were found to learn when engaging in tutored problem solving in the ASSISTment System. Students using the ASSISTment System also learned more when compared to paper-and-pencil problem-solving. This dissertation puts together a series of randomized controlled studies to build a comprehensive theory about when different types of tutoring feedback are more appropriate in an intelligent tutoring system. Data from these studies were used to analyze whether interactive tutored problem solving in an intelligent tutoring system is more effective than less interactive methods of allowing students to solve problems. This dissertation is novel in that it presents a theory that designers of intelligent tutoring systems could use to better adapt their software to the needs of students. One of the interesting results showed is that the effectiveness of tutored problem solving in an intelligent tutoring system is dependent on the math proficiency of the students. Students with low math proficiency learned more when they engaged in interactive tutoring sessions where they worked on one step at a time, and students with high math proficiency learned more when they were given the whole solution at once. More interactive methods of tutoring take more time versus less interactive methods. The data showed that it is worth the extra time it takes for students with low math proficiency. The main contribution of this dissertation is the development of a comprehensive theory of when educational technologies should use tutored problem solving to help students learn compared to other feedback mechanisms such as hints on demand, worked out solutions, worked examples and educational web pages

隠れマルコフ項目反応モデルにおけるウィンドウサイズの最適化

近年，大規模公開オンライン講座を始めとするeラーニングシステムが大きな注目を集めている．しかし，多くのeラーニングシステムは動的に学習者の知識状態を推定し，学習者に適応した支援をすることができていない．そのため，学習者の知識状態を推定するモデルの開発が大きな課題となっている．学習者の知識状態を推定する手法の一つに隠れマルコフ項目反応理論(HMIRT)が存在する．HMIRTは一般の項目反応理論(IRT)を時系列に拡張したモデルであり，以下の二つのパラメータを持つ．1)知識状態が過去の学習データにどれだけ依存するかを決定できるウィンドウサイズパラメータ．2)学習者の知識状態の変動幅を決定する分散パラメータ．既存のHMIRTでは，これらのパラメータが全ての時点に共通する，あらかじめ決定された固定パラメータであった．しかしながら，知識状態が過去の学習データにどの程度依存するかは取り組む項目によって異なるため，ウィンドウサイズパラメータを固定することで知識状態の推定精度が損なわれている恐れがある．さらに，分散パラメータを固定することで，学習者の知識状態の変動が全ての時点で一定となることもモデルの表現力を制限している．本研究では，これらの問題点を解決するために，ウィンドウサイズパラメータを各時点で変動できるよう拡張したHMIRTモデルを提案する．具体的には，貪欲法を用いて各項目ごとに最適なウィンドウサイズパラメータを推定する．評価実験では，実データを用いて従来のHMIRTモデルと提案モデルについて，未知の課題への反応予測精度を比較した．その結果，提案手法は既存手法と比較して高精度に未知の課題を予測できることが明らかとなり，ウィンドウサイズパラメータを各時点で変動させることが有効であることが示された．電気通信大学202

ベイズ母数推定を組み込んだDeep-IRT

テスト理論分野では，学習者のテスト(課題)への反応を基に，学習者の能力値を高精度に推定することが課題となっている。 近年では，学習者の能力値を正しく推定するために，従来からテスト理論分野で用いられている項目反応理論（Item Response Theory:IRT）に深層学習手法を組み合わせたDeep-IRTが開発されている．既存研究ではDeep-IRTはIRTより学習者の能力値を高精度に推定することが示されている。しかし、Deep-IRTはデータ数が少ない場合に学習データに過学習してしまう問題がある。本論文では、少数データにおける過学習を避けるためにベイズ母数推定を組み込んだDeep-IRTを提案する。提案手法ではニューラルネットワークにおける重みとバイアスパラメータを変分推定法を用いてベイズ推定することでパラメータの過学習を避けることができる。評価実験では少数データにおいて提案手法が既存手法よりも学習者の能力値を正しく推定することを示した。さらに，提案手法は学習者の課題への反応を 高精度に予測することを示した。電気通信大学202