We present a method of evaluating the immediate performance impact of user state misclassifications in spoken dialogue systems. We illustrate the method with a tutoring systemthatadaptstostudentuncertaintyoverand above correctness. First we define a ranking of user states representing local performance. Second, we compare user state trajectories when the first state is accurately classified versus misclassified. Trajectories are quantified using a previously proposed metricrepresentingthelikelihoodoftransitioning fromoneuserstatetoanother. Comparison of thetwosetsoftrajectoriesshowswhetheruser state misclassifications change the likelihood of subsequent higher or lower ranked states, relativetoaccurateclassification. Ourtutoring system results illustrate the case where user statemisclassificationincreasesthelikelihood of negative performance trajectories as compared toaccurate classification.
