Wearable sensor metric for fidgeting: screen engagement rather than interest causes NIMI of wrists and ankles

Measuring fidgeting is an important goal for the psychology of mind-wandering and for human computer interaction (HCI). Previous work measuring the movement of the head, torso and thigh during HCI has shown that engaging screen content leads to non-instrumental movement inhibition (NIMI). Camera-based methods for measuring wrist movements are limited by the occlusion, supination and friction of the arm. Here we used a high pass filtered magnitude of wearable tri-axial accelerometer recordings during 2-minute passive HCI stimuli as a surrogate for movement of the wrists and ankles. With 24 seated, healthy volunteers experiencing HCI, this metric showed that wrists moved significantly more than ankles. We found that NIMI could be detected in the wrists and ankles; it distinguished extremes of interest and boredom via restlessness. We conclude that both free-willed and forced screen engagement can elicit NIMI of the wrists and ankles

A trigger-substrate model for smiling during an automated formative quiz: engagement is the substrate, not frustration

INTRODUCTION: Automated tutoring systems aim to respond to the learner’s cognitive state in order to maintain engagement. The end-user’s state might be inferred by interactive timings, bodily movements or facial expressions. Problematic computerized stimuli are known to cause smiling during periods of frustration. METHODS: Forty-four seated, healthy participants (age range 18-35, 18 male) used a handheld trackball to answer a computer-presented, formative, 3-way multiple choice geography quiz, with 9 questions, lasting a total of 175 seconds. Frontal facial videos (10 Hz) were collected with a webcam and processed for facial expressions by CrowdEmotion using a pattern recognition algorithm. Interactivity was recorded by a keystroke logger (Inputlog 5.2). Subjective responses were collected immediately after each quiz using a panel of visual analogue scales (VAS). RESULTS: Smiling was fie-fold enriched during the instantaneous feedback segments of the quiz, and this was correlated with VAS ratings for engagement but not with happiness or frustration. Nevertheless, smiling rate was significantly higher after wrong answers compared to correct ones, and frustration was correlated with the number of questions answered incorrectly. CONCLUSION: The apparent disconnect between the increased smiling during incorrect answers but the lack of correlation between VAS frustration and smiles suggests a trigger-substrate model where engagement is the permissive substrate, while the noises made by the quiz after wrong answers may be the trigger