Hubbold, Roger

Jay, Caroline

English

The University of Manchester - Institutional Repository

Quantifying the Effects of Latency on Sensory Feedback in Distributed Virtual Environments  Caroline Jay & Roger Hubbold School of Computer Science University of Manchester  The effects of delaying visual feedback in a Distributed Virtual Environment (DVE) are well documented. The effects of delaying haptic feedback, however, are less well understood. The few studies that address the issue indicate that latency in haptic feedback is detrimental to performance, sometimes to a greater extent than delaying visual feedback; however, someone has yet to produce a definitive model of performance in the face of latency in haptic feedback. As part of the Collaborative Haptics Project at the University of Manchester (http://aig.cs.man.ac.uk/research/cohap/cohap.php), a series of experiments have attempted to address this issue, by introducing increasing levels of latency to perceptual tasks shown to benefit from haptic feedback. Contrary to expectation, latency in haptic feedback had little or no effect on task performance, indicating that its role in these tasks is far less important than that of vision. The implications of these results, and the difficulties of designing a paradigm for effectively quantifying the effects of haptic latency are discussed. Introduction  Mackenzie & Ware (1993) conducted the first and arguably the most important quantitative research looking at the effects of visual latency, in a study where participants completed a Fitts’ Law target acquisition task using a desktop display and mouse. Participants had to move the mouse from a starting point to a target with a latency of between 25msec and 225msec from moving the mouse to seeing the cursor move on the screen. The study produced two particularly interesting results.  Firstly, the threshold at which latency started to affect performance was shown to be approximately 75msec.  Secondly, the effects of delay depended upon task difficulty: the harder the task, the greater the detriment caused by latency. They incorporated their findings in a predictive model of performance, which factored the performance deterioration caused by an increase in latency into the Fitts’ law model that described how selection times increased according to the Index of Difficulty (ID). At 0 latency, the time to move from the start to the target (MT) is best described by model 1 (Fitts, 1964), where C1 and C2 are experimentally determined constants.  However, when latency is introduced, it interacts with ID and the two have a multiplicative effect on movement time, as shown in model 2.  1. MT = C1 + C2ID 2. MT = C1 + (C2 + C3 LATENCY)ID This study is perhaps the best known analysis of how latency in visual feedback affects performance, and has been used by many researchers to inform the design of distributed virtual environments.  Other models of performance deterioration in the face of delayed visual feedback have been proposed by Ware & Balakrishnan (1994), who have extended Mackenzie & Ware’s (1993) model to account for 3D reaching movements, and Day, Holt & Russell (1999), who have explained the effects of much longer delays (between 2 and 6 seconds) in terms of working memory disruption.   As yet, there is no literature that offers a similar explanation of the effects of latency for the haptic domain, although there is evidence that delayed haptic feedback causes problems. A study of latency in telesurgery (Ottensmeyer et all 2000) demonstrated that surgeons are more sensitive to a latency in haptic feedback than visual when performing a laparoscopy task. However, the levels of latency examined (600 or 1200msec), and the complex nature of the task meant the study could not provide a generic model of performance.  The experiments described below investigate the possibility of quantifying the effects of latency in haptic feedback using paradigms similar to those that have been used to understand the effects of latency in visual feedback. Haptic Target Acquisition  In a target acquisition task, haptic feedback over the target area significantly decreases selection times (Akamatsu and Mackenzie, 1996). An experiment was conducted to examine how users respond when haptic and/or visual feedback was subject to increasing amounts of latency.  Experiment 1  Design The experiment used a Fitts’ law target acquisition task, in which the participant moved a cursor from a point on the left of the screen to a red target on the right (see Figure 1). The study had a 2 x 2 x 4 x 5 within-subjects factorial design. The factors were width of the target (2 or 4cm), distance of the target from the start (12 or 24cm), type of feedback (visual delayed with no haptic feedback, visual and haptic delayed by the same amount, visual in real time with haptic delayed, haptic in real time with visual delayed) and amount of delay (0, 25, 50, 75 and 150msec). Equipment The study used a PHANToM desktop force feedback device powered by a Dell Precision 420 with dual Pentium III CPUs and a ReachIn display with a 91Hz Sony Trinitron monitor, with a resolution of 1280 x 1024. Participants used a Microsoft USB optical mouse to signal when they had hit the target. The virtual environment was constructed using MAVERIK [6] and the GHOST SDK, updated at a rate of 67 frames per second.  Procedure Five female and seven male participants, aged between 22 and 28, took part. Each participant held the PHANToM stylus in their dominant hand and the mouse in their other hand.  To start a trial, the participant placed the cursor over the start (a blue square of 0.4cm). As soon as the target appeared (after approximately 2 seconds) the participant moved the cursor to the target, and clicked the left mouse button on hitting it. The target disappeared and the participant moved the cursor back to the start. Haptic feedback was provided by a vibration of the stylus when the cursor was over the target.  Each participant completed a block of practice trials and then 20 test blocks, one of each of the combinations of feedback and delay, which remained constant throughout the block.  The blocks were ordered according to a Latin square.   Each block consisted of 22 trials: two practice trials and 5 of each of the 4 combinations of amplitude and distance, presented at random.  After a rest, the participant completed the blocks of trials again, in the reverse order. After each block the participant was asked to rate on a scale of 1 to 5 (1 being “very easy”, 5 being “very difficult”) how hard they found that particular set of trials, using the practice trials as a baseline.  Participants were not explicitly informed that feedback would be subject to lag.                     cursor start target Figure 1: Target acquisition stimuli Results The mean target selection times under the various lag and feedback conditions are shown in Figure 2.  As expected, visual delay has a considerable effect on performance: as the duration of lag increases, so do selection times.  A repeated measures ANOVA confirms this, yielding a significant main effect of visual lag starting at 70 msec (F4,44 = 155.93, p<0.001).  We can also see that when the lag in haptic feedback is synchronized with the lag in visual feedback (the VdHd condition), selection times are significantly faster than they are in the Vd condition where there is no haptic feedback (F3,33 = 62.44, p<0.001).  However, this is the extent to which performance is influenced by haptic feedback. Delaying haptic feedback whilst keeping visual feedback in real time (VDd) fails to increase selection times, and providing real time haptic feedback when visual feedback is delayed (VdH) does not decrease them.  The pattern of errors and difficulty ratings tells a similar story. In the conditions where visual feedback is delayed, error rates increase significantly with latency (F4,44 = 6.18, p<0.001). Delaying haptic feedback alone has no effect on the error rates (see figure 3). Participants rated the VHd condition as significantly easier than the other conditions (F3,33 = 6.49, p<0.001), reinforcing the fact that delaying haptic feedback alone fails to disrupt task performance in the same way as delaying visual feedback.  There was no significant difference between the other conditions, indicating that participants found the lag in visual feedback uniformly disrupting, regardless of whether haptic feedback was present or delayed (see figure 4).    90010001100120013001400150020 45 70 95 120 145 170Delay (msec)Selection time (msec)VdVdHVHdVdHd      01234567820 45 70 95 120 145 170Delay (msec)Mean no errors VdVHdVdHVdHd    Figure 2: Mean selection times according to delay for each of the feedback conditions. Figure 3: Percentage of incorrect trials according to delay for each type of feedback. 1.522.533.520 45 70 95 120 145 170Delay (msec)Mean difficulty ratingVdVdHVHdVdHd    Conclusion The experiment provides clear evidence of a visual dominance effect. Although haptic feedback over the target area decreases selection times, it is only useful when it is synchronized with visual feedback. If it is not, users rely solely on visual feedback, whether it is in real time or delayed. This pattern continues in the error data and user difficulty ratings. When visual feedback is delayed, both of these measures increase significantly with latency; when haptic feedback is delayed there is no effect on either.  Experiment 2  It is possible that users become more confident at relying on haptic feedback to indicate presence over the target as they become more experienced at performing the task. A second experiment examined this by assessing the performance of three of the original participants as they performed the experiment a further 3 times over the course of a week.  Results The mean time to complete each trial after 4 sessions of the experiment is shown in Figure 5. The small advantage provided by haptic feedback when visual feedback was delayed in the first session is reduced to nothing by the fourth session. Participants learn to cope better with the visual delay, rendering the haptic feedback redundant.  Conclusion The advantage provided by haptic feedback disappears as users gain more practice. Latency in haptic feedback continues not to affect performance.   800900100011001200130014000 25 50 75 100 125 150Delay (msec)Selection time (msec)VdVdHVHdVdHd Figure 4: Mean questionnaire rating according to delay for each type of feedback. Figure 5: Mean selection times after 4 sessions  Reciprocal Tapping with Haptics  It is likely that delayed haptic feedback in a target acquisition task fails to have an effect as haptic feedback is not essential to the task, but merely supplements the visual information. A second study used a task where haptic feedback is far more important: reciprocal tapping between two targets. Here, haptic feedback is crucial, as it provides the primary indicator that the participant has hit the target.  Design The experiment used a 3 x 5 within-subjects factorial design.  Factors were level of delay (0, 25, 50, 75, 150 msec on top of the system delay – see 2.2) and type of feedback that was delayed (visual delayed, haptic in real-time; haptic delayed, visual in real-time; both visual and haptic delayed).  Equipment The study was run on a PC with an AMD Athlon XP 2100+ processor (1733MHz) and 512 MB RAM. A 23” Silicon Graphics monitor constituted the visual display. Force feedback was provided by an FCS Control Systems HapticMaster. The haptic response was handled by the haptic controller at 2.5KHz. The frame rate of 27 fps meant that the environment was updated on average every 37 msec.  As the function to compute extra lag was inside this rendering loop, this formed the base level of delay in all conditions. The monitor’s refresh rate of 76Hz introduced a further 7 msec of visual delay, making a mean system delay of 44 msec for visual feedback.  Stimuli The environment used in the experiment consisted of two targets, 5cm in width, positioned 15cm apart (see Figure 6).  Visual feedback was provided by an orange sphere which corresponded to the position of the HapticMaster end effector. The target area was haptically rendered to feel like a solid plane, thus touching the target with the end effector felt like hitting a hard surface.  Procedure Three male and three female participants, aged between 23 and 29, took part in the study. The participant placed the cursor on the left hand target and then tapped between the targets 30 times as quickly and accurately as possible.  Each trial was timed, starting from the point at which the cursor first left the left hand target.  Target misses were also recorded. After completing a short practice session, participants completed a block of 15 trials, one for each combination of lag and feedback, ordered according to a Latin square. After a 15 minute break, participants completed a second block, with the trials in the reverse order.  At the end of each trial, participants were asked to rate, on a scale of 1 to 5, how hard they found it to complete the trial, with 1 being very easy and 5 being very difficult. The practice session, where there was no additional latency, provided the baseline for their ratings.                   target target cursorFigure 6: The stimuli used in the tapping experiment   19002100230025002700290031003300350037000 25 50 75 100 125 150Additional lag (msec)Inter-tap interval (msec)VdHVHdVdHd  0123456789100 25 50 75 100 125 150Additional lag (msec)Target missesVdHVHdVdHd  11.522.533.544.50 25 50 75 100 125 150Additional lag (msec)Mean difficulty ratingVdHVHdVdHd   Results The results show that, as previously documented, delaying visual feedback seriously degrades performance. It is also the case that a lag in haptic feedback has an effect; however, this is far less pronounced than for visual latency. The mean inter-tap intervals (the time it takes to travel from one target to the other) for each condition can be seen in Figure 7. A repeated measures ANOVA reveals that as in [7], there are significant main effects for both lag (F4,20 = 47.958, p < 0.001) and type of feedback (F2,10 = 17.514, p < 0.001).  In the VdHd condition, pairwise comparisons showed that the main effect of lag (F4,20 = 22.361, p < 0.001) becomes significant at 69 msec (44 msec already in the system with the addition of 25 msec). In the VdH condition, selection times are not significantly affected until 94 msec (F4,20 = 31.284, p < 0.001).  Figure 7: Mean inter-tap interval Figure 8: Mean number of target misses Figure 9: Mean difficulty ratings Aside from this initial variation, overall the VdH and VdHd conditions do not differ significantly. There is however, as previously mentioned, a highly significant difference between these conditions and the VHd condition (where only haptic feedback is delayed).  Although performance deterioration does increase with haptic lag (F4,20 = 3.081, p < 0.05), this does not take effect until the 187 msec level (37 msec delay already in the system, with an additional 150 msec), indicating that in this task, although haptic lag has an effect, it is far less disruptive to users than visual delay.  The mean number of target misses that occurred in each condition is displayed in Figure 8. In the conditions where visual feedback was delayed, there was a main effect for the number of target misses, becoming significant when lag reached the 94 msec level (F4,20 = 40.1, p < 0.001).  However, a lag in haptic feedback did not lead to an increase in the number of errors participants made.  The results of the questionnaire broadly reflect the selection time and error rate data (see Figure 5). Participants start to rate the task as significantly harder in the VdHd condition when lag reaches 94 msec (F4,20 = 15.178, p < 0.001), and in the VdH condition when lag reaches 119 msec (F4,20 = 12.528, p < 0.001). In contrast to this, haptic lag has no effect on participants’ perception of task difficulty.  Conclusion Delaying haptic feedback in a reciprocal tapping task does cause performance to deteriorate, but only when the delay is considerable: while movement times are significantly slowed by a lag of just 69 msec in visual feedback, they do not increase in response to haptic latency until it approaches 200 msec. The increase in movement time also appears to be the only way in which haptic delay affects performance – participants were not more likely to miss the target as lag increased, nor did they find the task more difficult. Again, this contrasted strongly with delayed visual feedback, which increased the number of mistakes participants made and their perception of task difficulty. Quantifying the Effects of Latency on Haptic Feedback  There is evidence that in complex tasks, delaying haptic feedback significantly harms performance (Ottensmeyer et al, 2000). However, there are currently no models that predict exactly how performance deteriorates as a function of latency in haptic feedback. The potential of two paradigms for quantifying its effects was described here.   In the target acquisition experiments, visual information dominated to such an extent that haptic feedback was ignored when it was not synchronized with visual. In the reciprocal tapping experiment, haptic feedback was more important to the task, and was affected by latency, but was far less sensitive to it than visual feedback. In this case (and probably in the case of target acquisition), this occurred because haptic feedback was discrete, rather than continuous, like visual feedback.  In order to precisely quantify the effects of latency, a paradigm should be simple, ideally only investigating one type of movement at a time. If a task is too complex, it is impossible to determine exactly which aspects of performance are affected by latency, and in what manner. The results reported here show that haptic feedback must also be essential to the task and continuous in order to be affected by latency.  References  Akamatsu, M. & MacKenzie, S. (1996).  Movement characteristics using a mouse with tactile and force feedback. International Journal of Human-Computer Studies, 45, 483-493.  Day, P.N., Holt, P.O. & Russell G.T. (1999). Modeling the effects of delayed visual feedback in real-time operator control loops: A cognitive perspective. In J.L. Alty (Ed.) (1999) Proceedings of the XVIII European Annual Conference on Human Decision Making and Manual Control, Loughborough, Group D Publications Ltd, pp. 70-79.  MacKenzie, S. & Ware, C. (1993). Lag as a Determinant of Human Performance in Interactive Systems, Proceedings of the ACM Conference on Human Factors in Computing Systems - INTERCHI, 488-493.  Ottensmeyer MP, Hu J, Thompson JM, Ren J, and Sheridan TB (Aug 2000) Investigations into performance of minimally invasive telesurgery with feedback time delays. Presence 9 (4), 369-382.  Ware, C. & Balakrishnan,  (1994).  Target acquisition in fish tank VR: The effects of lag and frame rate.  In W. A. Davis and B. Joe (eds.) Proceedings of Graphics Interface '94, 1-7.    

Investigations into performance of minimally invasive telesurgery with feedback time delays.

Lag as a Determinant of Human Performance in Interactive Systems,

Modeling the effects of delayed visual feedback in real-time operator control loops: A cognitive perspective.

Movement characteristics using a mouse with tactile and force feedback.

Target acquisition in fish tank VR: The effects of lag and frame rate.

Quantifying the effects of latency on sensory feedback in distributed virtual environments

Quantifying the effects of latency on sensory feedback in distributed virtual environments

Abstract

Similar works

Full text

Available Versions

The University of Manchester - Institutional Repository