Very Slow Search and Reach: Failure to Maximize Expected Gain in an Eye-Hand Coordination Task

We examined an eye-hand coordination task where optimal visual search and hand movement strategies were inter-related. Observers were asked to find and touch a target among five distractors on a touch screen. Their reward for touching the target was reduced by an amount proportional to how long they took to locate and reach to it. Coordinating the eye and the hand appropriately would markedly reduce the search-reach time. Using statistical decision theory we derived the sequence of interrelated eye and hand movements that would maximize expected gain and we predicted how hand movements should change as the eye gathered further information about target location. We recorded human observers' eye movements and hand movements and compared them with the optimal strategy that would have maximized expected gain. We found that most observers failed to adopt the optimal search-reach strategy. We analyze and describe the strategies they did adopt.Psycholog

Eye-hand dynamics in the search-reach task.

<p>Hand trajectories in the search-reach task are plotted for two typical observers, O1 (left sequence) and O8 (middle and right sequences). Each sequence is for one specific search order (labeled at top). Each panel is for one specific target position (labeled in the panel). Each trajectory is for one trial. Colors along the trajectories code stages of visual search. Red denotes that no objects have been examined. Cyan denotes that 1 object has been examined, and so on. Green denotes that the target has been found. Note where the hand trajectories change going directions and how the trajectories vary with search order.</p

Search slope and movement speed.

<p><b>A. Search time as a linear function of number of objects searched</b>. Data of a typical observer, O7, in the free search task. Each grey dot is for one trial. The black line is the fitting line, whose slope corresponds to the time to identify one object, . <b>B. Average trajectories for different target positions</b>. Data of the same observer O7 in the training of reach. Red filled circle denotes the starting position. Grey filled circles denote positions of objects. In each trail, the observer moved the finger from the starting position to one designated position. Black lines denote trajectories of hand movement averaged across trials. The number above an object denotes the measured movement speed towards the object (<i>cm/s</i>). Note that the movement speeds differ little in different movement directions. The mean speed across all target positions is regarded as the observer's movement speed, .</p

Visual search strategies in the search-reach task.

<p>To save time, observers should search the small cluster first rather than the large cluster first. The percentages of usage of the two visual search strategies were contrasted with each other. Star denotes a significant difference. Only two observers correctly used the small-cluster first strategy more often than the large-cluster first strategy. The rest of the observers showed no significant preference between the two.</p

An example of the stimulus array.

<p>The red circle is the starting position for the eye and the hand. Each gray circle with blue shapes inside is an object. Two clusters of objects are located to the left and right of the midline, on a virtual arc centered at the starting position. One cluster contains four objects, the other two. On half the trials the two-cluster is on the left as shown, on the other half, on the right. Each object is equally likely to be the target. One and only one of them is the target to be touched. See the Stimuli section for a definition of the target and distractors.</p

Hand movement strategies in the search-reach task.

<p><b>A. Mean movement speed before the target was found</b>. Error bar denotes the 95% confidence interval. Before the target was found, if the observer did not move at all, it might cost her more than 30% of the rewards. Observers' actual movement speed was significantly larger than zero, but six of the eight observers moved significantly and much slower than they did in the training of reach. <b>B. The update of movement directions for different orders of visual search</b>. In a trial where the observer searched no fewer than three objects before locating the target, we compared the position of the finger at the end of fixating the third object with that of the first object. The difference of angle (inset), , is plot separately for each observer and the search strategies of small-cluster first and large-cluster first. Positive for shifts towards the small cluster, negative for shifts towards the large cluster. Error bar denotes the standard error. The large-cluster first strategy for O1 and O2 had fewer than five valid trials and were not plot therefore. By the optimal hand movement strategy, the observer should shift towards the large cluster when searching the small cluster first, thus a negative , and vice versa. Six of the observers (O1∼O6) showed such a tendency. When searching small cluster first, five observers shifted significantly more towards the large cluster than that of large-cluster first, or significantly towards the large cluster (for observers who had not a large-cluster first comparison). Stars denote significant difference. <b>C. Mean initial movement direction for different orders of visual search</b>. For each trial with no less than one object searched before the target, the initial movement direction was defined as the direction from the starting position to the position of the finger at the end of fixating the first object. The direction was quantified in a polar coordinate centered at the starting position. The direction to the right was 0 degree; to the up, 90 degrees; to the left, 180 degrees. Error bar denotes the 95% confidence interval. The solid line denotes the optimal initial movement direction, which is towards the centroid of the six objects. The dash line denotes the middle of the two clusters. There was no significant difference in initial movement direction between the search orders. According to the mean initial movement direction, only two of the eight observers did not deviate significantly from optimal, while six of them were indistinguishable from a possible strategy of moving towards the middle of the two clusters.</p

Search-Reach: Human observers' performance compared to optimal.

<p><b>A. Efficiency</b>. Efficiency was defined as the average gain of successful trials divided by the maximal expected gain. Most observers were far from optimal. The median efficiency across observers was 78%. <b>B. Movement time after the target was found</b>. Black dot denotes the expected movement time after the target was found if the observer used the optimal visual search and hand movement strategies. All the observers' mean post-found movement time were larger than the minimum expected movement time. For 6 of the 8 observers, the difference was significant. In both A and B, each bar is for one observer. Error bar denotes the 95% confidence interval.</p

Optimal strategies of eye-hand coordination.

<p><b>A. Simulated expected search-reach time as a function of the order of search</b>. Each panel is for a different search and hand movement capacity. The objects are numbered from 1 to 6 (inset). Orders of search are indexed with sequences of numbers. For instance, 12 34 denotes the sequence of fixations: object 1 first, then 2, then 3, then 4, and finally either 5 and 6 or 6 and 5 in either order. See text. Different colors denote different orders of search. We assumed that the observer always uses the optimal strategy of hand movement that minimizes her expected search-reach time under the specific order of search. The displayed ranges of search time per object, , and hand movement speed, , include those for human observers measured in the experiment. Whatever the search or movement speed of the observer, the search order 1234, i.e. starting from the end of the small cluster and moving continuously towards the large cluster, leads to the minimum expected search-reach time. But note that in the range of subjects' visual search speed and hand movement speed, the difference between this optimal order and the other orders are larger when search is slower or hand movement is faster. The difference between 1234 and the other three orders starting from the small cluster (1265, 2134, and 2165) is negligible, but these four are obviously better than the two orders that start from the large cluster (6543 and 3456). For example, for a typical human observer (O7) with and , the simulated expected search-reach time of 1234 is 10.3 <i>s</i>. The other three orders of small-cluster first cause an increase of less than 0.1 <i>s</i>, but the larger-cluster first orders, 6543 and 3456, would cost 0.7 <i>s</i> and 1.2 <i>s</i> more. Given the reward structure, using the latter two inferior search orders would lead to a loss of 6% and 12%, relative to the optimal one. <b>B. Simulated optimal and close-to-optimal hand movement strategies</b>. The hand trajectories are simulated for a typical human observer (Observer 07) with and . Grey filled circles denote objects. The red filled circle at the bottom denotes the starting position of the finger. We illustrated the trajectories of optimal or close-to-optimal hand movement for two search orders, in different colors, blue for 1234, orange for 6543. Sequences of block arrows on the objects show the search orders. The lines originated from the starting position correspond to the trajectories of the finger. Each dot on a line marks the time when a new object gets identified. The optimal strategy for the hand (top) is the strategy that minimizes the expected search-reach time given a specific search order. The aim-for-centroid strategy (bottom) is to aim for the centroid of the objects that are still unidentified. The differences between the two panels are subtle. If the observer updates her movement aim after the identification of each new object, the expected search-reach time is almost the same as that of the optimal strategy.</p