Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance

The debate on effectiveness of virtual and mixed reality (VR/MR) tools for training professionals and operators is long-running with prominent contributions arguing that there are several shortfalls of experimental approaches and assessment criteria reported within the literature. In the automotive context, although car-makers were pioneers in the use of VR/MR tools for supporting designers, researchers started only recently to explore the effectiveness of VR/MR systems as mean for driving external operators of service centres to acquire the procedural skills necessary for car maintenance processes. In fact, from 463 journal articles on VR/MR tools for training published in the last thirty years, we identified only eight articles in which researchers experimentally tested the effectiveness of VR/MR tools for training service operators’ skills. To survey the current findings and the deficiencies of these eight studies, we use two main drivers: i) a well-known framework of organizational training programmes, and ii) a list of eleven evaluation criteria widely applied by researchers of different fields for assessing the effectiveness of training carried out with VR/MR systems. The analysis that we present allows us to: i) identify a trend among automotive researchers of focusing their analysis only on car service operators’ performance in terms of time and errors, by leaving unexplored important pre- and post-training aspects that could affect the effectiveness of VR/MR tools to deliver training contents – e.g., people skills, previous experience, cibersickness, presence and engagement, usability and satisfaction and ii) outline the future challenges for designing and assessing VR/MR tools for training car service operators

When simulated environments make the difference: the effectiveness of different types of training of car service procedures

An empirical analysis was performed to compare the effectiveness of different approaches to training a set of procedural skills to a sample of novice trainees. Sixty-five participants were randomly assigned to one of the following three training groups: (1) learning-by-doing in a 3D desktop virtual environment, (2) learning-by-observing a video (show-and-tell) explanation of the procedures, and (3) trial-and-error. In each group, participants were trained on two car service procedures. Participants were recalled to perform a procedure either 2 or 4 weeks after the training. The results showed that: (1) participants trained through the virtual approach of learning-by-doing performed both procedures significantly better (i.e. p < .05 in terms of errors and time) than people of non-virtual groups, (2) the virtual training group, after a period of non-use, were more effective than non-virtual training (i.e. p < .05) in their ability to recover their skills, (3) after a (simulated) long period from the training—i.e. up to 12 weeks—people who experienced 3D environments consistently performed better than people who received other kinds of training. The results also suggested that independently from the training group, trainees’ visuospatial abilities were a predictor of performance, at least for the complex service procedure, adj R2 = .460, and that post-training performances of people trained through virtual learning-by-doing are not affected by learning styles. Finally, a strong relationship (p < .001, R2 = .441) was identified between usability and trust in the use of the virtual training tool—i.e. the more the system was perceived as usable, the more it was perceived as trustable to acquire the competences

The influence of visual feedback and gender dynamics on performance, perception and communication strategies in CSCW

The effects of gender in human communication and human-computer interaction are well-known, yet little is understood about how it influences performance in the complex, collaborative tasks in computer-mediated settings – referred to as Computer-Supported Collaborative Work (CSCW) – that are increasingly fundamental to the way in which people work. In such tasks, visual feedback about objects and events is particularly valuable because it facilitates joint reference and attention, and enables the monitoring of people’s actions and task progress. As such, software to support CSCW frequently provides shared visual workspace. While numerous studies describe and explain the impact of visual feedback in CSCW, research has not considered whether there are differences in how females and males use it, are aided by it, or are affected by its absence. To address these knowledge gaps, this study explores the effect of gender – and its interactions within pairs – in CSCW, with and without visual feedback. An experimental study is reported in which mixed-gender and same-gender pairs communicate to complete a collaborative navigation task, with one of the participants being under the impression that s/he is interacting with a robot (to avoid gender-related social preconceptions). The study analyses performance, perceptions and communication strategies. As predicted, there was a significant benefit associated with visual feedback in terms of language economy and efficiency. However, it was also found that visual feedback may be disruptive to task performance, because it relaxes the users’ precision criteria and inflates their assumptions of shared perspective. While no actual performance difference was found between males and females in the navigation task, females rated their own performance less positively than did males. In terms of communication strategies, males had a strong tendency to introduce novel vocabulary when communication problems occurred, while females exhibited more conservative behaviour. When visual feedback was removed, females adapted their strategies drastically and effectively, increasing the quality and specificity of the verbal interaction, repeating and re-using vocabulary, while the behaviour of males remained consistent. These results are used to produce design recommendations for CSCW systems that will suit users of both genders and enable effective collaboration

A method for mapping and measuring users' mental models of the depth/breadth tradeoff

Understanding users' mental models can improve design and testing of interactive systems. Yet, the extraction of users' mental models and their representation are still a tough challenge. We addressed this question by focusing on how the depth/breadth tradeoff in web navigation structures is reflected in users' mental models. The approach presented here is based on gathering the data in an unlimited-hierarchy variation of card sorting technique and analyzing the results using hierarchic cluster analysis visualized with dendograms. We report a pilot study of our approach and compare the hierarchic cluster analysis and dendograms with ANOVA. The findings show significantly different perceptions of breadth versus depth in two different devices, a desktop computer and cellular phone. These findings are in contrast with findings of more traditional mental model assessment approaches, thus validating the necessity and the usefulness of the purposed new approach

Web navigation structures in cellular phones: The depth/breadth trade-off issue

One can browse the web with a variety of devices, including hand-held devices such as the cellular phone. The small screen of those devices poses some serious usability issues, one of which is the appropriate hierarchy depth of the web site. In this study, we empirically examined whether a broad navigation structure, which was found to be superior in regular screen-size platforms, also has an advantage for a small-screen device such as the cellular phone where it may require more movements and scrolling between screens of the same hierarchical level. Navigation times and success rates were measured for two search tasks in a mock web site that was built in two versions: one with a broad navigation structure and the other with a deep structure. Both structures were tested with cellular phone emulation and a standard desktop personal computer (PC). Results indicate that performance was better with the broad navigation structure for both the cellular phone and the PC. In addition, performance was better with the PC as compared to the cellular phone, and this difference was pronounced in the broad structure. The results are discussed in terms of the impact of device-independent characteristics of the hierarchy depth along with the theoretical account of increased working memory load, confusion and disorientation associated more with deep structures

Task complexity moderates the influence of descriptions in decisions from experience

Decisions-makers often have access to a combination of descriptive and experiential information, but limited research so far has explored decisions made using both. Three experiments explore the relationship between task complexity and the influence of descriptions. We show that in simple experience-based decision-making tasks, providing congruent descriptions has little influence on task performance in comparison to experience alone without descriptions, since learning via experience is relatively easy. In more complex tasks, which are slower and more demanding to learn experientially, descriptions have stronger influence and help participants identify their preferred choices. However, when the task gets too complex to be concisely described, the influence of descriptions is reduced hence showing a non-monotonic pattern of influence of descriptions according to task complexity. We also propose a cognitive model that incorporates descriptive information into the traditional reinforcement learning framework, with the impact of descriptions moderated by task complexity. This model fits the observed behavior better than previous models and replicates the observed non-monotonic relationship between impact of descriptions and task complexity. This research has implications for the development of effective warning labels that rely on simple descriptive information to trigger safer behavior in complex environments