Towards a Unified Knowledge-Based Approach to Modality Choice

This paper advances a unified knowledge-based approach to the process of choosing the most appropriate modality or combination of modalities in multimodal output generation. We propose a Modality Ontology (MO) that models the knowledge needed to support the two most fundamental processes determining modality choice – modality allocation (choosing the modality or set of modalities that can best support a particular type of information) and modality combination (selecting an optimal final combination of modalities). In the proposed ontology we model the main levels which collectively determine the characteristics of each modality and the specific relationships between different modalities that are important for multi-modal meaning making. This ontology aims to support the automatic selection of modalities and combinations of modalities that are suitable to convey the meaning of the intended message

Designing a training tool for imaging mental models

The training process can be conceptualized as the student acquiring an evolutionary sequence of classification-problem solving mental models. For example a physician learns (1) classification systems for patient symptoms, diagnostic procedures, diseases, and therapeutic interventions and (2) interrelationships among these classifications (e.g., how to use diagnostic procedures to collect data about a patient's symptoms in order to identify the disease so that therapeutic measures can be taken. This project developed functional specifications for a computer-based tool, Mental Link, that allows the evaluative imaging of such mental models. The fundamental design approach underlying this representational medium is traversal of virtual cognition space. Typically intangible cognitive entities and links among them are visible as a three-dimensional web that represents a knowledge structure. The tool has a high degree of flexibility and customizability to allow extension to other types of uses, such a front-end to an intelligent tutoring system, knowledge base, hypermedia system, or semantic network

A collaborative-project memory tool for participatory planning

Technology is more and more providing planners and designer with tools and methods to collect and communicate spatial data and assist spatial analysis. When we think about new technologies supporting planning we mainly think about GIS, urban modelling, simulation models and virtual reality. But many other challenges to the planning practice need for tools to support and improve planning activities. In this paper we discuss the need of new tools to support knowledge representation and knowledge sharing in participatory planning processes. The paper describes the use of a hypermedia and sensemaking tool (Compendium) to structure the knowledge produced in a real participatory planning process. In the present application Compendium has been used not for real-time capturing but for a post-hoc analysis of a real participatory planning experience. Compendium has been used to represent and reconstruct the group memory of consultation meetings in order to allow both the planning team and the citizens to navigate into the contents of those meetings. Moreover the paper describes the main features and potential of the use of Compendium in Participatory Planning domain, and it describes the results of the group memory reconstruction. Finally the case study opens reflections on the need of new planning technologies supporting participatory knowledge generation, representation and management

From signal to substance and back: Insights from environmental sound research to auditory display design

Presented at the 15th International Conference on Auditory Display (ICAD2009), Copenhagen, Denmark, May 18-22, 2009A persistent concern in the field of auditory display design has been how to effectively use environmental sounds, which are naturally occurring familiar non-speech, non-musical sounds. Environmental sounds represent physical events in the everyday world, and thus they have a semantic content that enables learning and recognition. However, unless used appropriately, their functions in auditory displays may cause problems. One of the main considerations in using environmental sounds as auditory icons is how to ensure the identifiability of the sound sources. The identifiability of an auditory icon depends on both the intrinsic acoustic properties of the sound it represents, and on the semantic fit of the sound to its context, i.e., whether the context is one in which the sound naturally occurs or would be unlikely to occur. Relatively recent research has yielded some insights into both of these factors. A second major consideration is how to use the source properties to represent events in the auditory display. This entails parameterizing the environmental sounds so the acoustics will both relate to source properties familiar to the user and convey meaningful new information to the user. Finally, particular considerations come into play when designing auditory displays for special populations, such as hearing impaired listeners who may not have access to all the acoustic information available to a normal hearing listener, or to elderly or other individuals whose cognitive resources may be diminished. Some guidelines for designing displays for these populations will be outlined

Supporting multimedia user interface design using mental models and representational expressiveness

This thesis addresses the problem of output media allocation in the design of multimedia user interfaces. The literature survey identifies a formal definition of the representational capabilities of different media.as important in this task. Equally important, though less prominent in the literature, is that the correct mental model of a domain is paramount for the successful completion of tasks. The thesis proposes an original linguistic and cognitive based descriptive framework, in two parts. The first part defines expressiveness, the amount of representational abstraction a medium provides over any domain. The second part describes how this expressiveness is linked to the mental models that media induce, and how this in turn affects task performance. It is postulated that the mental models induced by different media, will reflect the abstractive representation those media offer over the task domain. This must then be matched to the abstraction required by tasks to allow them to be effectively accomplished. A 34 subject experiment compares five media, of two levels of expressiveness, over a range of tasks, in a complex and dynamic domain. The results indicate that expressiveness may allow media to be matched more closely to tasks, if the mental models they are known to induce are considered. Finally, the thesis proposes a tentative framework for media allocation, and two example interfaces are designed using this framework. This framework is based on the matching of expressiveness to the abstraction of a domain required by tasks. The need for the methodology to take account of the user's cognitive capabilities is stressed, and the experimental results are seen as the beginning of this procedure

The Effect of Developmental Factors on the Use of an Electronic Communication Device.

This study addressed a poorly understood variable of voice output communication aid (VOCA) use, namely mnemonic development. Forty children without disabilities, aged from 49 to 60 months, were required to use an IntroTalker (Prentke Romich Company) to participate in a one-on-one storybook reading task with the investigator. They were provided with sixteen messages to use during a scripted reading of The Three Bears. Children could use a trained verbal elaboration strategy or generate their own methods to retrieve messages. Immediate and delayed responses of children under two training methods were compared, and the contribution of developmental factors, including metamemory, were evaluated. In the control method, Direct Instruction (DI), children learned to retrieve messages via an error-free method including simple repetition of a verbal elaboration mnemonic. In the experimental method, Active Problem Solving (APS), children were first led through a series of steps alerting them to the limitations of non-symbolic retrieval strategies and allowing them to make retrieval errors, thus discovered the relative value of various retrieval strategies. Next, they were taught to retrieve the remaining messages in the same way as the DI children. Children with low metamemory skill were less accurate in their retrieval of messages, and produced more Self-corrected Responses when trained under the APS condition. Children receiving APS training showed an initial retrieval advantage on the first story-retelling immediately following training; however, on the second day retrieval by DI subjects matched APS. It was concluded that DI children experienced a de facto active problem-solving condition on the first day, when they produced many errors, thereby enhancing the value of the verbal elaborations they had been trained to use. Effects of perceptual, semantic and syntactic development, and of conflicts between the canonical and experimental Three Bears script were evident. Although the use of a normally developing population is controversial in the area of augmentative communication, the strategy was found to produce valuable insights into the errors made by children. Overall, the task was easy for most of the children, thereby supporting the use of VOCAs with young children

Designing informative warning signals: Effects of indicator type, modality, and task demand on recognition speed and accuracy

An experiment investigated the assumption that natural indicators which exploit existing learned associations between a signal and an event make more effective warnings than previously unlearned symbolic indicators. Signal modality (visual, auditory) and task demand (low, high) were also manipulated. Warning effectiveness was indexed by accuracy and reaction time (RT) recorded during training and dual task test phases. Thirty-six participants were trained to recognize 4 natural and 4 symbolic indicators, either visual or auditory, paired with critical incidents from an aviation context. As hypothesized, accuracy was greater and RT was faster in response to natural indicators during the training phase. This pattern of responding was upheld in test phase conditions with respect to accuracy but observed in RT only in test phase conditions involving high demand and the auditory modality. Using the experiment as a specific example, we argue for the importance of considering the cognitive contribution of the user (viz., prior learned associations) in the warning design process. Drawing on semiotics and cognitive psychology, we highlight the indexical nature of so-called auditory icons or natural indicators and argue that the cogniser is an indispensable element in the tripartite nature of signification