Reference Resolution in Multi-modal Interaction: Position paper

In this position paper we present our research on multimodal interaction in and with virtual environments. The aim of this presentation is to emphasize the necessity to spend more research on reference resolution in multimodal contexts. In multi-modal interaction the human conversational partner can apply more than one modality in conveying his or her message to the environment in which a computer detects and interprets signals from different modalities. We show some naturally arising problems and how they are treated for different contexts. No generally applicable solutions are given

Maps, agents and dialogue for exploring a virtual world

In previous years we have been involved in several projects in which users (or visitors) had to find their way in information-rich virtual environments. 'Information-rich' means that the users do not know beforehand what is available in the environment, where to go in the environment to find the information and, moreover, users or visitors do not necessarily know exactly what they are looking for. Information-rich means also that the information may change during time. A second visit to the same environment will require different behavior of the visitor in order for him or her to obtain similar information than was available during a previous visit. In this paper we report about two projects and discuss our attempts to generalize from the different approaches and application domains to obtain a library of methods and tools to design and implement intelligent agents that inhabit virtual environments and where the agents support the navigation of the user/visitor

Agent Assistance: From Problem Solving to Music Teaching

We report on our research on agents that act and behave in a web learning environment. This research is part of a general approach to agents acting and behaving in virtual environments where they are involved in providing information, performing transactions, demonstrating products and, more generally, assisting users or visitors of the web environment in doing what they want or have been asked to do. While initially we hardly provided our agents with 'teaching knowledge', we now are in the process of making such knowledge explicit, especially in models that take into account that assisting and teaching takes place in a visualized and information-rich environment. Our main (embodied) tutor-agent is called Jacob; it knows about the Towers of Hanoi, a well-known problem that is offered to CS students to learn about recursion. Other agents we are working on assist a visitor in navigating in a virtual world or help the visitor in getting information. We are now designing a music teacher - using knowledge of software engineering and how to design multi-modal interactions, from previous projects

Reference resolution in multi-modal interaction: Preliminary observations

In this paper we present our research on multimodal interaction in and with virtual environments. The aim of this presentation is to emphasize the necessity to spend more research on reference resolution in multimodal contexts. In multi-modal interaction the human conversational partner can apply more than one modality in conveying his or her message to the environment in which a computer detects and interprets signals from different modalities. We show some naturally arising problems but do not give general solutions. Rather we decide to perform more detailed research on reference resolution in uni-modal contexts to obtain methods generalizable to multi-modal contexts. Since we try to build applications for a Dutch audience and since hardly any research has been done on reference resolution for Dutch, we give results on the resolution of anaphoric and deictic references in Dutch texts. We hope to be able to extend these results to our multimodal contexts later

Towards Communicating Agents and Avatars in Virtual Worlds

We report about ongoing research in a virtual reality environment where visitors can interact with agents that help them to obtain information, to perform certain transactions and to collaborate with them in order to get some tasks done. In addition, in a multi-user version of the system visitors can chat with each other. Our environment is a laboratory for research and for experiments with users interacting with agents in multimodal ways, referring to visualized information and making use of knowledge possessed by domain agents, but also by agents that represent other visitors of this environment. We discuss standards that are under development for designing such environments. Our environment models a local theatre in our hometown. We discuss our attempts to let this environment evolve into a theatre community where we do not only have goal-directed visitors buying tickets, but also visitors that that are not yet sure whether they want to buy or just want information or visitors who just want to look around, talk with others, etc. It is shown that we need a multi-user and multi-agent environment to realize our goals and that we need to have a unifying framework in order to be able to introduce and maintain different agents and user avatars with different abilities, including intellectual, interaction and animation abilities

Towards Multi-Modal Interactions in Virtual Environments: A Case Study

We present research on visualization and interaction in a realistic model of an existing theatre. This existing ‘Muziek¬centrum’ offers its visitors information about performances by means of a yearly brochure. In addition, it is possible to get information at an information desk in the theatre (during office hours), to get information by phone (by talking to a human or by using IVR). The database of the theater holds the information that is available at the beginning of the ‘theatre season’. Our aim is to make this information more accessible by using multi-modal accessible multi-media web pages. A more general aim is to do research in the area of web-based services, in particu¬lar interactions in virtual environments

Towards virtual communities on the Web: Actors and audience

We report about ongoing research in a virtual reality environment where visitors can interact with agents that help them to obtain information, to perform certain transactions and to collaborate with them in order to get some tasks done. Our environment models a theatre in our hometown. We discuss attempts to let this environment evolve into a theatre community where we do not only have goal-directed visitors, but also visitors that that are not sure whether they want to buy or just want information or visitors who just want to look around. It is shown that we need a multi-user and multiagent environment to realize our goals. Since our environment models a theatre it is also interesting to investigate the roles of performers and audience in this environment. For that reason we discuss capabilities and personalities of agents. Some notes on the historical development of networked communities are included

The display of electronic commerce within virtual environments

In today’s competitive business environment, the majority of companies are expected to be represented on the Internet in the form of an electronic commerce site. In an effort to keep up with current business trends, certain aspects of interface design such as those related to navigation and perception may be overlooked. For instance, the manner in which a visitor to the site might perceive the information displayed or the ease with which they navigate through the site may not be taken into consideration. This paper reports on the evaluation of the electronic commerce sites of three different companies, focusing specifically on the human factors issues such as perception and navigation. Heuristic evaluation, the most popular method for investigating user interface design, is the technique employed to assess each of these sites. In light of the results from the analysis of the evaluation data, virtual environments are suggested as a way of improving the navigation and perception display constraints