Conversational natural language interaction for place-related knowledge acquisition

We focus on the problems of using Natural Language inter- action to support pedestrians in their place-related knowledge acquisi- tion. Our case study for this discussion is a smartphone-based Natu- ral Language interface that allows users to acquire spatial and cultural knowledge of a city. The framework consists of a spoken dialogue-based information system and a smartphone client. The system is novel in com- bining geographic information system (GIS) modules such as a visibility engine with a question-answering (QA) system. Users can use the smart- phone client to engage in a variety of interleaved conversations such as navigating from A to B, using the QA functionality to learn more about points of interest (PoI) nearby, and searching for amenities and tourist attractions. This system explores a variety of research questions involving Natural Language interaction for acquisition of knowledge about space and place

Big data for smart operations and maintenance of buildings

The trend in industry to move towards smart buildings will in turn necessitate the move to smart operations and maintenance. As buildings lifecycle continues for a number of decades, various data about performance and operations need to be captured. There are various smart data collection tools available such as: mobile devices, social media, smart meters, sensors, satellites, camera footage, traffic flow reports, etc.. The analysis of the collected data can provide huge feedback for better buildings management. This research aims to adopt the concept of Big Data to capture the information and the knowledge of buildings operations; particularly for building maintenance and refurbishment. With the use of Building Information Modelling (BIM) systems to store various structured data of buildings, the unstructured data for various buildings operations will be also captured. For this purpose, a new system is proposed that integrates cloud-based Spoken Dialogue System (SDS), case-based reasoning, and BIM system. The proposed smart system acts as an interactive expert agent that seeks answers from buildings managers/users about building maintenance problems and help searching for solutions from previously stored knowledge cases. Capturing multi-modes data into BIM systems using the cloud-based spoken dialogue systems can utilise the high volume of data generated over building lifecycle. This can help design and operation teams to manage buildings, spaces, and services more efficiently. The data capture tools (including SDS) provide granular real-time data about utilization patterns which can improve the maintenance of buildings services and operations

LandMarkAR: An application to study virtual route instructions and the design of 3D landmarks for indoor pedestrian navigation with a mixed reality head-mounted display

Mixed Reality (MR) interfaces on head-mounted displays (HMDs) have the potential to replace screen-based interfaces as the primary interface to the digital world. They potentially offer a more immersive and less distracting experience compared to mobile phones, allowing users to stay focused on their environment and main goals while accessing digital information. Due to their ability to gracefully embed virtual information in the environment, MR HMDs could potentially alleviate some of the issues plaguing users of mobile pedestrian navigation systems, such as distraction, diminished route recall, and reduced spatial knowledge acquisition. However, the complexity of MR technology presents significant challenges, particularly for researchers with limited programming knowledge. This thesis presents “LandMarkAR” to address those challenges. “LandMarkAR” is a HoloLens application that allows researchers to create augmented territories to study human navigation with MR interfaces, even if they have little programming knowledge. “LandMarkAR” was designed using different methods from human-centered design (HCD), such as design thinking and think-aloud testing, and was developed with Unity and the Mixed Reality Toolkit (MRTK). With “LandMarkAR”, researchers can place and manipulate 3D objects as holograms in real-time, facilitating indoor navigation experiments using 3D objects that serve as turn-by-turn instructions, highlights of physical landmarks, or other information researchers may come up with. Researchers with varying technical expertise will be able to use “LandMarkAR” for MR navigation studies. They can opt to utilize the easy-to-use User Interface (UI) on the HoloLens or add custom functionality to the application directly in Unity. “LandMarkAR” empowers researchers to explore the full potential of MR interfaces in human navigation and create meaningful insights for their studies

Methodological triangulation in movement pattern research

The application of GIS and GPS methods together with qualitative sociological methods is one of the current fields of discussion in studies of human behaviour in space. The authors ask about the justification, manner and value of using the ‘triad’: quantitative GIS measurement methods – qualitative sociological methods – quasi-experiment. A mixed-method approach in an analysis of human movement patterns is introduced. Also, the role of the investigator in such projects is discussed

Tools in and out of sight : an analysis informed by Cultural-Historical Activity Theory of audio-haptic activities involving people with visual impairments supported by technology

The main purpose of this thesis is to present a Cultural-Historical Activity Theory (CHAT) based analysis of the activities conducted by and with visually impaired users supported by audio-haptic technology.This thesis covers several studies conducted in two projects. The studies evaluate the use of audio-haptic technologies to support and/or mediate the activities of people with visual impairment. The focus is on the activities involving access to two-dimensional information, such as pictures or maps. People with visual impairments can use commercially available solutions to explore static information (raised lined maps and pictures, for example). Solu-tions for dynamic access, such as drawing a picture or using a map while moving around, are more scarce. Two distinct projects were initiated to remedy the scarcity of dynamic access solutions, specifically focusing on two separate activities.The first project, HaptiMap, focused on pedestrian outdoors navigation through audio feedback and gestures mediated by a GPS equipped mobile phone. The second project, HIPP, focused on drawing and learning about 2D representations in a school setting with the help of haptic and audio feedback. In both cases, visual feedback was also present in the technology, enabling people with vision to take advantage of that modality too.The research questions addressed are: How can audio and haptic interaction mediate activ-ities for people with visual impairment? Are there features of the programming that help or hinder this mediation? How can CHAT, and specifically the Activity Checklist, be used to shape the design process, when designing audio haptic technology together with persons with visual impairments?Results show the usefulness of the Activity Checklist as a tool in the design process, and provide practical application examples. A general conclusion emphasises the importance of modularity, standards, and libre software in rehabilitation technology to support the development of the activities over time and to let the code evolve with them, as a lifelong iterative development process. The research also provides specific design recommendations for the design of the type of audio haptic systems involved

Nonstrict hierarchical reinforcement learning for interactive systems and robots

Conversational systems and robots that use reinforcement learning for policy optimization in large domains often face the problem of limited scalability. This problem has been addressed either by using function approximation techniques that estimate the approximate true value function of a policy or by using a hierarchical decomposition of a learning task into subtasks. We present a novel approach for dialogue policy optimization that combines the benefits of both hierarchical control and function approximation and that allows flexible transitions between dialogue subtasks to give human users more control over the dialogue. To this end, each reinforcement learning agent in the hierarchy is extended with a subtask transition function and a dynamic state space to allow flexible switching between subdialogues. In addition, the subtask policies are represented with linear function approximation in order to generalize the decision making to situations unseen in training. Our proposed approach is evaluated in an interactive conversational robot that learns to play quiz games. Experimental results, using simulation and real users, provide evidence that our proposed approach can lead to more flexible (natural) interactions than strict hierarchical control and that it is preferred by human users

DYNAMICS OF COLLABORATIVE NAVIGATION AND APPLYING DATA DRIVEN METHODS TO IMPROVE PEDESTRIAN NAVIGATION INSTRUCTIONS AT DECISION POINTS FOR PEOPLE OF VARYING SPATIAL APTITUDES

Cognitive Geography seeks to understand individual decision-making variations based on fundamental cognitive differences between people of varying spatial aptitudes. Understanding fundamental behavioral discrepancies among individuals is an important step to improve navigation algorithms and the overall travel experience. Contemporary navigation aids, although helpful in providing turn-by-turn directions, lack important capabilities to distinguish decision points for their features and importance. Existing systems lack the ability to generate landmark or decision point based instructions using real-time or crowd sourced data. Systems cannot customize personalized instructions for individuals based on inherent spatial ability, travel history, or situations. This dissertation presents a novel experimental setup to examine simultaneous wayfinding behavior for people of varying spatial abilities. This study reveals discrepancies in the information processing, landmark preference and spatial information communication among groups possessing differing abilities. Empirical data is used to validate computational salience techniques that endeavor to predict the difficulty of decision point use from the structure of the routes. Outlink score and outflux score, two meta-algorithms that derive secondary scores from existing metrics of network analysis, are explored. These two algorithms approximate human cognitive variation in navigation by analyzing neighboring and directional effect properties of decision point nodes within a routing network. The results are validated by a human wayfinding experiment, results show that these metrics generally improve the prediction of errors. In addition, a model of personalized weighting for users\u27 characteristics is derived from a SVMrank machine learning method. Such a system can effectively rank decision point difficulty based on user behavior and derive weighted models for navigators that reflect their individual tendencies. The weights reflect certain characteristics of groups. Such models can serve as personal travel profiles, and potentially be used to complement sense-of-direction surveys in classifying wayfinders. A prototype with augmented instructions for pedestrian navigation is created and tested, with particular focus on investigating how augmented instructions at particular decision points affect spatial learning. The results demonstrate that survey knowledge acquisition is improved for people with low spatial ability while decreased for people of high spatial ability. Finally, contributions are summarized, conclusions are provided, and future implications are discussed