Flights in my hands : coherence concerns in designing Strip'TIC, a tangible space for air traffic controllers

Best Paper Honorable Mention awardInternational audienceWe reflect upon the design of a paper-based tangible interactive space to support air traffic control. We have observed, studied, prototyped and discussed with controllers a new mixed interaction system based on Anoto, video projection, and tracking. Starting from the understanding of the benefits of tangible paper strips, our goal is to study how mixed physical and virtual augmented data can support the controllers' mental work. The context of the activity led us to depart from models that are proposed in tangible interfaces research where coherence is based on how physical objects are representative of virtual objects. We propose a new account of coherence in a mixed interaction system that integrates externalization mechanisms. We found that physical objects play two roles: they act both as representation of mental objects and as tangible artifacts for interacting with augmented features. We observed that virtual objects represent physical ones, and not the reverse, and, being virtual representations of physical objects, should seamlessly converge with the cognitive role of the physical object. Finally, we show how coherence is achieved by providing a seamless interactive space

Investigation and development of a tangible technology framework for highly complex and abstract concepts

The ubiquitous integration of computer-supported learning tools within the educational domain has led educators to continuously seek effective technological platforms for teaching and learning. Overcoming the inherent limitations of traditional educational approaches, interactive and tangible computing platforms have consequently garnered increased interest in the pursuit of embedding active learning pedagogies within curricula. However, whilst Tangible User Interface (TUI) systems have been successfully developed to edutain children in various research contexts, TUI architectures have seen limited deployment towards more advanced educational pursuits. Thus, in contrast to current domain research, this study investigates the effectiveness and suitability of adopting TUI systems for enhancing the learning experience of abstract and complex computational science and technology-based concepts within higher educational institutions (HEI)s. Based on the proposal of a contextually apt TUI architecture, the research describes the design and development of eight distinct TUI frameworks embodying innovate interactive paradigms through tabletop peripherals, graphical design factors, and active tangible manipulatives. These computationally coupled design elements are evaluated through summative and formative experimental methodologies for their ability to aid in the effective teaching and learning of diverse threshold concepts experienced in computational science. In addition, through the design and adoption of a technology acceptance model for educational technology (TAM4Edu), the suitability of TUI frameworks in HEI education is empirically evaluated across a myriad of determinants for modelling students’ behavioural intention. In light of the statistically significant results obtained in both academic knowledge gain (μ = 25.8%) and student satisfaction (μ = 12.7%), the study outlines the affordances provided through TUI design for various constituents of active learning theories and modalities. Thus, based on an empirical and pedagogical analyses, a set of design guidelines is defined within this research to direct the effective development of TUI design elements for teaching and learning abstract threshold concepts in HEI adaptations

Desenvolvimento e avaliação de um sistema de realidade aumentada para visualização cartográfica

Orientador : Prof. Dr. Jorge Antonio Silva CentenoTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências da Terra, Programa de Pós-Graduação em Ciências Geodésicas. Defesa: Curitiba, 16/02/2017Inclui referências : f. 174-95Resumo: Nas últimas décadas é notável o aparecimento de sistemas computacionais que favorecem os usuários a executar suas tarefas. Dentre os sistemas que prometem auxiliar os usuários a executar suas tarefas - inclusive as tarefas de leitura de mapa - os baseados em Realidade Aumentada (RA) mostram-se bastante promissores. Estas interfaces computacionais mais naturais podem possibilitar novas formas de interação com a informação e com o ambiente ao redor do usuário. Entretanto, pouco deste tema tem sido explorado, principalmente, no Brasil. Por esse motivo, elaborou-se a presente pesquisa com a intenção de desenvolver e analisar alguns protótipos de Realidade Aumentada para a Cartografia, para interação com mapas e dados geográficos. Neste trabalho abordou-se e utilizou-se de diferentes metodologias para RA, como com e sem marcadores fiduciais especiais, em versão desktop ou para dispositivos móveis. Foram utilizados dados geoespaciais referentes à características físicas, políticas, sociais e demográficas, econômicas e ambientais, integradas sobre diferentes tipos de mapas impressos, de temáticos a topográficos. Explorou-se também a utilização vários formatos de dados, de vetorial a raster, 2D, 3D, animações, etc. É possível destacar que os sistemas de Realidade Aumentada desenvolvidos trazem novas possibilidades de manipulação de dados geográficos, trazendo novas oportunidades de interação com o mapa impresso ou dados geoespaciais digitais. Porém, destaca-se que o desenvolvimento para dispositivos móveis e sem a utilização de marcadores fiduciais externos apresenta-se altamente atrativo, uma vez que este sistema de RA pode ser considerado como uma "lente mágica" ou uma "janela" para o mundo virtual, onde o usuário pode ter acesso a outras informações geoespaciais digitais sobrepostas ao ambiente real através destes dispositivos móveis. Ainda, com a execução de testes com usuários verificou-se que os sistemas de RA desenvolvidos trouxeram significativas melhorias para seus usuários na execução das tarefas, assim como ganhos em quesitos como motivação, confiança, diminuição de demandas físicas e mentais, etc. Com relação à preferencia dos usuários, a grande maioria dos usuários entrevistados preferiu trabalhar com estes sistemas de RA desenvolvidos do que apenas com os mapas impressos. Estudos também sobre se o conhecimento prévio do usuário poderia influenciar na execução das tarefas indicaram que, por mais que os sistemas de RA tenham auxiliado a grande maioria dos usuários, os sistemas de RA possuíram maior aceite e trouxeram maiores contribuições aos usuários não especialistas. Também, verificou-se que, por várias vezes, os usuários tendiam a "brincar" mais com o sistema de RA do que com o mapa impresso, passando mais tempo analisando determinada área ou fenômeno geográfico, demonstrando assim maior interação com os dados e espaço geográfico. Em todos os testes executados foi possível verificar que os usuários se mostravam mais interessados em executar suas tarefas quando apoiados pelos sistemas de RA, por vezes executando as tarefas sorrindo ou fazendo exclamações e comentários positivos sobre o sistema de RA. Desta forma, aplicações de Realidade Aumentada, como a descrita neste trabalho, se tornam importantes ferramentas no auxílio às questões profissionais como planejamento e gestão do espaço geográfico, ou até à área da educação, ensino e aprendizagem, uma vez que a integração dos dados em um ambiente mais interativo pode auxiliar na melhoria do processo de entendimento e compreensão do espaço geográfico por parte dos seus usuários. Palavras-chave: Realidade Aumentada, dados geoespaciais, interatividade, teste com usuários.Abstract: In the last decades it is notable the increase of computational systems that help users to perform their tasks. Among the systems that promise to help users to perform their tasks - including map-reading tasks - those based on Augmented Reality (AR) are quite promising. These natural computational interfaces may allow new forms of interaction with the information and the real world around the users. However, just a little has been explored about it in the world - and especially in Brazil. For this reason, we elaborated this research to development and analyze some prototypes of Augmented Reality for Cartography, for interaction with maps and geographic data, addressing and using different methodologies, such as with or without special fiducial markers, for desktop computers or mobile devices. We have used geospatial data referring to physical, political, social, demographic, economic and environmental characteristics, integrated over different types of printed maps, from thematic to topographic maps. It is possible to highlight that the Augmented Reality systems developed in this work brought new opportunities of interaction with printed maps or digital geospatial data. Moreover, the AR systems brought the possibility of exploration of several formats, as vectors and rasters, 2D, 3D, animations, etc. It was verified that - independently of the methodologies used - with AR systems it was possible to interact with a printed physical map and virtual geographic data simultaneously. However, it is possible to affirm that the AR system developed for mobile devices and using the markerless approach is highly attractive. With this approach the AR system can be considered as a "magic lens" or a "window" to the virtual world, where users could access digital geospatial information superimposed on the real environment through these mobile AR systems. Also, with users' tests, it was verified that the AR systems developed have brought significant improvements to the execution of the tasks, such as gains in motivation, confidence, decrease of physical and mental demands, etc. About the user's preference, a large majority of users interviewed preferred to work with these AR systems than just the printed maps. Also, it has been found that sometimes users tended to "play" more with the AR system than with the printed maps, spending more time analyzing a particular area or geographic phenomenon, demonstrating more interaction with geographic data. Also, it has been verified that users were more interested in performing their tasks when supported by the AR systems - performing their tasks smiling or making exclamations and positive comments about the AR system. By this way, applications of Augmented Reality as a described in this work become an important tool for assistance to professional questions like geographic space planning and management, or to the educational area, since the integration of the geospatial data in an interactive environment can help to improve the process of understanding and comprehension of the geographical space by its users. Keywords: Augmented Reality, geospatial data, interactivity, users' tests

Augmented analyses: supporting the study of ubiquitous computing systems

Ubiquitous computing is becoming an increasingly prevalent part of our everyday lives. The reliance of society upon such devices as mobile phones, coupled with the increasing complexity of those devices is an example of how our everyday human-human interaction is affected by this phenomenon. Social scientists studying human-human interaction must now take into account the effects of these technologies not just on the interaction itself, but also on the approach required to study it. User evaluation is a challenging topic in ubiquitous computing. It is generally considered to be difficult, certainly more so than in previous computational settings. Heterogeneity in design, distributed and mobile users, invisible sensing systems and so on, all add up to render traditional methods of observation and evaluation insufficient to construct a complete view of interactional activity. These challenges necessitate the development of new observational technologies. This thesis explores some of those challenges and demonstrates that system logs, with suitable methods of synchronising, filtering and visualising them for use in conjunction with more traditional observational approaches such as video, can be used to overcome many of these issues. Through a review of both the literature of the field, and the state of the art of computer aided qualitative data analysis software (CAQDAS), a series of guidelines are constructed showing what would be required of a software toolkit to meet the challenges of studying ubiquitous computing systems. It outlines the design and implementation of two such software packages, \textit{Replayer} and \textit{Digital Replay System}, which approach the problem from different angles, the former being focussed on visualising and exploring the data in system logs and the latter focussing on supporting the methods used by social scientists to perform qualitative analyses. The thesis shows through case studies how this technique can be applied to add significant value to the qualitative analysis of ubiquitous computing systems: how the coordination of system logs and other media can help us find information in the data that would otherwise be inaccessible; an ability to perform studies in locations/settings that would otherwise be impossible, or at least very difficult; and how creating accessible qualitative data analysis tools allows people to study particular settings or technologies who could not have studied them before. This software aims to demonstrate the direction in which other CAQDAS packages may have to move in order to support the study of the characteristics of human-computer and human-human interaction in a world increasingly reliant upon ubiquitous computing technology

Mobile Pen and Paper Interaction

Although smartphones, tablets and other mobile devices become increasingly popular, pen and paper continue to play an important role in mobile settings, such as note taking or creative discussions. However, information on paper documents remains static and usage practices involving sharing, researching, linking or in any other way digitally processing information on paper are hindered by the gap between the digital and physical worlds. A considerable body of research has leveraged digital pen technology in order to overcome this problem with respect to static settings, however, systematically neglecting the mobile domain. Only recently, several approaches began exploring the mobile domain and developing initial insights into mobile pen-and-paper interaction (mPPI), e.g., to publish digital sketches, [Cowan et al., 2011], link paper and digital artifacts, [Pietrzak et al., 2012] or compose music, [Tsandilas, 2012]. However, applications designed to integrate the most common mobile tools pen, paper and mobile devices, thereby combining the benefits of both worlds in a hybrid mPPI ensemble, are hindered by the lack of supporting infrastructures and limited theoretical understanding of interaction design in the domain. This thesis advances the field by contributing a novel infrastructural approach toward supporting mPPI. It allows applications employing digital pen technology in controlling interactive functionality while preserving mobile characteristics of pen and paper. In addition, it contributes a conceptual framework of user interaction in the domain suiting to serve as basis for novel mPPI toolkits. Such toolkits ease development of mPPI solutions by focusing on expressing interaction rather than designing user interfaces by means of rigid widget sets. As such, they provide the link between infrastructure and interaction in the domain. Lastly, this thesis presents a novel, empirically substantiated theory of interaction in hybrid mPPI ensembles. This theory informs interaction design of mPPI, ultimately allowing to develop compelling and engaging interactive systems employing this modality

Contributions to the science of controlled transformation

writing completed in april 2013My research activities pertain to "Informatics" and in particular "Interactive Graphics" i.e. dynamic graphics on a 2D screen that a user can interact with by means of input devices such as a mouse or a multitouch surface. I have conducted research on Interactive Graphics along three themes: interactive graphics development (how should developers design the architecture of the code corresponding to graphical interactions?), interactive graphic design (what graphical interactions should User Experience (UX) specialists use in their system?) and interactive graphics design process (how should UX specialists design? Which method should they apply?) I invented the MDPC architecture that relies on Picking views and Inverse transforms. This improves the modularity of programs and improves the usability of the specification and the implementation of interactive graphics thanks to the simplification of description. In order to improve the performance of rich-graphic software using this architecture, I explored the concepts of graphical compilers and led a PhD thesis on the topic. The thesis explored the approach and contributed both in terms of description simplification and of software engineering facilitation. Finally, I have applied the simplification of description principles to the problem of shape covering avoidance by relying on new efficient hardware support for parallelized and memory-based algorithms. Together with my colleagues, we have explored the design and assessment of expanding targets, animation and sound, interaction with numerous tangled trajectories, multi-user interaction and tangible interaction. I have identified and defined Structural Interaction, a new interaction paradigm that follows the steps of the direct and instrumental interaction paradigms. I directed a PhD thesis on this topic and together with my student we designed and assessed interaction techniques for structural interaction. I was involved in the design of the "Technology Probes" concept i.e. runnable prototypes to feed the design process. Together with colleagues, I designed VideoProbe, one such Technology Probe. I became interested in more conceptual tools targeted at graphical representation. I led two PhD theses on the topic and explored the characterization of visualization, how to design representations with visual variables or ecological perception and how to design visual interfaces to improve visual scanning. I discovered that those conceptual tools could be applied to programming languages and showed how the representation of code, be it textual or "visual" undergoes visual perception phenomena. This has led me to consider our discipline as the "Science of Controlled Transformations". The fifth chapter is an attempt at providing this new account of "Informatics" based on what users, programmers and researchers actually do with interactive systems. I also describe how my work can be considered as contributing to the science of controlled transformations

Context-based multimodal interpretation : an integrated approach to multimodal fusion and discourse processing

This thesis is concerned with the context-based interpretation of verbal and nonverbal contributions to interactions in multimodal multiparty dialogue systems. On the basis of a detailed analysis of context-dependent multimodal discourse phenomena, a comprehensive context model is developed. This context model supports the resolution of a variety of referring and elliptical expressions as well as the processing and reactive generation of turn-taking signals and the identification of the intended addressee(s) of a contribution. A major goal of this thesis is the development of a generic component for multimodal fusion and discourse processing. Based on the integration of this component into three distinct multimodal dialogue systems, the generic applicability of the approach is shown.Diese Dissertation befasst sich mit der kontextbasierten Interpretation von verbalen und nonverbalen Gesprächsbeiträgen im Rahmen von multimodalen Dialogsystemen. Im Rahmen dieser Arbeit wird, basierend auf einer detaillierten Analyse multimodaler Diskursphänomene, ein umfassendes Modell des Gesprächskontextes erarbeitet. Dieses Modell soll sowohl die Verarbeitung einer Vielzahl von referentiellen und elliptischen Ausdrücken, als auch die Erzeugung reaktiver Aktionen wie sie für den Sprecherwechsel benötigt werden unterstützen. Ein zentrales Ziel dieser Arbeit ist die Entwicklung einer generischen Komponente zur multimodalen Fusion und Diskursverarbeitung. Anhand der Integration dieser Komponente in drei unterschiedliche Dialogsysteme soll der generische Charakter dieser Komponente gezeigt werden

Interaktive Suchprozesse in komplexen Arbeitssituationen - Ein Retrieval Framework

In recent years a steady increase of information produced in organizations can be noticed. In order to stay competitive, companies have a growing interest in reusing existing knowledge from past projects. Furthermore, a complete picture of the available information is necessary to be able to make informed decisions. The variety and complexity of information in modern organizations often exceeds the capabilities of the currently deployed enterprise search solutions. The reasons for that are manifold and range from non-linked information from multiple software systems to missing functionality to support users during search tasks. Existing search engines often do not support the search paradigms necessary in these environments. On many occasions, users are not aware of the results they will find during the formulation of the search queries. Additionally, the aspect of knowledge building and the identification of new insights into the available data is a priority for the users. Therefore, search paradigms are useful to provide users with tools that support exploratory navigation in a data set and help them to recognize relationships between search results. The goal of this publication is the introduction of a framework that supports exploratory searches in an organizational setting. The described LFRP-framework is built on top of four pillars. 1. The multi-layer functionality allows users to formulate complex search queries referring to more than one result type. Therewith, it enables search queries that - starting from a set of relevant projects - allow selections of documents that are linked to these projects. 2. The search paradigm of faceted searching supports users in formulating search queries incrementally by offering dynamic and valid filter criteria that avoid empty result sets. 3. By combining the concept of faceted search with the capability to influence the search result order based on filter criteria, users can define in a fine-grained way which criteria values shall be weighted stronger or weaker in the search results. The interaction with the ranking is conducted transparently by the so-called user preference functions. 4. The last pillar consists of the visualization type of parallel coordinates covering two tasks in the search user interface of the LFRP-Framework. On the one hand, users formulate their search queries solely graphically in the parallel coordinates and on the other hand they obtain a visual representation of the search results and are able to discover relationships between search results and their facets. The framework is introduced formally from a query model point of view as well as a prototypical implementation. It enables users to access large linked data sets by navigation and constitutes a contribution to a comprehensive information strategy for organizations.Seit einigen Jahren ist ein stetiges Ansteigen der Menge an Informationen, die in Unternehmen erzeugt werden, festzustellen. Um als Unternehmen wettbewerbsfähig zu bleiben, ist es notwendig, vorhandenes Wissen wiederzuverwenden, um aus vergangenen Projektergebnissen profitieren zu können. Weiterhin ist ein vollständiges Informationsbild unabdingbar, um informierte Entscheidungen treffen zu können. Die Informationsvielfalt in modernen Unternehmen übersteigt häufig die Fähigkeiten aktuell anzutreffender unternehmensweiter Suchlösungen. Die Gründe hierfür sind vielfältig und reichen von nicht verknüpften Informationen aus verschiedenen Softwaresystemen bis hin zu fehlenden Funktionen, um den Nutzer bei der Suche zu unterstützen. Vorhandene Suchfunktionen im Unternehmen unterstützen häufig nicht die Suchparadigmen, die in diesem Umfeld notwendig sind. Vielfach ist den Suchenden bei der Formulierung ihrer Suchanfrage nicht bekannt, welche Ergebnisse sie finden werden. Stattdessen steht der Aspekt des Wissensaufbaus und der Gewinnung neuer Einsichten in den vorhandenen Daten im Vordergrund. Hierzu werden Suchparadigmen benötigt, die dem Nutzer Werkzeuge zur Verfügung stellen, die ein exploratives Navigieren im Datenbestand erlauben und ihnen bei der Erkennung von Zusammenhängen in den Suchergebnissen unterstützen. Das Ziel dieser Arbeit ist die Vorstellung eines Rahmenwerks, dass explorative Suchvorhaben im Unternehmensumfeld unterstützt. Das beschriebene LFRP-Framework baut auf vier Säulen auf. 1. Die Multi-Layer Funktionalität erlaubt es Nutzern, komplexe Suchanfragen zu formulieren, die sich auf mehr als einen Ergebnistyp beziehen. Dies ermöglicht beispielsweise Suchabfragen, die - ausgehend von einer Menge von relevanten vergangenen Projekten - Selektionen auf den dazugehörigen Dokumenten erlauben. 2. Das Suchparadigma der facettierten Suche unterstützt Nutzer bei der inkrementellen Formulierung von Suchanfragen mithilfe von dynamisch angebotenen Filterkriterien und vermeidet leere Ergebnismengen durch die Bereitstellung gültiger Filterkriterien. 3. Die Erweiterung der facettierten Suche um die Möglichkeit, die Suchergebnisreihenfolge basierend auf Filterkriterien zu beeinflussen, erlaubt es Nutzern feingranular vorzugeben, welche Kriterienausprägungen im Suchergebnis stärker gewichtet werden sollen. Für den Nutzer geschieht die Beeinflussung des Rankings transparent über sogenannte Nutzerpräferenzfunktionen. 4. Die letzte Säule umfasst die Visualisierung der parallelen Koordinaten, die in der Suchoberfläche des LFRP-Frameworks zwei Aufgaben übernimmt. Zum einen formuliert der Nutzer damit die Suchanfrage ausschließlich grafisch über die Visualisierung und zum anderen erhält er eine grafische Repräsentation der Suchergebnisse und kann so leichter Beziehungen zwischen Suchergebnissen und deren Facetten erkennen. Das Framework, welches in dieser Arbeit formal aus Sicht des Anfragemodells sowie als prototypische Umsetzung betrachtet wird, ermöglicht Nutzern den navigierenden Zugriff auf große vernetze Datenbestände und stellt einen Baustein einer umfassenden Informationsstrategie für Unternehmen dar