Gesture Interaction at a Distance

The aim of this work is to explore, from a perspective of human behavior, which\ud gestures are suited to control large display surfaces from a short distance away; why that is so; and, equally important, how such an interface can be made a reality. A well-known example of the type of interface that is the focus in this thesis is portrayed in the science fiction movie ‘Minority Report’. The lead character of this movie uses hand gestures such as pointing, picking-up and throwing-away to interact with a wall-sized display in a believable way. Believable, because the gestures are familiar from everyday life and because the interface responds predictably. Although only fictional in this movie, such gesture-based interfaces can, when realized, be applied in any environment that is equipped with large display surfaces. For example, in a laboratory for analyzing and interpreting large data sets; in interactive shopping windows to casually browse a product list; and in the operating room to easily access a patient’s MRI scans. The common denominator is that the user cannot or may not touch the display: the interaction occurs at arms-length and larger distances

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

The design of fashionable wearables to promote sustainable behaviours in smart cities

A falta de sustentabilidade que ameaça o nosso futuro necessita de atenção urgente, exigindo uma abordagem holística. Contudo, ações explicitamente destinadas a mudar o comportamento humano não são muito comuns quando, paradoxalmente, grande parte do impacto ambiental está associado ao comportamento das pessoas na sua vida quotidiana. Muitas ações insustentáveis estão atualmente a ter lugar nas cidades, causando impactos negativos. As cidades oferecem respostas a muitas necessidades da vida, o que motiva as pessoas a procurarem estes espaços, causando uma concentração populacional. Esta procura do meio urbano decorre da expectativa de ter melhor qualidade de vida. A mobilidade urbana, por seu lado, que pode ser vista como um facilitador do acesso a múltiplas componentes da vida quotidiana, incluindo empregos, educação, cuidados de saúde e muitos outros pode, também, gerar congestionamentos, poluição, um acrescido consumo energético e aumento de stress, afetando diretamente a qualidade de vida das pessoas. Neste contexto, as escolhas pessoais quanto à mobilidade têm um impacto substancial na vida quotidiana. Contudo, dado que as cidades estão a tornar-se “inteligentes”, graças à aplicação de tecnologias emergentes de identificação, deteção e comunicação, nomeadamente com recurso à tecnologia da Internet das Coisas (IoT – Internet of Things), abrem-se novas possibilidade para alteração dos comportamentos da vida quotidiana, com impacto na sustentabilidade. Este desenvolvimento tecnológico tem vindo a ser explorado em múltiplas áreas, incluindo energia / instalações, transportes, cuidados de saúde, segurança, monitorização doméstica, entre outras, sendo reconhecido como uma tendência com forte crescimento a curto prazo. Neste contexto, este estudo pretendeu explorar a possibilidade de promover comportamentos de mobilidade considerados mais sustentáveis através do design de artigos de moda que possam funcionar como elementos ativos no sistema de uma cidade “inteligente”. A possibilidade de combinação de funcionalidade e sedução, o poder motivador da moda, combinado com tecnologia avançada e sensores, foi entendido como tendo fortes argumentos para ser bem-sucedido na modificação do comportamento. Como ocorre com muitos outros produtos, os produtos da moda são pensados para responder às expetativas, necessidades e limitações das pessoas, onde convivem aspetos funcionais, ergonómicos, afetivos e emocionais, fortemente focados na experiência e no envolvimento, mas com assumido enfoque na componente estética, estilo, estatuto e formação de identidade. Estes são parte essencial da mudança de comportamento. As tecnologias digitais embutidas nos têxteis oferecem oportunidades significativas de alargamento das funcionalidades dos produtos de moda, incluindo a comunicação, transformação da informação do nosso corpo em dados, conduzindo energia, brilhando, crescendo e muitos outros. Alguns bons exemplos deste tipo podem já ser encontrados na promoção da segurança humana, saúde/ bem-estar, treino militar e entretenimento. Nesta investigação, o termo “fashionable wearables” é adotado para enfatizar a combinação de moda e tecnologia. Importa ressalvar que o objetivo deste estudo não foi trabalhar questões de moda sustentável, que é outra questão vital, mas que fica fora do recorte deste trabalho. O principal objetivo foi explorar como e em que medida as soluções de “fashionable wearables” podem promover comportamentos mais sustentáveis em cidades inteligentes. As escolhas de mobilidade foram selecionadas como comportamentos-alvo que se pretendem alterar. Para o conseguir, foi utilizada uma metodologia mista, de acordo com a abordagem do Design Centrado no Humano. A abordagem do Design para a Mudança de Comportamento foi utilizada para fornecer soluções a um nível do design conceptual. Este estudo de doutoramento foi pensado como uma investigação pelo design, no qual o design conceptual desempenhou um papel formativo na geração do conhecimento. Utilizámos dados qualitativos e quantitativos para explorar o contexto, identificar os tópicos, gerar ideias e soluções e verificar conceitos para cumprir objetivos. A metodologia proposta foi dividida em cinco fases: (1) Compreender, (2) Descobrir, (3) Design Conceptual, (4) Protótipo, e (5) Avaliação. Na primeira fase, aprofundámos os conhecimentos teóricos sobre temas de investigação e fizemos revisões de literatura. Compreendemos a noção atual de sustentabilidade e do comportamento sustentável com entrevistas a peritos e métodos de mapeamento de conceitos. Identificámos comportamentos sustentáveis e exemplificámo-los considerando domínios da vida quotidiana e tipos de comportamento. Determinámos os comportamentos que têm um impacto mais negativo na vida quotidiana. No final, decidimos concentrar-nos nos comportamentos de mobilidade. Na segunda fase, explorámos os obstáculos à adoção e/ou manutenção de comportamentos sustentáveis. Conduzimos sessões de grupos focais para descobrir necessidades dos utilizadores, exigências, problemas e desculpas razoáveis para não adotar ações de mobilidade sustentável e níveis de preferência/satisfação dos modos de mobilidade. Depois de traçarmos as oportunidades potenciais para um maior desenvolvimento, identificámos os comportamentos sustentáveis alvo, em que nos iriamos focar, em particular o caminhar e partilhar bicicletas, na vida da cidade. Na terceira fase, identificámos utilizadores-alvo, desenvolvemos personas e um cenário narrativo representando potenciais perfis de utilizadores e problemas. Ilustrámos storyboards, gerámos ideias. Apesar de o estudo se rever no paradigma sócio-critico, as soluções conceptuais geradas foram baseadas em Estratégias de Mudança de Comportamento e problemas observados em potenciais utilizadores, bem como em oportunidades futuras recolhidas a partir de relatórios de previsão e não apenas nas possibilidades tecnológicas atuais. Realizámos entrevistas de avaliação por peritos para alimentar a ideação, fazer a avaliação e refinamento das propostas, com a participação de especialistas em diferentes campos do saber, incluindo design de interfaces, produto, têxtil, moda, interação e ciência, incluindo engenharia, computador, física, software. Na quarta fase, visualizámos ideias conceptuais de produtos e sistemas com “storyboarding” de soluções. Desenvolvemos “mock-ups” e um protótipo em vídeo do vestuário de moda, para permitir a avaliação da experiência de utilização antecipada. O protótipo de vídeo forjou o que seriam as futuras ofertas de tecnologia inteligente em termos de interface e exibição do e-textile com animação por computador e ferramentas de edição de vídeo. Demos aos potenciais utilizadores a impressão de que estavam a interagir com um sistema real antes deste existir, o que proporcionou a tangibilização possível da solução para a fase de avaliação. Na última fase, avaliámos o desenho conceptual dum “fashionable wearable” para testar a motivação do utilizador em aderir ao comportamento desejável e a usabilidade do artefacto. Utilizámos o Questionário de Experiência de Utilização (UEQ), avaliando a sua experiência considerando a perspicuidade, eficiência, estímulo, atratividade, novidade e fiabilidade do design conceptual. Obtivemos também feedback e sugestões dos utilizadores. O conceito foi avaliado positivamente. Encontrámos um estímulo significativo para a mudança de comportamento, que sugere que os artigos de moda, se usados como parte de uma estratégia de mudança de comportamento, podem influenciar a adoção de comportamentos mais sustentáveis no contexto da mobilidade urbana em cidades “inteligentes”. Em última análise, esta investigação de doutoramento contribuiu para o campo do design de moda, design para comportamento sustentável, design para mudança de comportamento e sustentabilidade.The lack of sustainability that threatens our future needs urgent attention, requiring a holistic approach. However, actions explicitly aimed at changing human behaviour are not very common when, paradoxically, much of the environmental impact is associated with the behaviour of users in everyday life. Many unsustainable actions are currently taking place in cities, causing negative impacts. Cities offer access to many requirements of life that causes a population shift. Significantly, mobility facilitates access to all necessities of life in urban areas. Considering the potential to generate congestion, pollution and preventing freedom of accessibility, mobility can directly affect the quality of air and quality of life. Since cities are becoming smart thanks to emerging technologies, smart mobility has growing in importance and concerns more about being effective and sustainable. In this matter, mobility choices have a substantial impact on daily life. This study intended to promote mobility behaviours considered more sustainable through the design of fashionable wearables that can work as active elements of the system in the smart city context. The combination of body-related functionality and seduction, as well as the motivating power of fashion, combined with advanced technology and sensors, provided a strong motivation for contributing to behaviour change. The main objective was to explore how and to what extent solutions of fashionable wearables can promote more sustainable behaviours in smart cities. Mobility choices were selected as target behaviours that are desired to be altered. To achieve this, a mixed methodology according to the Human-Centred Design approach was used to gain information about needs and demands. Design for Behaviour Change strategies were used to provide solutions at a conceptual design level. We evaluated the conceptual design with a key element of fashionable wearable to test the motivation of the user to use and adhere the usability of the design to determine behaviour intervention. The conceptual design was evaluated positively, and we found the meaningful stimulus for the behaviour change

Mobile Augmented Reality Application for Monitoring Industrial Systems

Augmented Reality (AR) applications have constantly evolved during the past few years in the mobile technology field. The potential of Mobile Augmented Reality (MAR) permits to enrich the reality with digital information by integrating the physical context of the user, localization-based data, mobile embedded sensors and Internet connectivity makes out of MAR a promising technology to be used in a wide range of real use cases. Nowadays the AR communities are growing and more companies are investing in Virtual Reality (VR) and MAR technologies. Lots of applications have been developed for medical, military, entertainment, manufacturing and industrial environments. Besides this, with the exponential grow up of the mobile market, this type of applications are becoming closer to final users. This work not only details the fundaments and concepts of AR but also describes the design and implementation of a MAR application within the area of industrial systems, specifically for the domain of discrete manufacturing. The application is designed to be used in mobile devices and special attention was taken, by using the latest technological trends in this area. The developed MAR application is intended to be used by people working with Manufacturing Systems (MS), allowing an intuitive and better visualization of huge amounts of data which are generated in the industrial system. An industrial testbed was used for testing this diploma work. Several tracking patterns were deployed along the production line to create points where the user can experience an AR interaction. The MAR application displays 3D objects in front of the user view by tracking the AR markers using di erent visualization gadgets, mobile devices or smart wearable devices (AR glasses). The developed application is designed based on the requirements and specifications of a project named: Pro-active decision support for data-intensive environments (ASTUTE), supported by the European Union (EU). The final results of the implemented application is included as a part of ASTUTE, in the demonstrator of virtual control room for building and manufacturing process management

Augmented reality device for first response scenarios

A prototype of a wearable computer system is proposed and implemented using commercial off-shelf components. The system is designed to allow the user to access location-specific information about an environment, and to provide capability for user tracking. Areas of applicability include primarily first response scenarios, with possible applications in maintenance or construction of buildings and other structures. Necessary preparation of the target environment prior to system\u27s deployment is limited to noninvasive labeling using optical fiducial markers. The system relies on computational vision methods for registration of labels and user position. With the system the user has access to on-demand information relevant to a particular real-world location. Team collaboration is assisted by user tracking and real-time visualizations of team member positions within the environment. The user interface and display methods are inspired by Augmented Reality1 (AR) techniques, incorporating a video-see-through Head Mounted Display (HMD) and fingerbending sensor glove.*. 1Augmented reality (AR) is a field of computer research which deals with the combination of real world and computer generated data. At present, most AR research is concerned with the use of live video imagery which is digitally processed and augmented by the addition of computer generated graphics. Advanced research includes the use of motion tracking data, fiducial marker recognition using machine vision, and the construction of controlled environments containing any number of sensors and actuators. (Source: Wikipedia) *This dissertation is a compound document (contains both a paper copy and a CD as part of the dissertation). The CD requires the following system requirements: Adobe Acrobat; Microsoft Office; Windows MediaPlayer or RealPlayer

Gesture based persuasive interfaces for public ambient displays

Dissertação de Mestrado em Engenharia Informática 2º Semestre, 2011/2012This Master thesis studies how Public Ambient Displays (PAD) can be used as a tool to achieve behaviour change, through persuasive technology. In order to reach the goals of the thesis, an interactive public ambient display system called Motion-based Ambient Interactive Display (MAID) was developed. MAID is driven to motivate behaviour changes regarding domestic energy consumption, through a persuasive game interface based on gesture recognition technology. The developed prototype guides players through the different rooms of a house, where they have to find out what is wrong and practice the correct actions to save energy, using similar gestures to the ones they would use in real life to achieve the same goals. The system provides feedback regarding the consequences of each action, in order to make users aware of the consequences of their actions. The implementation of MAID is based on a purpose built, highly configurable and modular framework. It allows the administrator to fine tune and tweak the application to the necessities of the setup location constraints, by adjusting basic display properties, change image content or even modify the scripted gameplay itself. The scripted game system is flexible enough to allow the repurposing of the framework, beyond the previously defined theme, for future studies. The MAID was subjected to user testing, in order to show that it is possible to create a persuasive PAD interface, using seamless interaction methods, with the currently available technology, and use it to spread awareness of a cause, leading to behaviour change.Fundação para a Ciência e Tecnologia - project DEAP (PTDC/AAC-AMB/104834/2008); CITI/DI/FCT/UNL (PEst-OE/EEI/UI0527/201

Inferring Complex Activities for Context-aware Systems within Smart Environments

The rising ageing population worldwide and the prevalence of age-related conditions such as physical fragility, mental impairments and chronic diseases have significantly impacted the quality of life and caused a shortage of health and care services. Over-stretched healthcare providers are leading to a paradigm shift in public healthcare provisioning. Thus, Ambient Assisted Living (AAL) using Smart Homes (SH) technologies has been rigorously investigated to help address the aforementioned problems. Human Activity Recognition (HAR) is a critical component in AAL systems which enables applications such as just-in-time assistance, behaviour analysis, anomalies detection and emergency notifications. This thesis is aimed at investigating challenges faced in accurately recognising Activities of Daily Living (ADLs) performed by single or multiple inhabitants within smart environments. Specifically, this thesis explores five complementary research challenges in HAR. The first study contributes to knowledge by developing a semantic-enabled data segmentation approach with user-preferences. The second study takes the segmented set of sensor data to investigate and recognise human ADLs at multi-granular action level; coarse- and fine-grained action level. At the coarse-grained actions level, semantic relationships between the sensor, object and ADLs are deduced, whereas, at fine-grained action level, object usage at the satisfactory threshold with the evidence fused from multimodal sensor data is leveraged to verify the intended actions. Moreover, due to imprecise/vague interpretations of multimodal sensors and data fusion challenges, fuzzy set theory and fuzzy web ontology language (fuzzy-OWL) are leveraged. The third study focuses on incorporating uncertainties caused in HAR due to factors such as technological failure, object malfunction, and human errors. Hence, existing studies uncertainty theories and approaches are analysed and based on the findings, probabilistic ontology (PR-OWL) based HAR approach is proposed. The fourth study extends the first three studies to distinguish activities conducted by more than one inhabitant in a shared smart environment with the use of discriminative sensor-based techniques and time-series pattern analysis. The final study investigates in a suitable system architecture with a real-time smart environment tailored to AAL system and proposes microservices architecture with sensor-based off-the-shelf and bespoke sensing methods. The initial semantic-enabled data segmentation study was evaluated with 100% and 97.8% accuracy to segment sensor events under single and mixed activities scenarios. However, the average classification time taken to segment each sensor events have suffered from 3971ms and 62183ms for single and mixed activities scenarios, respectively. The second study to detect fine-grained-level user actions was evaluated with 30 and 153 fuzzy rules to detect two fine-grained movements with a pre-collected dataset from the real-time smart environment. The result of the second study indicate good average accuracy of 83.33% and 100% but with the high average duration of 24648ms and 105318ms, and posing further challenges for the scalability of fusion rule creations. The third study was evaluated by incorporating PR-OWL ontology with ADL ontologies and Semantic-Sensor-Network (SSN) ontology to define four types of uncertainties presented in the kitchen-based activity. The fourth study illustrated a case study to extended single-user AR to multi-user AR by combining RFID tags and fingerprint sensors discriminative sensors to identify and associate user actions with the aid of time-series analysis. The last study responds to the computations and performance requirements for the four studies by analysing and proposing microservices-based system architecture for AAL system. A future research investigation towards adopting fog/edge computing paradigms from cloud computing is discussed for higher availability, reduced network traffic/energy, cost, and creating a decentralised system. As a result of the five studies, this thesis develops a knowledge-driven framework to estimate and recognise multi-user activities at fine-grained level user actions. This framework integrates three complementary ontologies to conceptualise factual, fuzzy and uncertainties in the environment/ADLs, time-series analysis and discriminative sensing environment. Moreover, a distributed software architecture, multimodal sensor-based hardware prototypes, and other supportive utility tools such as simulator and synthetic ADL data generator for the experimentation were developed to support the evaluation of the proposed approaches. The distributed system is platform-independent and currently supported by an Android mobile application and web-browser based client interfaces for retrieving information such as live sensor events and HAR results

Human-Computer Interaction

In this book the reader will find a collection of 31 papers presenting different facets of Human Computer Interaction, the result of research projects and experiments as well as new approaches to design user interfaces. The book is organized according to the following main topics in a sequential order: new interaction paradigms, multimodality, usability studies on several interaction mechanisms, human factors, universal design and development methodologies and tools