Multi-Device Design in Contexts of Interchange and Task Migration

Com a miniaturização dos componentes digitais e o vasto desenvolvimento tecnológico dos últimos anos, a sociedade tem presenciado a redefinição dos "computadores pessoais" pelo advento dos dispositivos móveis. Além da inovação, eles introduziram o desafio do design multi-dispositivo para as aplicações desktop. Enquanto algumas abordagens criaram interfaces móveis sem aproveitar qualquer modelo, outras buscaram adaptações automáticas visando reduzir a sobrecarga de designo Em ambas, o foco do design deixou de ser o usuário, tornando as interfaces tão diferentes ao ponto de comprometerem a usabilidade na realização de uma mesma tarefa em vários dispositivos. Esta tese afirma que não existe uma abordagem de design multi-dispositivo capaz de garantir boa usabilidade em todos os contextos porque o usuário pode escolher apenas uma forma de acesso à aplicação ou alternar seu uso por meio de vários dispositivos. No primeiro caso, o usuário aprende a usar a interface para realizar suas tarefas, sendo relevante uma abordagem que aproveite os recursos do dispositivo e trate suas limitações. No segundo, o usuário já conhece uma das interfaces, o que gera uma expectativa no uso das demais. Logo, é necessário combinar abordagens com objetivos diferentes para atender ao usuário de acordo com o seu contexto de uso. Neste sentido, propõe-se o design multi-dispositivo por meio da preservação de uma hierarquia de prioridades de consistência definida em três níveis. Enquanto os dois primeiros dão suporte à expectativa do usuário em contextos de uso alternado (propensos à execução de tarefas em dispositivos diferentes) e migração de tarefas (iniciando tarefas com um dispositivo e concluindo com outro), o terceiro nível garante a personalização das tarefas de maior interesse visando eficiência e satisfação de uso em um dispositivo específico. A avaliação desta metodologia foi feita por meio de um experimento com três interfaces de pocket PC construídas a partir de uma aplicação desktop do domínio de Educação a Distância: a primeira delas era uma réplica da original (Migração Direta), a segunda não mantinha consistência de layout e era baseada em um processo de design personalizado adequado ao dispositivo (Linear) e a terceira aplicava apenas os dois primeiros níveis da hierarquia de prioridades (Overview). Os resultados da avaliação subjetiva mostraram que a abordagem Overview foi capaz de manter o modelo mental do usuário com maior precisão por preservar os atributos de facilidade, eficiência e segurança de uso na interação inter-dispositivo. Além disso, os resultados medidos para a eficácia (exatidão das respostas) e eficiênciá (tempo médio de execução das tarefas) foram iguais ou melhores com essa abordagem. Por outro lado, os usuários revelaram uma preferência pela personalização de tarefas presente na abordagem Linear. Este resultado dá suporte à proposta desta tese, mostrando que a eficácia gerada pelos dois primeiros níveis da hierarquia de prioridades (percepção e execução das tarefas) deve ser combinada com o terceiro nível de personalização. Para isso, sugere-se a disponibilização de padrões de interface criados pelo designer para escolha do usuário durante a interação. Essa combinação deve garantir usabilidade no acesso a uma aplicação feito sempre por um mesmo dispositivo ou em contextos de uso alternado e migração de tarefasWith the miniaturization of digital components and the vast technological development of the past years, society has remarked the redefinition of "personal computers" by the advent of modern mobile devices. Besides the innovation, these handhelds also introduced the challenge to develop multi-device interfaces for today's desktop applications. While some created mobile interfaces from scratch to get the best from the devices, others looked for automatic adaptations to reduce the load imposed to the designeI. In both cases, the user wasn't the focus anymore, which resulted interfaces so different from each other to the point of compromising usability when peHorming one task on many devices. This thesis claims that there is no multi-device approach capable to provi de full usability in every context because the user may choose only one interface to access the application or interchange its use via many devices. In the first case, the user learns to perform tasks with the given device, which makes relevant an approach that takes advantage of its resources and solves its limitations. In the second, the user already knows one of the available interfaces, which generates an expectation for the others. Therefore, it is necessary to combine approaches with different goals and suit the user according to the appropriate context. In this sense, we propose multi-device design via maintenance of a consistency priorities hierarchy defined in three levels. The first two levels give support to the user's expectation in contexts of interchange (prone to task execution with different devices) and task migration (starting tasks with one device and finishing with other). On the other side, the third level provides task personalization according to the user's interest towards higher efficiency and satisfaction of use with a specific device. The evaluation of this methodology was conducted by an experiment with three pocket PC interfaces designed from an e-learning desktop application: the first interface was an exact replica of the original desktop version (Direct Migration), the second didn't maintain layout consistency and was based in a personalized design process adequate to the device (Linear) while the third applied only the first two levels of the consistency priorities hierarchy (Overview). The subjective evaluation results pointed the Overview approach as the best to maintain the user's mental model by preserving easiness, efficiency and safety of use on inter-device interaction. Additionally, both measured efficacy (task result accuracy) and efficiency (task execution mean time) were the same or even better with this approach. On the other hand, users revealed their preference for the task personalization present in the Linear approach. This result gives support to our proposal, corroborating that the efficacy generated by the first two levels of the consistency priorities hierarchy (task perception and execution) should be combined with the third level of personalization. This could be done by letting designers create interface patterns and make them available to users during interaction. Such combination should guarantee usability while constantly accessing one application through the same device or in contexts of alternated use and task migratio

Generating Graphical User Interfaces for Software Product Lines: A Constraint-based Approach

Due to a high competitive pressure on the global software market, in many areas the software industry is moving from hand crafting to semi-automatic or automatic software construction based on Software Product Lines (SPL). Techniques to automate the construction of software products from SPLs are widely available. These can handle variability in source code artifacts but they are inappropriate to handle variability in Graphical User Interfaces (GUIs). The main reason is that they are not designed to handle such fine grained configurations as they are required to configure GUI frameworks or toolkits. To nevertheless employ them in GUI generation tasks is complex and time consuming. However, in the Human Computer Interaction (HCI) community approaches to develop GUIs in a model-based manner and with constraint-based techniques are worked on that help automate the construction of GUIs. Therefore, the main hypothesis of the proposed research is that constraint-based GUIs techniques are a well suited basis for reducing the customization effort of generated GUIs of SPLs. The paper proposes a research plan to employ these new HCI techniques in generating GUIs for SPLs

Enabling hand-crafted visual markers at scale

As locative media and augmented reality spread into the everyday world so it becomes important to create aesthetic visual markers at scale. We explore a designer-centred approach in which skilled designers handcraft seed designs that are automatically recombined to create many markers as subtle variants of a common theme. First, we extend the d-touch topological approach to creating visual markers that has previously been shown to support creative design with two new techniques: area order codes and visual checksums. We then show how the topological structure of such markers provides the basis for recombining designs to generate many variations. We demonstrate our approach through the creation of beautiful, personalized and interactive wallpaper. We reflect on how technologies must enable designers to balance goals of scalability, aesthetics and reliability in creating beautiful interactive decoration. Copyright is held by the owner/author(s).This research has been supported by the Horizon Digital Economy Research Institute (EPSRC Grant No. EP/G065802/1 and EP/M02315X/1) and the ‘Living With Interactive Decorative Patterns’ project (EPSRC Grant No. EP/L023717/1)

Blending the Material and Digital World for Hybrid Interfaces

The development of digital technologies in the 21st century is progressing continuously and new device classes such as tablets, smartphones or smartwatches are finding their way into our everyday lives. However, this development also poses problems, as these prevailing touch and gestural interfaces often lack tangibility, take little account of haptic qualities and therefore require full attention from their users. Compared to traditional tools and analog interfaces, the human skills to experience and manipulate material in its natural environment and context remain unexploited. To combine the best of both, a key question is how it is possible to blend the material world and digital world to design and realize novel hybrid interfaces in a meaningful way. Research on Tangible User Interfaces (TUIs) investigates the coupling between physical objects and virtual data. In contrast, hybrid interfaces, which specifically aim to digitally enrich analog artifacts of everyday work, have not yet been sufficiently researched and systematically discussed. Therefore, this doctoral thesis rethinks how user interfaces can provide useful digital functionality while maintaining their physical properties and familiar patterns of use in the real world. However, the development of such hybrid interfaces raises overarching research questions about the design: Which kind of physical interfaces are worth exploring? What type of digital enhancement will improve existing interfaces? How can hybrid interfaces retain their physical properties while enabling new digital functions? What are suitable methods to explore different design? And how to support technology-enthusiast users in prototyping? For a systematic investigation, the thesis builds on a design-oriented, exploratory and iterative development process using digital fabrication methods and novel materials. As a main contribution, four specific research projects are presented that apply and discuss different visual and interactive augmentation principles along real-world applications. The applications range from digitally-enhanced paper, interactive cords over visual watch strap extensions to novel prototyping tools for smart garments. While almost all of them integrate visual feedback and haptic input, none of them are built on rigid, rectangular pixel screens or use standard input modalities, as they all aim to reveal new design approaches. The dissertation shows how valuable it can be to rethink familiar, analog applications while thoughtfully extending them digitally. Finally, this thesis’ extensive work of engineering versatile research platforms is accompanied by overarching conceptual work, user evaluations and technical experiments, as well as literature reviews.Die Durchdringung digitaler Technologien im 21. Jahrhundert schreitet stetig voran und neue Geräteklassen wie Tablets, Smartphones oder Smartwatches erobern unseren Alltag. Diese Entwicklung birgt aber auch Probleme, denn die vorherrschenden berührungsempfindlichen Oberflächen berücksichtigen kaum haptische Qualitäten und erfordern daher die volle Aufmerksamkeit ihrer Nutzer:innen. Im Vergleich zu traditionellen Werkzeugen und analogen Schnittstellen bleiben die menschlichen Fähigkeiten ungenutzt, die Umwelt mit allen Sinnen zu begreifen und wahrzunehmen. Um das Beste aus beiden Welten zu vereinen, stellt sich daher die Frage, wie neuartige hybride Schnittstellen sinnvoll gestaltet und realisiert werden können, um die materielle und die digitale Welt zu verschmelzen. In der Forschung zu Tangible User Interfaces (TUIs) wird die Verbindung zwischen physischen Objekten und virtuellen Daten untersucht. Noch nicht ausreichend erforscht wurden hingegen hybride Schnittstellen, die speziell darauf abzielen, physische Gegenstände des Alltags digital zu erweitern und anhand geeigneter Designparameter und Entwurfsräume systematisch zu untersuchen. In dieser Dissertation wird daher untersucht, wie Materialität und Digitalität nahtlos ineinander übergehen können. Es soll erforscht werden, wie künftige Benutzungsschnittstellen nützliche digitale Funktionen bereitstellen können, ohne ihre physischen Eigenschaften und vertrauten Nutzungsmuster in der realen Welt zu verlieren. Die Entwicklung solcher hybriden Ansätze wirft jedoch übergreifende Forschungsfragen zum Design auf: Welche Arten von physischen Schnittstellen sind es wert, betrachtet zu werden? Welche Art von digitaler Erweiterung verbessert das Bestehende? Wie können hybride Konzepte ihre physischen Eigenschaften beibehalten und gleichzeitig neue digitale Funktionen ermöglichen? Was sind geeignete Methoden, um verschiedene Designs zu erforschen? Wie kann man Technologiebegeisterte bei der Erstellung von Prototypen unterstützen? Für eine systematische Untersuchung stützt sich die Arbeit auf einen designorientierten, explorativen und iterativen Entwicklungsprozess unter Verwendung digitaler Fabrikationsmethoden und neuartiger Materialien. Im Hauptteil werden vier Forschungsprojekte vorgestellt, die verschiedene visuelle und interaktive Prinzipien entlang realer Anwendungen diskutieren. Die Szenarien reichen von digital angereichertem Papier, interaktiven Kordeln über visuelle Erweiterungen von Uhrarmbändern bis hin zu neuartigen Prototyping-Tools für intelligente Kleidungsstücke. Um neue Designansätze aufzuzeigen, integrieren nahezu alle visuelles Feedback und haptische Eingaben, um Alternativen zu Standard-Eingabemodalitäten auf starren Pixelbildschirmen zu schaffen. Die Dissertation hat gezeigt, wie wertvoll es sein kann, bekannte, analoge Anwendungen zu überdenken und sie dabei gleichzeitig mit Bedacht digital zu erweitern. Dabei umfasst die vorliegende Arbeit sowohl realisierte technische Forschungsplattformen als auch übergreifende konzeptionelle Arbeiten, Nutzerstudien und technische Experimente sowie die Analyse existierender Forschungsarbeiten

Head Up Games : on the design, creation and evaluation of interactive outdoor games for children

This thesis proposes a new genre of outdoor games for children, namely Head Up Games. The concept was inspired by the observation that existing pervasive outdoor games for children were mostly played head down, as the predominantly screen-based interaction of existing games required constant attention of the children. First, the vision of Head Up Games is described and illustrated with several design cases (Chapter 2). In contrast to the head down games, Head Up Games aim to encourage and support rich social interaction and physical activity, play behaviors that are similar to play behaviors seen in traditional outdoor games (such as tag and hide-and-seek). The design process of Head Up Games poses several challenges. In User Centered Design it is commonly accepted to start the development of a new product using low-fi mock-ups, e.g., paper prototypes, and evaluate these with end-users. In the case of Head Up Games this proved to be difficult, as the emerging game experience is significantly altered when using paper prototypes. Therefore, a study was carried out that used high-fi prototypes, i.e. working, interactive, prototypes, from a very early stage in the design process (Chapter 3). This way, the effect of interactions on the game experience can be addressed earlier and better in the design process. Furthemore, having access to technology early in the design process, allows designers to better explore the design space. However, designers often do not possess adequate skills to quickly prototype interactive products, particularly products that need to be evaluated in an outdoor context. Such a development is often costly and time-consuming. Therefore, the RaPIDO platform was developed (Chapter 4). The platform not only includes the appropriate hardware for creating outdoor games, but is also bundled with software libraries, to allow designers not specifically trained in software engineering to adopt the platform easily. RaPIDO was evaluated using a case study methodology with two Industrial Design master students. The evaluation not only focused on the usability of the platform, but, more importantly, how the use of the platform affected the design process. The main conclusion of the study was that the designers indeed were able to rapidly create mobile games, and that the hardware offered was suitable for creating outdoor games. Furthermore, issues were identified with regard to writing the game software, e.g., managing the complexity of the software. Finally, for evaluating Head Up Games with children two methods were applied: the Outdoor Play Observation Scheme (OPOS) was used to quantify the intended play behavior. Furthermore, GroupSorter was developed to provide a framework to interview a group of children simultaneously, resulting in qualitative comments. Both OPOS and GroupSorter were applied for evaluating three Head Up Games, which are described in Chapter 5

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

The Continuum Architecture: Towards Enabling Chaotic Ubiquitous Computing

Interactions in the style of the ubiquitous computing paradigm are possible today, but only in handcrafted environments within one administrative and technological realm. This thesis describes an architecture (called Continuum), a design that realises the architecture, and a proof-of-concept implementation that brings ubiquitous computing to chaotic environments. Essentially, Continuum enables an ecology at the edge of the network, between users, competing service providers from overlapping administrative domains, competing internet service providers, content providers, and software developers that want to add value to the user experience. Continuum makes the ubiquitous computing functionality orthogonal to other application logic. Existing web applications are augmented for ubiquitous computing with functionality that is dynamically compiled and injected by a middleware proxy into the web pages requested by a web browser at the user?s mobile device. This enables adaptability to environment variability, manageability without user involvement, and expansibility without changes to the mobile. The middleware manipulates self-contained software units with precise functionality (called frames), which help the user interact with contextual services in conjunction with the data to which they are attached. The middleware and frame design explicitly incorporates the possibility of discrepancies between the assumptions of ubiquitous-computing software developers and field realities: multiple administrative domains, unavailable service, unavailable software, and missing contextual information. A framework for discovery and authorisation addresses the chaos inherent to the paradigm through the notion of role assertions acquired dynamically by the user. Each assertion represents service access credentials and contains bootstrapping points for service discovery on behalf of the holding user. A proof-of-concept prototype validates the design, and implements several frames that demonstrate general functionality, including driving discovery queries over multiple service discovery protocols and making equivalences between service types, across discovery protocols

Multi-modal post-editing of machine translation

As MT quality continues to improve, more and more translators switch from traditional translation from scratch to PE of MT output, which has been shown to save time and reduce errors. Instead of mainly generating text, translators are now asked to correct errors within otherwise helpful translation proposals, where repetitive MT errors make the process tiresome, while hard-to-spot errors make PE a cognitively demanding activity. Our contribution is three-fold: first, we explore whether interaction modalities other than mouse and keyboard could well support PE by creating and testing the MMPE translation environment. MMPE allows translators to cross out or hand-write text, drag and drop words for reordering, use spoken commands or hand gestures to manipulate text, or to combine any of these input modalities. Second, our interviews revealed that translators see value in automatically receiving additional translation support when a high CL is detected during PE. We therefore developed a sensor framework using a wide range of physiological and behavioral data to estimate perceived CL and tested it in three studies, showing that multi-modal, eye, heart, and skin measures can be used to make translation environments cognition-aware. Third, we present two multi-encoder Transformer architectures for APE and discuss how these can adapt MT output to a domain and thereby avoid correcting repetitive MT errors.Angesichts der stetig steigenden Qualität maschineller Übersetzungssysteme (MÜ) post-editieren (PE) immer mehr Übersetzer die MÜ-Ausgabe, was im Vergleich zur herkömmlichen Übersetzung Zeit spart und Fehler reduziert. Anstatt primär Text zu generieren, müssen Übersetzer nun Fehler in ansonsten hilfreichen Übersetzungsvorschlägen korrigieren. Dennoch bleibt die Arbeit durch wiederkehrende MÜ-Fehler mühsam und schwer zu erkennende Fehler fordern die Übersetzer kognitiv. Wir tragen auf drei Ebenen zur Verbesserung des PE bei: Erstens untersuchen wir, ob andere Interaktionsmodalitäten als Maus und Tastatur das PE unterstützen können, indem wir die Übersetzungsumgebung MMPE entwickeln und testen. MMPE ermöglicht es, Text handschriftlich, per Sprache oder über Handgesten zu verändern, Wörter per Drag & Drop neu anzuordnen oder all diese Eingabemodalitäten zu kombinieren. Zweitens stellen wir ein Sensor-Framework vor, das eine Vielzahl physiologischer und verhaltensbezogener Messwerte verwendet, um die kognitive Last (KL) abzuschätzen. In drei Studien konnten wir zeigen, dass multimodale Messung von Augen-, Herz- und Hautmerkmalen verwendet werden kann, um Übersetzungsumgebungen an die KL der Übersetzer anzupassen. Drittens stellen wir zwei Multi-Encoder-Transformer-Architekturen für das automatische Post-Editieren (APE) vor und erörtern, wie diese die MÜ-Ausgabe an eine Domäne anpassen und dadurch die Korrektur von sich wiederholenden MÜ-Fehlern vermeiden können.Deutsche Forschungsgemeinschaft (DFG), Projekt MMP