Exploring human-object interaction through force vector measurement

Thesis: S.M., Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2019Cataloged from PDF version of thesis.Includes bibliographical references (pages 101-107).I introduce SCALE, a project aiming to further understand Human-Object Interaction through the real-time analysis of force vector signals, which I have defined as "Force-based Interaction" in this thesis. Force conveys fundamental information in Force-based Interaction, including force intensity, its direction, and object weight - information otherwise difficult to be accessed or inferred from other sensing modalities. To explore the design space of force-based interaction, I have developed the SCALE toolkit, which is composed of modularized 3d-axis force sensors and application APIs. In collaboration with big industry companies, this system has been applied to a variety of application domains and settings, including a retail store, a smart home and a farmers market. In this thesis, I have proposed a base system SCALE, and two additional advanced projects titled KI/OSK and DepthTouch, which build upon the SCALE project.by Takatoshi Yoshida.S.M.S.M. Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Science

Embedding a Grid of Load Cells into a Dining Table for Automatic Monitoring and Detection of Eating Events

This dissertation describes a “smart dining table” that can detect and measure consumption events. This work is motivated by the growing problem of obesity, which is a global problem and an epidemic in the United States and Europe. Chapter 1 gives a background on the economic burden of obesity and its comorbidities. For the assessment of obesity, we briefly describe the classic dietary assessment tools and discuss their drawback and the necessity of using more objective, accurate, low-cost, and in-situ automatic dietary assessment tools. We explain in short various technologies used for automatic dietary assessment such as acoustic-, motion-, or image-based systems. This is followed by a literature review of prior works related to the detection of weights and locations of objects sitting on a table surface. Finally, we state the novelty of this work. In chapter 2, we describe the construction of a table that uses an embedded grid of load cells to sense the weights and positions of objects. The main challenge is aligning the tops of adjacent load cells to within a few micrometer tolerance, which we accomplish using a novel inversion process during construction. Experimental tests found that object weights distributed across 4 to 16 load cells could be measured with 99.97±0.1% accuracy. Testing the surface for flatness at 58 points showed that we achieved approximately 4.2±0.5 um deviation among adjacent 2x2 grid of tiles. Through empirical measurements we determined that the table has a 40.2 signal-to-noise ratio when detecting the smallest expected intake amount (0.5 g) from a normal meal (approximate total weight is 560 g), indicating that a tiny amount of intake can be detected well above the noise level of the sensors. In chapter 3, we describe a pilot experiment that tests the capability of the table to monitor eating. Eleven human subjects were video recorded for ground truth while eating a meal on the table using a plate, bowl, and cup. To detect consumption events, we describe an algorithm that analyzes the grid of weight measurements in the format of an image. The algorithm segments the image into multiple objects, tracks them over time, and uses a set of rules to detect and measure individual bites of food and drinks of liquid. On average, each meal consisted of 62 consumption events. Event detection accuracy was very high, with an F1-score per subject of 0.91 to 1.0, and an F1 score per container of 0.97 for the plate and bowl, and 0.99 for the cup. The experiment demonstrates that our device is capable of detecting and measuring individual consumption events during a meal. Chapter 4 compares the capability of our new tool to monitor eating against previous works that have also monitored table surfaces. We completed a literature search and identified the three state-of-the-art methods to be used for comparison. The main limitation of all previous methods is that they used only one load cell for monitoring, so only the total surface weight can be analyzed. To simulate their operations, the weights of our grid of load cells were summed up to use the 2D data as 1D. Data were prepared according to the requirements of each method. Four metrics were used to evaluate the comparison: precision, recall, accuracy, and F1-score. Our method scored the highest in recall, accuracy, and F1-score; compared to all other methods, our method scored 13-21% higher for recall, 8-28% higher for accuracy, and 10-18% higher for F1-score. For precision, our method scored 97% that is just 1% lower than the highest precision, which was 98%. In summary, this dissertation describes novel hardware, a pilot experiment, and a comparison against current state-of-the-art tools. We also believe our methods could be used to build a similar surface for other applications besides monitoring consumption

Proceedings of the 3rd IUI Workshop on Interacting with Smart Objects

These are the Proceedings of the 3rd IUI Workshop on Interacting with Smart Objects. Objects that we use in our everyday life are expanding their restricted interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are thus able to capture information, process and store it and interact with their environments, turning them into smart objects

Remote tactile feedback on interactive surfaces

Direct touch input on interactive surfaces has become a predominating standard for the manipulation of digital information in our everyday lives. However, compared to our rich interchange with the physical world, the interaction with touch-based systems is limited in terms of flexibility of input and expressiveness of output. Particularly, the lack of tactile feedback greatly reduces the general usability of a touch-based system and hinders from a productive entanglement of the virtual information with the physical world. This thesis proposes remote tactile feedback as a novel method to provide programmed tactile stimuli supporting direct touch interactions. The overall principle is to spatially decouple the location of touch input (e.g. fingertip or hand) and the location of the tactile sensation on the user's body (e.g. forearm or back). Remote tactile feedback is an alternative concept which avoids particular challenges of existing approaches. Moreover, the principle provides inherent characteristics which can accommodate for the requirements of current and future touch interfaces. To define the design space, the thesis provides a structured overview of current forms of touch surfaces and identifies trends towards non-planar and non-rigid forms with more versatile input mechanisms. Furthermore, a classification highlights limitations of the current methods to generate tactile feedback on touch-based systems. The proposed notion of tactile sensory relocation is a form of sensory substitution. Underlying neurological and psychological principles corroborate the approach. Thus, characteristics of the human sense of touch and principles from sensory substitution help to create a technical and conceptual framework for remote tactile feedback. Three consecutive user studies measure and compare the effects of both direct and remote tactile feedback on the performance and the subjective ratings of the user. Furthermore, the experiments investigate different body locations for the application of tactile stimuli. The results show high subjective preferences for tactile feedback, regardless of its type of application. Additionally, the data reveals no significant differences between the effects of direct and remote stimuli. The results back the feasibility of the approach and provide parameters for the design of stimuli and the effective use of the concept. The main part of the thesis describes the systematical exploration and analysis of the inherent characteristics of remote tactile feedback. Four specific features of the principle are identified: (1) the simplification of the integration of cutaneous stimuli, (2) the transmission of proactive, reactive and detached feedback, (3) the increased expressiveness of tactile sensations and (4) the provision of tactile feedback during multi-touch. In each class, several prototypical remote tactile interfaces are used in evaluations to analyze the concept. For example, the PhantomStation utilizes psychophysical phenomena to reduce the number of single tactile actuators. An evaluation with the prototype compares standard actuator technologies with each other in order to enable simple and scalable implementations. The ThermalTouch prototype creates remote thermal stimuli to reproduce material characteristics on standard touchscreens. The results show a stable rate of virtual object discrimination based on remotely applied temperature profiles. The AutmotiveRTF system is implemented in a vehicle and supports the driver's input on the in-vehicle-infotainment system. A field study with the system focuses on evaluating the effects of proactive and reactive feedback on the user's performance. The main contributions of the dissertation are: First, the thesis introduces the principle of remote tactile feedback and defines a design space for this approach as an alternative method to provide non-visual cues on interactive surfaces. Second, the thesis describes technical examples to rapidly prototype remote tactile feedback systems. Third, these prototypes are deployed in several evaluations which highlight the beneficial subjective and objective effects of the approach. Finally, the thesis presents features and inherent characteristics of remote tactile feedback as a means to support the interaction on today's touchscreens and future interactive surfaces.Die Interaktion mit berührungsempfindlichen Oberflächen ist heute ein Standard für die Manipulation von digitaler Information. Jedoch weist die Bedienung dieser interaktiven Bildschirme starke Einschränkungen hinsichtlich der Flexibilität bei der Eingabe und der Ausdruckskraft der Ausgabe auf, wenn man sie mit den vielfältigen Möglichkeiten des Umgangs mit Objekten in unserer Alltagswelt vergleicht. Besonders die nicht vorhandenen Tastsinnesrückmeldungen vermindern stark die Benutzbarkeit solcher Systeme und verhindern eine effektive Verknüpfung von virtueller Information und physischer Welt. Die vorliegende Dissertation beschreibt den Ansatz der 'distalen taktilen Rückmeldungen' als neuartige Möglichkeit zur Vermittlung programmierter Tastsinnesreize an Benutzer interaktiver Oberflächen. Das Grundprinzip dabei ist die räumliche Trennung zwischen der Eingabe durch Berührung (z.B. mit der Fingerspitze) und dem daraus resultierenden taktilen Reiz am Körper der Benutzer (z.B. am Rücken). Dabei vermeidet das Konzept der distalen taktilen Rückmeldungen einzelne technische und konzeptionelle Nachteile existierender Ansätze. Zusätzlich bringt es Interaktionsmöglichkeiten mit sich, die den Eigenheiten der Interaktion mit aktuellen und auch zukünftigen berührungsempfindlichen Oberflächen Rechnung tragen. Zu Beginn zeigt ein Überblick zu relevanten Arbeiten den aktuellen Forschungstrend hin zu nicht-flachen und verformbaren berührungsempfindlichen Oberflächen sowie zu vielfältigeren Eingabemethoden. Eine Klassifizierung ordnet existierende technische Verfahren zur Erzeugung von künstlichen Tastsinnesreizen und stellt jeweils konzeptuelle und technische Herausforderungen dar. Der in dieser Arbeit vorgeschlagene Ansatz der Verlagerung von Tastsinnesreizen ist eine Form der sensorischen Substitution, zugrunde liegende neurologische und psychologische Prinzipien untermauern das Vorgehen. Die Wirkprinzipien des menschlichen Tastsinnes und die Systeme zur sensorischen Substitution liefern daher konzeptionelle und technische Richtlinien zur Umsetzung der distalen taktilen Rückmeldungen. Drei aufeinander aufbauende Benutzerstudien vergleichen die Auswirkungen von direkten und distalen taktilen Rückmeldungen auf die Leistung und das Verhalten von Benutzern sowie deren subjektive Bewertung der Interaktion. Außerdem werden in den Experimenten die Effekte von Tastsinnesreizen an verschiedenen Körperstellen untersucht. Die Ergebnisse zeigen starke Präferenzen für Tastsinnesrückmeldungen, unabhängig von deren Applikationsort. Die Daten ergeben weiterhin keine signifikanten Unterschiede bei den quantitativen Effekten von direktem und distalen Rückmeldungen. Diese Ergebnisse befürworten die Realisierbarkeit des Ansatzes und zeigen Richtlinien für weitere praktische Umsetzungen auf. Der Hauptteil der Dissertation beschreibt die systematische Untersuchung und Analyse der inhärenten Möglichkeiten, die sich aus der Vermittlung distaler taktiler Rückmeldungen ergeben. Vier verschiedene Charakteristika werden identifiziert: (1) die vereinfachte Integration von Tastsinnesreizen, (2) die Vermittlung von proaktiven, reaktiven und entkoppelten Rückmeldungen, (3) die erhöhte Bandbreite der taktilen Signale und (4) die Darstellung von individuellen Tastsinnesreizen für verschiedene Kontaktpunkte mit der berührungsempfindlichen Oberfläche. Jedes dieser Prinzipien wird durch prototypische Systeme umgesetzt und in Benutzerstudien analysiert. Beispielsweise nutzt das System PhantomStation psychophysikalische Illusionen, um die Anzahl der einzelnen Reizgeber zu reduzieren. In einer Evaluierung des Prototypen werden mehrere Aktuatortechnologien verglichen, um einfache und skalierbare Ansätze zu identifizieren. Der ThermalTouch-Prototyp wird dazu genutzt, distale thermale Reize zu vermitteln, um so Materialeigenschaften auf Berührungsbildschirmen darstellen zu können. Eine Benutzerstudie zeigt, dass sich auf Basis dieser Temperaturverläufe virtuelle Objekte unterscheiden lassen. Das AutomotiveRTF-System wird schließlich in ein Kraftfahrzeug integriert, um den Fahrer bei der Eingabe auf dem Informations- und Unterhaltungssystem zu unterstützen. Eine Feldstudie untersucht die Auswirkungen der proaktiven und reaktiven Rückmeldungen auf die Benutzerleistung. Die vorliegende Dissertation leistet mehrere Beiträge zur Mensch-Maschine-Interaktion: Das Prinzip der distalen taktilen Rückmeldungen wird eingeführt als Alternative zur Erzeugung nicht-visueller Rückmeldungen auf interaktiven Oberflächen. Es werden technische Verfahrensweisen zur prototypischen Implementierung solcher Systeme vorgeschlagen. Diese technischen Prototypen werden in einer Vielzahl verschiedener Benutzerstudien eingesetzt, welche die quantitativen und qualitativen Vorteile des Ansatzes aufzeigen. Schließlich wird gezeigt, wie sich das Prinzip zur Unterstützung heutiger und zukünftiger Interaktionsformen mit berührungsempfindlichen Bildschirmen nutzen lässt

Toolkits for the Development of Hybrid Games: from Tangible Tabletops to Interactive Spaces

Durante los últimos años, los dispositivos tabletop han sido considerados el entorno ideal para los juegos híbridos, los cuales combinan técnicas de juego tradicional, como el uso de objetos físicos para interactuar con el juego de una forma natural, con las nuevas posibilidades que los tabletops ofrecen de aumentar el espacio de juego con imágenes digitales y audio.Sin embargo, los juegos híbridos no se restringen simplemente a tabletops, pudiéndose jugar también en entornos más amplios en los que convergen otros paradigmas de interacción. Por esta razón, el uso de juegos híbridos en Espacios Interactivos está ganando fuerza, pero el número y heterogeneidad de dispositivos y estilos de interacción que se encuentran en estos entornos hace que el diseño y prototipado de juegos sea una tarea difícil. Por lo tanto, el gran reto se encuentra en ofrecer a diseñadores y desarrolladores herramientas apropiadas para la creación de estas aplicaciones.En esta línea de trabajo, el grupo Affective Lab lanzó el proyecto JUGUEMOS (TIN2015-67149-C3-1R), un proyecto nacional centrado en el desarrollo de juegos híbridos en entornos interactivos. Esta Tesis Doctoral se enmarca en este proyecto.El primer paso de la realización de esta tesis fue establecer los dos objetivos principales (Capítulo 1):1) El primer objetivo que se estableció fue profundizar en el uso de tabletops tangibles en terapia con niños con necesidades especiales. Durante los últimos años el grupo Affective Lab había visto la potencialidad de los tabletops tangibles para el trabajo con niños pequeños, pero todavía era necesario llevar a cabo más experiencias y evaluaciones en el ámbito terapéutico, así como explorar si otros grupos de usuarios (adultos con problemas cognitivos) podían beneficiarse de las características de los tabletops.2) El segundo objetivo consistió en diseñar e implementar un toolkit para el desarrollo de juegos híbridos para espacios interactivos. Se decidió que el toolkit estuviera dirigido a desarrolladores para facilitar su trabajo a la hora de crear este tipo de aplicaciones.Una vez establecidos los objetivos, se realizó un estado del arte a su vez dividido en dos partes (Capítulo 2):1) Se realizó una categorización de juegos híbridos para entender y extraer sus principales características, así como los principales retos que surgen al desarrollar este tipo de juegos. También se estudiaron toolkits cuyo objetivo era el desarrollo de juegos híbridos.2) Se estudiaron juegos híbridos desarrollados para niños con necesidades especiales y adultos con problemas cognitivos que hacían uso de la Interacción Tangible y tabletops, así como toolkits dirigidos a terapeutas o educadores para ayudarles en la creación de actividades para sus pacientes.Para llevar a cabo las experiencias y evaluaciones relacionadas con el primer objetivo, se hizo uso del tabletop tangible NIKVision, desarrollado previamente por el grupo Affective Lab, y el toolkit KitVision, una herramienta dirigida a profesionales sin conocimientos de programación para la creación de actividades tangibles y que fue desarrollado durante el Proyecto Final de Carrera de la autora. En el Capítulo 3 de esta Tesis se comenta brevemente el tabletop NIKVision y la arquitectura de KitVision, se describen las evaluaciones que se llevaron a cabo con terapeutas con el objetivo de mejorar y probar la utilidad del toolkit, y se explica una experiencia de un año durante la cual una terapeuta ocupacional de ASAPME, una asociaciónque trabaja con adultos con problemas cognitivos, estuvo usando el tabletop y el toolkit sin supervisión.En el Capítulo 4 se describen diferentes experiencias con KitVision que se llevaron a cabo:- Gracias a una colaboración con la Residencia Romareda, NIKVision y KitVision fueron instalados provisionalmente en la residencia y, tras una evaluación inicial, se desarrollaron tres nuevas actividades para los usuarios de la residencia.- Gracias a la colaboración con ENMOvimienTO y con uno de los centros de Atención Temprana del Instituto Aragonés de Servicios Sociales (IASS), ambos enfocados a trabajar con niños con problemas de aprendizaje, se pudieron realizar evaluaciones que nos permitieron mejorar KitVision y crear nuevas actividades específicamente diseñadas para ellos.- Finalmente, gracias a una colaboración con Atenciona, pudimos evaluar actividades con niños con Trastorno por Déficit de Atención e Hiperactividad (TDAH) y extraer una serie de directrices para diseñar actividades para este tipo de niños. También pudimos llevar a cabo una experiencia de Diseño Participativo con estos niños.El completo desarrollo del toolkit JUGUEMOS, para la creación de juegos híbridos en espacios interactivos, se explica en el Capítulo 5. En este apartado primero se describe el Espacio Interactivo JUGUEMOS que sirvió de base para desarrollar el toolkit. Después se explican con detalle las decisiones de diseño que se tomaron, el modelo de abstracción que se usó para diseñar los juegos, y la arquitectura del toolkit. También se detallan las distintas fases de implementación que se llevaron a cabo, basadas en los tres retos que se extrajeron en el estado del arte: (1) integrar diferentes dispositivos, (2) gestionar salidas gráficas diversas y (3) facilitar la codificación del juego. Finalmente, se presentan dos prototipos de juegos que se desarrollaron durante las dos estancias de investigación que la autora realizó.Finalmente, en el Capítulo 6 se describen los tres casos de uso que se realizaron para tener una primera valoración de la usabilidad del toolkit JUGUEMOS: (1) una evaluación con estudiantes de Máster en la que se implementó un juego completamente funcional para el Espacio Interactivo JUGUEMOS, (2) un juego que fue completamente desarrollado usando el toolkit JUGUEMOS una vez que éste se acabó de implementar, (3) una experiencia que involucró a dos grupos multidisciplinares compuestos por diseñadores y desarrolladores, en la que tuvieron que colaborar para diseñar e implementar dos prototipos de juegos híbridos para el espacio interactivo.<br /

Augmenting Learning Activities with Contextual Information Scent

Students often have information needs while carrying out a multitude of learning activities at universities. When information is needed for investigating a problem, the student may interrupt the work and switch to an information seeking task. As Internet connectivity becomes ubiquitous, searching information has been routinized and integrated in the learning experience. However, information needs are not always fully recognized, or they can not be well articulated. A MOOC student may perceive a video to be difficult, but fails to express what information can be helpful. Sometimes it is improper to interrupt the learning task for searching information, especially when social factors are concerned, e.g. in a seminar talk. These situations create research potentials for making ambient information cues, hereafter referred to as contextual information scent (CIS), available to address students' situational information needs in learning activities. The CIS is designed to combine context-awareness with information seeking, ambient interaction as well as serendipitous encounter. In this thesis, we investigate the CIS mainly in collaborative learning activities. We explore three different contexts: conversation, groupware interaction and video content for MOOC learning. RaindropSearch investigates capturing conversational words as CIS for building search queries, while the TileSearch triggers Web searches based on group discussions and retrieved image and Wikipedia results as CIS for serendipitous interactions. These two explorations both focus on conversation context and provide initial insights into the CIS design practice. Next, we present MeetHub Search, which includes three CIS components based on text interactions in a groupware. Our last prototype, the BOOC Player employs textbook pages as CIS and links them to MOOC videos during the course of collaborative video viewing. All prototypes show how we manipulated design parameters to reduce distraction, increase relevance and ensure timeliness. The studies also exhibit the influence of group dynamics on the use of CIS. We finally extend our research scope to individual MOOC learning and summarize the design insights obtained from MOOC analytics. The contributions of this thesis are summarized as (1) a dedicated research framework derived from both research literature and requirement analysis for recognizing the design challenges, design principles and design space of CIS. The framework lays the foundation for us to explore different contexts in this thesis, where we generated (2) design implications that identify the key attributes of CIS. Last but not least, we employed (3) a variety of evaluation methodologies in this thesis for assessing the usability as well as the benefit and appeal of CIS

Designing Hybrid Interactions through an Understanding of the Affordances of Physical and Digital Technologies

Two recent technological advances have extended the diversity of domains and social contexts of Human-Computer Interaction: the embedding of computing capabilities into physical hand-held objects, and the emergence of large interactive surfaces, such as tabletops and wall boards. Both interactive surfaces and small computational devices usually allow for direct and space-multiplex input, i.e., for the spatial coincidence of physical action and digital output, in multiple points simultaneously. Such a powerful combination opens novel opportunities for the design of what are considered as hybrid interactions in this work. This thesis explores the affordances of physical interaction as resources for interface design of such hybrid interactions. The hybrid systems that are elaborated in this work are envisioned to support specific social and physical contexts, such as collaborative cooking in a domestic kitchen, or collaborative creativity in a design process. In particular, different aspects of physicality characteristic of those specific domains are explored, with the aim of promoting skill transfer across domains. irst, different approaches to the design of space-multiplex, function-specific interfaces are considered and investigated. Such design approaches build on related work on Graspable User Interfaces and extend the design space to direct touch interfaces such as touch-sensitive surfaces, in different sizes and orientations (i.e., tablets, interactive tabletops, and walls). These approaches are instantiated in the design of several experience prototypes: These are evaluated in different settings to assess the contextual implications of integrating aspects of physicality in the design of the interface. Such implications are observed both at the pragmatic level of interaction (i.e., patterns of users' behaviors on first contact with the interface), as well as on user' subjective response. The results indicate that the context of interaction affects the perception of the affordances of the system, and that some qualities of physicality such as the 3D space of manipulation and relative haptic feedback can affect the feeling of engagement and control. Building on these findings, two controlled studies are conducted to observe more systematically the implications of integrating some of the qualities of physical interaction into the design of hybrid ones. The results indicate that, despite the fact that several aspects of physical interaction are mimicked in the interface, the interaction with digital media is quite different and seems to reveal existing mental models and expectations resulting from previous experience with the WIMP paradigm on the desktop PC