Computational Methods for Measurement of Visual Attention from Videos towards Large-Scale Behavioral Analysis

Visual attention is one of the most important aspects of human social behavior, visual navigation, and interaction with the world, revealing information about their social, cognitive, and affective states. Although monitor-based and wearable eye trackers are widely available, they are not sufficient to support the large-scale collection of naturalistic gaze data in face-to-face social interactions or during interactions with 3D environments. Wearable eye trackers are burdensome to participants and bring issues of calibration, compliance, cost, and battery life. The ability to automatically measure attention from ordinary videos would deliver scalable, dense, and objective measurements to use in practice. This thesis investigates several computational methods to measure visual attention from videos using computer vision and its use for quantifying visual social cues such as eye contact and joint attention. Specifically, three methods are investigated. First, I present methods for detection of looks to camera in first-person view and its use for eye contact detection. Experimental results show that the presented method can achieve the first human expert-level detection of eye contact. Second, I develop a method for tracking heads in a 3d space for measuring attentional shifts. Lastly, I propose spatiotemporal deep neural networks for detecting time-varying attention targets in video and present its application for the detection of shared attention and joint attention. The method achieves state-of-the-art results on different benchmark datasets on attention measurement as well as the first empirical result on clinically-relevant gaze shift classification. Presented approaches have the benefit of linking gaze estimation to the broader tasks of action recognition and dynamic visual scene understanding, and bears potential as a useful tool for understanding attention in various contexts such as human social interactions, skill assessments, and human-robot interactions.Ph.D

Supporting Collaborative Learning in Computer-Enhanced Environments

As computers have expanded into almost every aspect of our lives, the ever-present graphical user interface (GUI) has begun facing its limitations. Demanding its own share of attention, GUIs move some of the users\u27 focus away from the task, particularly when the task is 3D in nature or requires collaboration. Researchers are therefore exploring other means of human-computer interaction. Individually, some of these new techniques show promise, but it is the combination of multiple approaches into larger systems that will allow us to more fully replicate our natural behavior within a computing environment. As computers become more capable of understanding our varied natural behavior (speech, gesture, etc.), the less we need to adjust our behavior to conform to computers\u27 requirements. Such capabilities are particularly useful where children are involved, and make using computers in education all the more appealing. Herein are described two approaches and implementations of educational computer systems that work not by user manipulation of virtual objects, but rather, by user manipulation of physical objects within their environment. These systems demonstrate how new technologies can promote collaborative learning among students, thereby enhancing both the students\u27 knowledge and their ability to work together to achieve even greater learning. With these systems, the horizon of computer-facilitated collaborative learning has been expanded. Included among this expansion is identification of issues for general and special education students, and applications in a variety of domains, which have been suggested

Proceedings of the 9th international conference on disability, virtual reality and associated technologies (ICDVRAT 2012)

The proceedings of the conferenc

Kaasaegsete kognitiivsete ja sotsiaalsete sekkumistehnikate loomine pediaatrilises neurorehabilitatsioonis ajukahjustusega lastel

Väitekirja elektrooniline versioon ei sisalda publikatsiooneOmandatud ajukahjustusega lapsed vajavad kaasuva kognitiivse ja sotsiaalse defitsiidi diagnostikat ja rehabilitatsiooni, mis on oluline lapse akadeemilise edukuse ja elukvaliteedi parandamisel. Neurorehabilitatsioon on plaanipärane sekkumine, mille eesmärk on kompenseerida või kergendada ajukahjustusest põhjustatud defitsiiti. Antud doktoritöö eesmärk oli uute arvutipõhiste meetodite kasutuselevõtmine omandatud ajukahjustusega laste kognitiivses ja sotsiaalses neurorehabilitatioonis. Treeningdisainid loodi tähelepanu, ruumitaju ja sotsiaalse kompetentsi häirete raviks. Uuringus osales 59 epilepsia, ajutrauma või tikkide diagnoosiga ja 47 tervet kontrollgrupi last vanuses 8–12 aastat. Patsiendid läbisid rehabilitatsiooni (10 treeningut) koos eelneva ja treeningujärgse testimisega. Tähelepanu ja ruumitaju treeningus kasutati arvutipõhist ForamenRehab lastele kohandatud programmi väljatöötatud treeningprotokollidega. Sotsiaalsete häirete raviks loodi esmalt struktureeritud neurorehabilitatsiooni mudel, mis koosnes sotsiaalse kompetentsi olulistest komponentidest, nende hindamismeetoditest ja rehabilitatsiooni vahenditest: puutetundlikud lauad Snowflake Multiteach Tabletop ja Diamond Touch Table, ning virtuaalreaalsuse keskkonnad. Tulemusena esines patsientidel treeningu eelselt väljendunud tähelepanu, ruumitaju ja sotsiaalse kompetentsi defitsiit. Arvutipõhised ja virtuaalreaalsuse programmid olid efektiivsed kognitiivsete häirete ravis. Patsientidel esines treeningute järgselt oluline paranemine kahes tähelepanu komponendis (tähelepanu jagamine ja seiramine) ja kolmes ruumitaju komponendis (visuaal-konstruktiivsed võimed, visuaalne tähelepanu ja nägemis-ruumitaju) ning raviefekt oli säilinud järeltestimisel 1,3 aastat hiljem. Sotsiaalse rehabilitatsiooni järgselt paranesid oluliselt patsientide vaimuteooria (Theory of Mind) ja emotsioonide äratundmine, kasutati rohkem koostööoskuseid, verbaalset ja mitteverbaalset kommunikatsiooni ning pragmaatika oskuseid. Uuringute tugevuseks oli sajaprotsendiline ravisoostumus ning positiivne tagasiside. Olulised on töö käigus välja töötatud teaduspõhised sekkumisprotokollid ja uued tehnoloogiapõhised rehabilitatsioonimeetodid häirunud funktsioonide spetsiifiliseks raviks lastel.Children with acquired brain injury (ABI) need diagnosis of accompanying cognitive and socio-emotional deficits and neurorehabilitation to enhance their future academic success and quality of life. Neurorehabilitation is a systematic intervention designed to compensate for or remediate the impairments caused by brain injury. The main aim of the thesis was implementing new computer-based programs, multitouch-multiuser tabletops and virtual reality in cognitive and social neurorehabilitation for children with ABI. Rehabilitation designs were developed for the treatment of attention, visuospatial, and social competence deficits. 59 children aged 8–12 years with ABI diagnosis (epilepsy, traumatic brain injury or tic disorder) and 47 healthy controls participated. Study group patients completed 10 training sessions guided by therapists. Pre-intervention assessments, and outcome assessments immediately and 1.31 years after the rehabilitation were carried out. ForamenRehab computer-programme was adapted to children and intervention protocols were created for attention and visuospatial function remediation. For social deficit remediation, the structured neurorehabilitation model was created, composed of the main components of social competence with evaluation and intervention tools: Snowflake Multiteach Tabletop, Diamond Touch Table and virtual reality programmes. Pre-intervention assessments showed that children with ABI had significant deficits in attention, visuospatial abilities and social competence functions. Computer-based and virtual reality programs were effective in the remediation of cognitive deficits in patients. After training, the patients had improved performance in two attention (complex attention and tracking) and three visuospatial components (visual organization, visual attention and visuospatial perception). The positive training effect had preserved after 1.3 years in follow-up assessments. Additionally, after social deficit rehabilitation, the patients showed improvements in Theory of Mind and emotion recognition, and they used more cooperation, communication, and pragmatic skills. The patients’ compliance was 100% and feedback was positive for all three interventions. In sum, the developed evidence-based intervention protocols and new technology-based rehabilitation methods are important in the remediation of specific cognitive deficits in children.https://www.ester.ee/record=b528718

Proceedings of the 7th international conference on disability, virtual reality and associated technologies, with ArtAbilitation (ICDVRAT 2008)

The proceedings of the conferenc

Presence 2005: the eighth annual international workshop on presence, 21-23 September, 2005 University College London (Conference proceedings)

OVERVIEW (taken from the CALL FOR PAPERS) Academics and practitioners with an interest in the concept of (tele)presence are invited to submit their work for presentation at PRESENCE 2005 at University College London in London, England, September 21-23, 2005. The eighth in a series of highly successful international workshops, PRESENCE 2005 will provide an open discussion forum to share ideas regarding concepts and theories, measurement techniques, technology, and applications related to presence, the psychological state or subjective perception in which a person fails to accurately and completely acknowledge the role of technology in an experience, including the sense of 'being there' experienced by users of advanced media such as virtual reality. The concept of presence in virtual environments has been around for at least 15 years, and the earlier idea of telepresence at least since Minsky's seminal paper in 1980. Recently there has been a burst of funded research activity in this area for the first time with the European FET Presence Research initiative. What do we really know about presence and its determinants? How can presence be successfully delivered with today's technology? This conference invites papers that are based on empirical results from studies of presence and related issues and/or which contribute to the technology for the delivery of presence. Papers that make substantial advances in theoretical understanding of presence are also welcome. The interest is not solely in virtual environments but in mixed reality environments. Submissions will be reviewed more rigorously than in previous conferences. High quality papers are therefore sought which make substantial contributions to the field. Approximately 20 papers will be selected for two successive special issues for the journal Presence: Teleoperators and Virtual Environments. PRESENCE 2005 takes place in London and is hosted by University College London. The conference is organized by ISPR, the International Society for Presence Research and is supported by the European Commission's FET Presence Research Initiative through the Presencia and IST OMNIPRES projects and by University College London

Gamification Strategies as Socialization Tools in the Context of Paediatric Hospitalization

[ES] En la presente tesis doctoral se aborda el problema de la socialización de los pacientes pediátricos en el contexto de su ingreso hospitalario, estudiando las posibilidades de mejorar este ámbito de la salud del paciente mediante diversas estrategias y herramientas tecnológicas teniendo en cuenta la problemática planteada y las dificultades propias del contexto. En primer lugar, se realizará una evaluación del estado del arte, así como una justificación de la problemática existente, planteando claramente los objetivos e hipótesis del presente trabajo, así como la metodología seguida. A continuación, se presentarán las diferentes herramientas implementadas y evaluadas, analizando los resultados obtenidos y situándolos en su contexto y marco tecnológico. Estas herramientas, que combinan actividades y aplicaciones presenciales, remotas o híbridas, implementan diferentes estrategias para poder valorar cuáles son las más adecuadas o que presentan un mayor impacto en el paciente. Tras esto, se propone un modelo de diseño basado en los resultados obtenidos en estudios anteriores, así como su posible aplicación en diferentes ámbitos, discutiendo el mismo y justificando las decisiones de diseño adoptadas. Finalmente, se establecerán las conclusiones obtenidas tras el análisis de los resultados y se plantearán cuáles son las posibles aplicaciones y trabajos futuros relacionados con el trabajo realizado en el marco de la tesis doctoral.[CA] Aquesta tesi doctoral aborda el problema de la socialització dels pacients pediàtrics durant la seua estada hospitalària, i s'estudien les possibilitats de millorar aquest àmbit de la salut del pacient mitjançant diverses estratègies i eines tecnològiques, considerant la problemàtica plantejada i les dificultats pròpies de l'entorn. En primer lloc, es realitzarà una avaluació de l'estat de l'art, així com una justificació de la problemàtica existent, i es plantejaran clarament els objectius i la hipòtesi del treball, així com la metodologia emprada. Seguidament, es presentaran les diferents eines implementades i avaluades, analitzant els resultats obtinguts i situant-los al seu context i marc tecnològic. Aquestes eines, que combinen activitats i aplicacions presencials, remotes o híbrides, implementen diferents estratègies per a poder valorar quines són les més adients, o les que presenten un major impacte en el pacient. A continuació, es proposa un model de disseny basat en els resultats obtinguts als estudis previs, així com la seua possible aplicació en diferents àmbits, discutint el model i justificant les decisions de disseny adoptades. Finalment, s'establiran les conclusions obtingudes després d'analitzar els resultats i es plantejaran les possibles aplicacions i treballs futurs relacionats amb els continguts desenvolupats durant la tesi doctoral.[EN] In this document, the problems for socialization for pediatric patients in the context of their hospital internment are considered, analyzing the possibilities of improving this aspect of the patient's health through the use of different strategies and technological tools considering the difficulties posed by the environment and the context. In the first place, an evaluation of the state of the literature and a justification of the existing problematic will be presented, clearly stating the objectives and hypothesis of this work, and also the methodology that has been followed. Afterwards, the different tools that have been implemented and evaluated will be described, analyzing the obtained results and placing them in their context and technological frame. These tools, that combine physically placed activities and applications with remote or hybrid ones, implement different strategies in order to be able to analyze which are the most convenient or the ones that show a higher benefit for the patient. In the following section, a design model, which is based on the results obtained in the previous studies, is proposed, with the justification of the design decisions made and a description of different applications and contexts in which the proposed model could be useful. Finally, the conclusions obtained after the analysis of the results will be presented, and a short discussion about possible applications and future related works will be done.Thanks to the Ministry for Education, that have funded my research through grant FPU17/03333Montaner Marco, J. (2023). Gamification Strategies as Socialization Tools in the Context of Paediatric Hospitalization [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/20155

Designing Activities for Collaboration at Classroom Scale Using Shared Technology

Although researchers, teachers and policy makers broadly agree on the benefits of collaborative learning, there appears to be less clarity regarding how effective collaboration can be realised at classroom scale. Research in Computer-Supported Collaborative Learning (CSCL), Human-Computer Interaction (HCI), simulation-based learning and related fields has produced a considerable range of applications that aim to support collaboration in classrooms. Grounded in well-established theories of how humans learn, many such applications have shown promising results within the context of small research studies. However, most of those research-driven applications never matured beyond the prototype stage and few are available today as products that schools can easily use and adopt. Many systems lack flexibility or require too much time, hardware, technical skills or other resources to be effectively implemented. Furthermore, teachers can be overwhelmed by managing large groups of students engaged in complex, computer-supported tasks. This thesis investigates how forms of whole-classroom activity can be supported by combining shareable technologies with simulation, team play and orchestration. New designs are explored to help large groups engage and discuss at multiple scales (from pairs and small groups to the entire classroom) in ways that effectively include each student and use the teacher's limited resources efficiently. Moreover, this research aims to devise and validate a conceptual framework that can guide future design, orchestration and evaluation of such activities. Three in-situ studies were conducted to address these goals. The first study involved the design of a climate change simulation to support a professional training course. Iterative design and video analysis resulted in the formulation of the Collaborative Learning Orchestration for Verbal Engagement and Reflection (CLOVER) framework. This framework comprises a suite of conceptual tools and recommendations that aim to help designers and teachers create, orchestrate and evaluate decision-based simulations for whole-classroom use. Two follow-up studies were conducted to validate the usability and usefulness of CLOVER. One of them aimed to replicate the previous findings in a similar context and resulted in the design of a sustainable, whole-classroom simulation for students to discuss finance decisions. The other used CLOVER to expand an existing desktop application (a~language comprehension task for children) to classroom scale. In sum, the three studies provide substantial empirical evidence, suggesting that CLOVER-based applications can effectively reconcile learning needs (collaboration) and technological affordances (shareable devices) with the inherent benefits and constraints of teacher-driven, co-located environments. Furthermore, the findings contribute to a better understanding of what it means to design for sustainability in this context

Kindertivity: Usability and Communicability Strategies for Interactive Surfaces and Pre-Kindergarten Children

Tesis por compendio[ES] La tecnología multi-táctil se ha convertido en una de las más emergentes tras experimentar un enorme crecimiento desde sus pasos iniciales en los años ochenta hasta su amplia aceptación y uso en la actualidad. Por una parte, la tecnología multi-táctil se basa en el estilo de interacción de manipulación directa el cual proporciona a los usuarios la ventaja de ver los objetos y las acciones de interés, sustituir comandos escritos por acciones de señalado y, además, permite la realización de acciones rápidas, reversibles e incrementales evitando el uso de instrucciones complejas. Por otra parte, diversos trabajos han evaluado las virtudes derivadas de utilizar conjuntamente la manipulación directa con el toque directo mostrando que es posible evitar los problemas inherentes a otras técnicas de interacción como el ratón y el teclado. Por lo tanto, aprovechando la interacción natural e intuitiva proporcionada por la tecnología multi-táctil, ésta parece una forma ideal para dar soporte a la creación de escenarios educativos dirigidos a niños en edad preescolar. Sin embargo, a pesar de la existencia de diversos estudios que evalúan la idoneidad de utilizar el estilo de interacción de manipulación directa, existe una falta de trabajos abordando el uso dispositivos basados en superficies táctiles con niños de una temprana edad. Asimismo, en la actualidad existe una creciente tendencia a diseñar aplicaciones educativas y lúdicas dirigidas a niños en edad preescolar utilizando dispositivos multi-táctiles como los teléfonos inteligentes o las tabletas. Además, diversos informes señalan que los niños son usuarios frecuentes de este tipo de dispositivos y los utilizan incluso antes de ser capaces de hablar. Sin embargo, a pesar de este crecimiento en el uso de la tecnología multi-táctil y su aparente idoneidad para ser utilizado en el desarrollo de aplicaciones educativas para niños en edad preescolar, no existen unas interacciones universales y estandarizadas para preescolares a la hora de utilizar dispositivos táctiles ya que habitualmente sólo se utilizan dos gestos básicos (básicamente, el toque con un dedo para seleccionar y el arrastre con un dedo para el movimiento). Por lo tanto, existe una clara necesidad de llevar a cabo estudios empíricos para contribuir y avanzar en el diseño de aplicaciones que den un soporte adecuado y encaje con las habilidades de los niños en su temprano desarrollo. Por tanto, esta tesis propone, diseña y evalúa diversas estrategias de usabilidad y comunicabilidad adaptadas a los niños en edad preescolar para establecer la base para el diseño y desarrollo de futuras aplicaciones basadas en dispositivos táctiles dirigidas a preescolares. Estas estrategias llevarán a la adecuada definición de guías de diseño que permitirán a los niños aprovechar al máximo la tecnología multi-táctil, harán posible el desarrollo de nuevas y atractivas aplicaciones y, eventualmente, también podrán ayudar al desarrollo cognitivo y motor de los niños.[CA] La tecnologia multi-tàctil s'ha convertit en una de les més emergents després d'experimentar un enorme creixement des dels seus passos inicials als anys vuitanta fins l'actualitat on es àmpliament acceptada i utilitzada. D'una banda, la tecnologia multi-tàctil es basa en l'estil d'interacció de manipulació directa, el qual proporciona als usuaris l'avantatge de veure els objectes i les accions d'interès, substituir comandos escrits per accions d'assenyalament i, a més, permet la realització d'accions, ràpides, reversibles i incrementals evitant l'ús d'instruccions complexes. D'altra banda, diversos treballs han avaluat les virtuts derivades d'utilitzar conjuntament la manipulació directa amb el toc directe mostrant que és possible evitar els problemes inherents a altres tècniques d'interacció com el ratolí i el teclat. Per tant, aprofitant la interacció natural i intuïtiva proporcionada per la tecnologia multi-tàctil, aquesta sembla una forma ideal per donar suport a la creació d'escenaris educatius per a xiquets en edat preescolar. No obstant això, malgrat l'existència de diversos estudis que avaluen la idoneïtat d'utilitzar l'estil d'interacció de manipulació directa, existeix una manca de treballs abordant l'ús de dispositius basats en superfícies tàctils amb xiquets d'edat primerenca. Així mateix, en l'actualitat existeix una creixent tendència a dissenyar aplicacions educatives i lúdiques dirigides a xiquets en edat preescolar utilitzant dispositius tàctils com els telèfons intel¿ligents o les tauletes. A més, diversos informes assenyalen que els xiquets són usuaris freqüents d'aquests tipus de dispositius i els utilitzen fins i tot abans de ser capaços de parlar. Malgrat aquest creixement en l'ús de la tecnologia multi-tàctil i la seua aparent idoneïtat per a ser utilitzada en el desenvolupament d'aplicacions educatives per a xiquets en edat preescolar, no existeixen unes interaccions universals i estandarditzades per a preescolars a l'hora d'utilitzar dispositius tàctils ja que habitualment només s'utilitzen dos gestos bàsics (bàsicament, el toc amb un dit per a seleccionar i l'arrossegament amb un dit per al moviment). Per tant, hi ha una clara necessitat de dur a terme estudis empírics per a contribuir i avançar en el disseny d'aplicacions que donen un suport adequat i s'ajusten amb les habilitats dels xiquets en el seu primerenc desenvolupament. Per tant, la tesi proposa, dissenya i avalua diverses estratègies de usabilitat i comunicabilitat adaptades als xiquets en edat preescolar per tal d'establir la base per al disseny i desenvolupament de futures aplicacions basades en dispositius tàctils dirigides a preescolars. Aquestes estratègies portaran a l'adequada definició de guies de disseny que permetran als xiquets aprofitar al màxim la tecnologia multi-tàctil, faran possible el desenvolupament de noves i atractives aplicacions i, eventualment, podran també ajudar al desenvolupament cognitiu i motor dels xiquets.[EN] Multi-touch technology has become one of the most emergent technologies and has had an enormous growth since its initial steps in the eighties to be widespread accepted and used in the present. On the one hand, multi-touch technology relies on the direct manipulation interaction style which gives users the advantage to view the objects and actions of interest, replace typed commands by pointing actions and to perform rapid, reversible and incremental actions avoiding using complex instructions. On the other hand, several works have evaluated the virtues when joining direct manipulation with direct-touching showing that it solves the problems inherent in other interaction devices, such as those involving mouse or keyboard. Hence, taking advantage of the intuitive and natural interaction provided by multi-touch technology it seems an ideal way to support educational scenarios targeted to kindergarten children. Although several works have assessed the suitability of using the direct manipulation style with children, there is a lack of works addressing the use of touchscreen devices by this specific type of users. Moreover, there is a growing trend of designing educational and playful applications targeted to kindergarten children based on touchscreen devices such as smartphones and tablets. In addition, several reports point out that children use touchscreen devices even before they are able to speak and they are frequent users of devices such as smartphones and tablets. However, despite this growth in the use of multi-touch technology by children and its apparent suitability to be used to develop applications targeted to young children, there is a lack of standardized and universally accepted interactions for young children when using touchscreen devices since only two basic gestures are commonly used (basically, consisting of only one-finger touch for selection and one-finger drag for movement). Hence, there is a need of carrying out empirical studies to help and advance in the design of applications that adequately support and fit with children's development and skills. Therefore, this thesis proposes, designs and evaluates several usability and communicability strategies tailored to children in their early development stage to establish the design and development of future applications targeted to kindergarten children. These strategies will lead to define appropriate design strategies that enable infants to take full advantage of multi-touch technology, would make it possible to develop attractive new applications and, eventually, could also aid children's cognitive and motor development.Nácher Soler, VE. (2019). Kindertivity: Usability and Communicability Strategies for Interactive Surfaces and Pre-Kindergarten Children [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/116833TESISCompendi