Serious Games Application for Memory Training Using Egocentric Images

Mild cognitive impairment is the early stage of several neurodegenerative diseases, such as Alzheimer's. In this work, we address the use of lifelogging as a tool to obtain pictures from a patient's daily life from an egocentric point of view. We propose to use them in combination with serious games as a way to provide a non-pharmacological treatment to improve their quality of life. To do so, we introduce a novel computer vision technique that classifies rich and non rich egocentric images and uses them in serious games. We present results over a dataset composed by 10,997 images, recorded by 7 different users, achieving 79% of F1-score. Our model presents the first method used for automatic egocentric images selection applicable to serious games.Comment: 11 page

Egocentric rich images detection in a serious game for Alzheimer’s patients

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2017, Director: Petia Radeva[en] Alzheimer’s is a progressive neurodegenerative disease, right now irreversible and related with ageing. It is caused by neuronal deterioration which leads to memory loss, intellectual impairment and personality disorders and behavior. Several international reports warn that Alzheimer’s is the most common form of dementia, affecting between 60% and 70% of the total number of cases. In addition, it has been estimated that at least 5% of the world’s population with more than 60 years old will suffer the disease at some point in their life. Despite the social and economic impact of the disease, there is still no treatment that can cure dementia or prevent its evolution. It is for this reason that efforts to find solutions that diminish the progressive development of the disease have been growing increasingly, building a society aware of these problems. The World Health Organization recognized dementia as a public priority in the report: "Dementia: a public health priority"[27]. The TV program L.La Marató de TV3"decided to dedicate the show of the year 2013 to the eurodegenerative diseases. Granted by it, this work has been aimed at reinforcing the inclusion of the new technologies in areas related with health to promote the improvement of Alzheimer’s patients. The framework developed in this project consists on the incorporation of a wearable camera that automatically takes pictures of the day of the patient. Subsequently, a selection is made, through the detection of objects in the images and a classification using Machine Learning algorithms, of those photographs which can bring value to the cognitive reinforcement of the patient. A serious game based on this pictures has been built to enhance this behavior

Implementation of an evaluation platform for Alzheimer patients based on Egocentric Sequences Description

Treballs Finals de Grau d'Enginyeria Informàtica, Facultat de Matemàtiques, Universitat de Barcelona, Any: 2017, Director: Marc Bolaños Solà i Petia RadevaNumerous international population-based studies have been conducted to document the frequency of MCI, estimating its prevalence to be between 15% and 20% in persons 60 years and older, making it a common condition encountered by clinicians[17]. This number is predicted to increase to 75.6 million in 2030, and 135.5 million in 2050[14], leading to deep social and economical costs. The most common dementia type is Alzheimer (between 50% and 70% of the cases) and its early detection can greatly affect the recovery of the patient. That is why it is important to have tools for its early diagnosis and follow-up. Serious games, with an increasing popularity, are a good way to MCI as an early stage of Alzheimer and improve the memory capacities of the patients. These video games focusing on different stages of the illness can help doctors to document and check the progress of the illness. This work aims on developing a software for patients with MCI, which is the lack of memory and other human characteristics like reasoning and language. These individuals usually progress to Alzheimer disease, but if detected early, in some cases they can also remain stable or even recover with time. To help them to exercise their memory, we propose that our program uses their own experiences caught by a wearable camera. This software will provide images of the patient’s life in order to do exercises that will evaluate their ability to remember and reason about the scenes they visualize. With these tests, the doctors will be able to see the evolution of the patients, and help them to diagnose and track the illness. In this project, we additionally work on an application, for the first time, of Deep Neural Networks for the automatic generation of descriptions of egocentric sequences. This will serve as the first step to automate the evaluation process by automatically comparing the subjective descriptions provided by the patients to the objective ones generated by our system

An immersive virtual reality task with physical movement for the assessment of spatial memory

Tesis por compendioLa Realidad Virtual (RV) aún no se ha explotado en la evaluación de la memoria espacial. Los sistemas actuales de RV para la evaluación de la memoria espacial incluyen interacción mediante el uso de dispositivos tradicionales (por ejemplo, pantallas y teclados). Este tipo de interacción clásica con el sistema es ineficaz porque la sensación de presencia del usuario es muy pobre. Si el usuario no tiene la sensación de "estar ahí" en el entorno virtual, es posible que la memoria espacial no se evalúe correctamente. Por tanto, desarrollar un sistema que permita a los participantes estar activos en el entorno virtual, podría evaluar la memoria espacial de la misma forma que se puede evaluar en un entorno real. El objetivo principal de esta tesis fue diseñar, desarrollar y validar un sistema inmersivo de RV en el que el usuario pueda interactuar mediante movimiento físico. El entorno virtual se diseñó a partir de una ciudad con forma cuadrada. La tarea cognitiva, basada en dicho entorno, constaba de seis niveles. El objetivo de los niveles para la tarea principal era evaluar la memoria a corto plazo de los niños para la localización de objetos en el entorno virtual. El sistema de RV incorporó dos tipos de interacción. 1) Una condición física activa (caminar físicamente sobre una Wii Balance Board y cambiar la dirección girando un volante inalámbrico). 2) Una condición inactiva (estando de pie y usando un gamepad). Para la visualización de la tarea, se utilizó una pantalla grande y estéreo. Para la validación, se realizaron dos estudios (N=212). El rendimiento de nuestra tarea se comparó con métodos tradicionales. Se encontraron correlaciones entre nuestra tarea y los métodos tradicionales, lo que indica que nuestra tarea ha demostrado ser una herramienta válida para evaluar la memoria espacial a corto plazo en niños. Con respecto al tipo de interacción, los resultados mostraron que no hubo diferencias estadísticamente significativas con respecto a la puntuación obtenida en nuestra tarea y en función de la interacción utilizada. Éste es un buen resultado porque significa que la tarea es adecuada para la evaluación de la memoria espacial y que los dos tipos de interacción se pueden utilizar para este fin. Con respecto a las diferencias de género en la puntuación de la tarea y Corsi Block Tapping Test, los resultados indicaron que no hubo diferencias estadísticamente significativas para el género. Con respecto a facilidad de uso y satisfacción, se demostró que el uso de la condición inactiva no difirió significativamente de la condición física activa para las preguntas de usabilidad y satisfacción. Previamente al desarrollo del sistema de RV mencionado, desarrollamos un sistema de RV, con interfaces de usuario naturales (NUI) y una pantalla autoestereoscópica, para aprendizaje dental. El sistema incluye dos modos: fondo neutro y mundo real. Este sistema fue validado con 33 estudiantes. Con este primer desarrollo, se adquirió el conocimiento necesario para poder afrontar el segundo desarrollo, núcleo de la tesis. Las siguientes conclusiones generales se extrajeron de los 2 desarrollos y los 3 estudios: Aprendizaje - Los sistemas de RV con autoestereoscopía, con diferentes fondos y NUI, han demostrado ser herramientas eficaces para aprender la morfología dental - Con este tipo de sistemas, los niños pueden aprender y, al mismo tiempo, pueden divertirse - La estereoscopía y NUI son apropiadas para el desarrollo de juegos educativos y pueden ser explotadas en su desarrollo Memoria espacial - Los sistemas de RV, con estereoscopía y dos interfaces de usuario diferentes, han demostrado ser herramientas fiables y efectivas para evaluar la memoria espacial en niños - Con este tipo de sistemas, los niños pueden ser evaluados mientras se divierten - Nuestra tarea y tareas similares podrían usarse para evaluación y entrenamiento de la memoria en niños y adultosLa Realitat Virtual (RV) encara no s'ha explotat en l'avaluació de la memòria espacial. Els sistemes actuals de RV per a l'avaluació de la memòria espacial inclouen interacció mitjançant l'ús de dispositius tradicionals (per exemple, pantalles, ratolins o teclats). Aquest tipus d'interacció clàssica amb el sistema és ineficaç perquè la sensació de presència de l'usuari és molt pobre. Si l'usuari no té la sensació de "ser-hi" en l'entorn virtual, és possible que la memòria espacial no siga avaluada correctament. Per tant, desenvolupar un sistema que permeta als participants estar actius en l'entorn virtual, podria avaluar la memòria espacial de la mateixa manera que es pot avaluar en un entorn real. L'objectiu principal d'aquesta tesi va ser dissenyar, desenvolupar i validar un sistema immersiu de RV en el qual l'usuari puga interactuar mitjançant moviment físic. L'entorn virtual es va dissenyar a partir d'una ciutat amb forma quadrada. La tasca cognitiva constava de sis nivells. L'objectiu dels nivells per a la tasca principal era avaluar la memòria a curt termini dels xiquets per a la localització d'objectes en l'entorn virtual. El sistema de RV va incorporar dos tipus d'interacció. 1) Una condició física activa (caminar físicament sobre una Wii Balance Board i canviar la direcció girant un volant inalàmbric). 2) Una condició inactiva (estar dret i fent servir un gamepad). Per a la visualització de la tasca, es va utilitzar una pantalla gran i estèreo. Per a la validació, es van realitzar dos estudis (N=212). El rendiment de la nostra tasca es va comparar amb mètodes tradicionals. Es van trobar correlacions entre la nostra tasca i els mètodes tradicionals, el que indica que la nostra tasca ha demostrat ser una eina vàlida per avaluar la memòria espacial a curt termini en xiquets. Pel que fa al tipus d'interacció, els resultats van mostrar que no hi va haver diferències estadísticament significatives respecte a la puntuació obtinguda en la nostra tasca i en funció de la interacció utilitzada. Aquest és un bon resultat perquè significa que la tasca és adequada per a l'avaluació de la memòria espacial i que els dos tipus d'interacció es poden utilitzar per a aquest fi. Pel que fa a les diferències de gènere en la puntuació de la tasca i Corsi Block Tapping Test, els resultats van indicar que no hi va haver diferències estadísticament significatives per al gènere. Pel que fa a facilitat d'ús i satisfacció, els nostres estudis han demostrat que l'ús de la condició inactiva no va diferir significativament de la condició física activa per a les preguntes d'usabilitat i satisfacció. Prèviament al desenvolupament del sistema de RV esmentat, vam desenvolupar un sistema de RV, amb interfícies d'usuari naturals (NUI) i una pantalla autoestereoscòpica, per aprenentatge dental. El sistema inclou dos tipus de fons: fons neutre i món real. Aquest sistema va ser validat amb 33 estudiants. Amb aquest primer desenvolupament, es va adquirir el coneixement necessari per poder afrontar el segon desenvolupament, nucli de la tesi. Les següents conclusions generals es van extraure dels dos desenvolupaments i els tres estudis: Aprenentatge - Els sistemes de RV amb autoestereoscòpia, amb diferents fons i NUI, han demostrat ser eines eficaces per a aprendre la morfologia dental - Amb aquest tipus de sistemes, els xiquets poden aprendre i, al mateix temps, poden divertir-se - La estereoscòpia i NUI són apropiades per al desenvolupament de jocs educatius i poden ser explotades en el seu desenvolupament Memòria espacial - Els sistemes de RV, amb estereoscòpia i dues interfícies d'usuari diferents, han demostrat ser eines fiables i efectives per avaluar la memòria espacial en xiquets - Amb aquest tipus de sistemes, els xiquets poden ser avaluats mentre es diverteixen - La nostra tasca i tasques similars podrien usar-se per avaluació i entrenament de la memòria en xiquet i adultsVirtual Reality (VR) has not yet been exploited in the assessment of spatial memory. Current VR systems for the assessment of spatial memory include interaction by using traditional devices (e.g., computer screens, mouses or keyboards). This classical type of interaction with the system is ineffective because the user's sense of presence is very poor. If the user does not have the feeling of "being there" in the virtual environment, the spatial memory may not be assessed correctly. The user's physical movement can contribute to a high level of presence. Therefore, by developing a system that allows the participants to become active in a virtual environment, spatial memory can be assessed as the same way it could be evaluated in a real environment. The main objective of this thesis was to design, develop and validate an immersive VR system in which the user could interact by physical movements. The virtual environment was designed based on a city square. The cognitive task, based on this environment, comprised six levels. The goal of the levels for the main task was to assess children short-term memory for object location in the virtual environment. The VR system incorporated two types of interaction. 1) A physical active condition (physically walking on a Wii Balance Board and changing the direction by turning a wireless steering wheel). 2) An inactive condition (stand up and use a gamepad). For the visualization of the task, a large stereo screen was used. For the validation, two studies were carried out. The performance of our task was compared with traditional methods (the Corsi Block Tapping Test). We carried out two studies involving 212 children. Correlations were found between our task and traditional methods, indicating that our task has proven to be a valid tool for assessing spatial short-term memory in children. With regard to the interaction type, the results showed that there were no statistically significant differences regarding the score obtained in our task based on the interaction used (inactive condition vs. physical active condition). Although unexpected, this is a good result because it means that the task is well suited for the assessment of spatial memory and that the two interaction types can be used for this purpose. With regard to gender differences in the task score and the Corsi Block Tapping Test, the results indicated that there were no statistically significant differences for gender. With regard to usability and satisfaction, our studies have shown that the use of the inactive condition did not differ significantly from the physical active condition for the usability and satisfaction questions. Previously to the development of the mentioned VR system, we developed a VR system, with Natural User Interfaces (NUI) and an autostereoscopic screen, for dental learning. The system included two modes: neutral and real world background. This system was validated with 33 dentistry students. With this first development, the required knowledge for facing the second development, core of the thesis, was acquired. The following general conclusions were extracted from the two developments and the three studies: Learning - Autostereoscopic VR systems, with different background modes and NUI, have proven to be effective tools for learning teeth morphology - With this type of systems, children can learn and at the same time, they can have a good time - Stereoscopy and NUI are appropriated for developing educational games and they can be exploited in their development Spatial memory - VR systems, with stereoscopy and two different user interfaces (inactive and physical active conditions), have proven to be reliable and effective tools to assess spatial memory in children - With this type of systems, the children can be assessed meanwhile are having a good time - Our task and similar tasks could be used for assessment and training of spatial memory in children and adultsRodríguez Andrés, D. (2018). An immersive virtual reality task with physical movement for the assessment of spatial memory [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/114823TESISCompendi

Spatial cognition in virtual environments

Since the last decades of the past century, Virtual Reality (VR) has been developed also as a methodology in research, besides a set of helpful applications in medical field (trainings for surgeons, but also rehabilitation tools). In science, there is still no agreement if the use of this technology in research on cognitive processes allows us to generalize results found in a Virtual Environment (VE) to the human behavior or cognition in the real world. This happens because of a series of differences found in basic perceptual processes (for example, depth perception) suggest a big difference in visual environmental representation capabilities of Virtual scenarios. On the other side, in literature quite a lot of studies can be found, which give a proof of VEs reliability in more than one field (trainings and rehabilitation, but also in some research paradigms). The main aim of this thesis is to investigate if, and in which cases, these two different views can be integrated and shed a new light and insights on the use of VR in research. Through the many experiments conducted in the "Virtual Development and Training Center" of the Fraunhofer Institute in Magdeburg, we addressed both low-level spatial processes (within an "evaluation of distances paradigm") and high-level spatial cognition (using a navigation and visuospatial planning task, called "3D Maps"), trying to address, at the same time, also practical problems as, for example, the use of stereoscopy in VEs or the problem of "Simulator Sickness" during navigation in immersive VEs. The results obtained with our research fill some gaps in literature about spatial cognition in VR and allow us to suggest that the use of VEs in research is quite reliable, mainly if the investigated processes are from the higher level of complexity. In this case, in fact, human brain "adapts" pretty well even to a "new" reality like the one offered by the VR, providing of course a familiarization period and the possibility to interact with the environment; the behavior will then be “like if” the environment was real: what is strongly lacking, at the moment, is the possibility to give a completely multisensorial experience, which is a very important issue in order to get the best from this kind of “visualization” of an artificial world. From a low-level point of view, we can confirm what already found in literature, that there are some basic differences in how our visual system perceives important spatial cues as depth and relationships between objects, and, therefore, we cannot talk about "similar environments" talking about VR and reality. The idea that VR is a "different" reality, offering potentially unlimited possibilities of use, even overcoming some physical limits of the real world, in which this "new" reality can be acquired by our cognitive system just by interacting with it, is therefore discussed in the conclusions of this work

Spatial cognition in virtual environments

Since the last decades of the past century, Virtual Reality (VR) has been developed also as a methodology in research, besides a set of helpful applications in medical field (trainings for surgeons, but also rehabilitation tools). In science, there is still no agreement if the use of this technology in research on cognitive processes allows us to generalize results found in a Virtual Environment (VE) to the human behavior or cognition in the real world. This happens because of a series of differences found in basic perceptual processes (for example, depth perception) suggest a big difference in visual environmental representation capabilities of Virtual scenarios. On the other side, in literature quite a lot of studies can be found, which give a proof of VEs reliability in more than one field (trainings and rehabilitation, but also in some research paradigms). The main aim of this thesis is to investigate if, and in which cases, these two different views can be integrated and shed a new light and insights on the use of VR in research. Through the many experiments conducted in the "Virtual Development and Training Center" of the Fraunhofer Institute in Magdeburg, we addressed both low-level spatial processes (within an "evaluation of distances paradigm") and high-level spatial cognition (using a navigation and visuospatial planning task, called "3D Maps"), trying to address, at the same time, also practical problems as, for example, the use of stereoscopy in VEs or the problem of "Simulator Sickness" during navigation in immersive VEs. The results obtained with our research fill some gaps in literature about spatial cognition in VR and allow us to suggest that the use of VEs in research is quite reliable, mainly if the investigated processes are from the higher level of complexity. In this case, in fact, human brain "adapts" pretty well even to a "new" reality like the one offered by the VR, providing of course a familiarization period and the possibility to interact with the environment; the behavior will then be “like if” the environment was real: what is strongly lacking, at the moment, is the possibility to give a completely multisensorial experience, which is a very important issue in order to get the best from this kind of “visualization” of an artificial world. From a low-level point of view, we can confirm what already found in literature, that there are some basic differences in how our visual system perceives important spatial cues as depth and relationships between objects, and, therefore, we cannot talk about "similar environments" talking about VR and reality. The idea that VR is a "different" reality, offering potentially unlimited possibilities of use, even overcoming some physical limits of the real world, in which this "new" reality can be acquired by our cognitive system just by interacting with it, is therefore discussed in the conclusions of this work

Topic modelling for routine discovery from egocentric photo-streams

Developing tools to understand and visualize lifestyle is of high interest when addressing the improvement of habits and well-being of people. Routine, defined as the usual things that a person does daily, helps describe the individuals' lifestyle. With this paper, we are the first ones to address the development of novel tools for automatic discovery of routine days of an individual from his/her egocentric images. In the proposed model, sequences of images are firstly characterized by semantic labels detected by pre-trained CNNs. Then, these features are organized in temporal-semantic documents to later be embedded into a topic models space. Finally, Dynamic-Time-Warping and Spectral-Clustering methods are used for final day routine/non-routine discrimination. Moreover, we introduce a new EgoRoutine-dataset, a collection of 104 egocentric days with more than 100.000 images recorded by 7 users. Results show that routine can be discovered and behavioural patterns can be observed

Design and evaluation of auditory spatial cues for decision making within a game environment for persons with visual impairments

An audio platform game was created and evaluated in order to answer the question of whether or not an audio game could be designed that effectively conveys the spatial information necessary for persons with visual impairments to successfully navigate the game levels and respond to audio cues in time to avoid obstacles. The game used several types of audio cues (sounds and speech) to convey the spatial setup (map) of the game world. Most audio-only players seemed to be able to create a workable mental map from the game\u27s sound cues alone, pointing to potential for the further development of similar audio games for persons with visual impairments. The research also investigated the navigational strategies used by persons with visual impairments and the accuracy of the participants\u27 mental maps as a consequence of their navigational strategy. A comparisons of the maps created by visually impaired participants with those created by sighted participants playing the game with and without graphics, showed no statistically significant difference in map accuracy between groups. However, there was a marked difference between the number of invented objects when we compared this value between the sighted audio-only group and the other groups, which could serve as an area for future research