Remote Visual Observation of Real Places Through Virtual Reality Headsets

Virtual Reality has always represented a fascinating yet powerful opportunity that has attracted studies and technology developments, especially since the latest release on the market of powerful high-resolution and wide field-of-view VR headsets. While the great potential of such VR systems is common and accepted knowledge, issues remain related to how to design systems and setups capable of fully exploiting the latest hardware advances. The aim of the proposed research is to study and understand how to increase the perceived level of realism and sense of presence when remotely observing real places through VR headset displays. Hence, to produce a set of guidelines that give directions to system designers about how to optimize the display-camera setup to enhance performance, focusing on remote visual observation of real places. The outcome of this investigation represents unique knowledge that is believed to be very beneficial for better VR headset designs towards improved remote observation systems. To achieve the proposed goal, this thesis presents a thorough investigation of existing literature and previous researches, which is carried out systematically to identify the most important factors ruling realism, depth perception, comfort, and sense of presence in VR headset observation. Once identified, these factors are further discussed and assessed through a series of experiments and usability studies, based on a predefined set of research questions. More specifically, the role of familiarity with the observed place, the role of the environment characteristics shown to the viewer, and the role of the display used for the remote observation of the virtual environment are further investigated. To gain more insights, two usability studies are proposed with the aim of defining guidelines and best practices. The main outcomes from the two studies demonstrate that test users can experience an enhanced realistic observation when natural features, higher resolution displays, natural illumination, and high image contrast are used in Mobile VR. In terms of comfort, simple scene layouts and relaxing environments are considered ideal to reduce visual fatigue and eye strain. Furthermore, sense of presence increases when observed environments induce strong emotions, and depth perception improves in VR when several monocular cues such as lights and shadows are combined with binocular depth cues. Based on these results, this investigation then presents a focused evaluation on the outcomes and introduces an innovative eye-adapted High Dynamic Range (HDR) approach, which the author believes to be of great improvement in the context of remote observation when combined with eye-tracked VR headsets. Within this purpose, a third user study is proposed to compare static HDR and eye-adapted HDR observation in VR, to assess that the latter can improve realism, depth perception, sense of presence, and in certain cases even comfort. Results from this last study confirmed the author expectations, proving that eye-adapted HDR and eye tracking should be used to achieve best visual performances for remote observation in modern VR systems

Fatal attraction: identifying mobile devices through electromagnetic emissions

Smartphones are increasingly augmented with sensors for a variety of purposes. In this paper, we show how magnetic field emissions can be used to fingerprint smartphones. Previous work on identification rely on specific characteristics that vary with the settings and components available on a device. This limits the number of devices on which one approach is effective. By contrast, all electronic devices emit a magnetic field which is accessible either through the API or measured through an external device. We conducted an in-the-wild study over four months and collected mobile sensor data from 175 devices. In our experiments we observed that the electromagnetic field measured by the magnetometer identifies devices with an accuracy of 98.9%. Furthermore, we show that even if the sensor was removed from the device or access to it was discontinued, identification would still be possible from a secondary device in close proximity to the target. Our findings suggest that the magnetic field emitted by smartphones is unique and fingerprinting devices based on this feature can be performed without the knowledge or cooperation of users

Desarrollo de un sistema de seguimiento de usuarios con iPhone para visualizarlos en un modelo 3D

El objetivo de este proyecto es desarrollar un sistema de seguimiento de usuarios con un iPhone y un modelo 3D del campus de la Technical University of Denmark. El usuario podrá activar el seguimiento tras abrir una aplicación en el iPhone siempre y cuando se encuentre en alguna de las áreas donde haya un modelo 3D disponible. Los usuarios que hayan activado el seguimiento serán mostrados en estos modelos 3D en forma de avatares. Los modelos 3D junto con los avatares pueden ser visualizados usando cualquier navegador de escritorio en la página web realsite.dk. Los sensores GPS de los Smartphones no son normalmente muy precisos. Para desarrollar buenos algoritmos en el sistema de seguimiento requerido, la precisión de este sensor tiene que ser analizada. Por esta razón el proyecto empieza con un extenso estudio de la precisión de los sistemas de localización en el iPhone y de los parámetros que pueden configurarse. Se estudian tanto posiciones fijas como en movimiento. Este estudio revela que el error medio en posiciones estáticas es en torno a 8 metros y bastante mayor para las posiciones en movimiento. Sin embargo es muy rápido determinando la primera posición lo cual lo hace en menos de 10 segundos en la mayoría de los casos. Utilizando los resultados de este estudio, se han diseñado varios filtros para eliminar las posiciones menos precisas. Además, también se ha desarrollado una técnica que permite detectar cuando el usuario entra dentro de un edificio sin usar ninguna información adicional más que la que los servicios de localización ofrecen. Las dos partes mas importantes de este sistema han sido desarrolladas en su totalidad en este proyecto fin de carrera. Estas son una aplicación para el sistema operativo móvil iOS y un algoritmo para representar a los avatares de los usuarios en los modelos 3D. La aplicación recoge las posiciones de los usuarios, utilizando el GPS del dispositivo, las filtra, las guarda y las manda a un servidor de internet donde son almacenadas en una base de datos. También permite visualizar las sesiones anteriores en las que el seguimiento ha sido activado y tomar una foto que será utilizada en el avatar del usuario. La representación de los avatares en el modelo no se puede llevar a cabo con las posiciones que el dispositivo iOS obtiene ya que no son suficientemente precisas. Por lo que se diseñó un algoritmo que genera a partir de las posiciones GPS recibidas una ruta realista, factible y libre de obstáculos en el modelo. Un detalle importante por ejemplo, es que hace que los avatares utilicen escaleras y puertas de edificios cuando se detecta que han cambiado de altitud o entrado a un edificio respectivamente

Hardware and software fingerprinting of mobile devices

This dissertation presents novel and practical algorithms to identify the software and hardware components on mobile devices. In particular, we make significant contributions in two challenging areas: library fingerprinting, to identify third-party software libraries, and device fingerprinting, to identify individual hardware components. Our work has significant implications for the privacy and security of mobile platforms. Software-based library fingerprinting can be used to detect vulnerable libraries and uncover large-scale data collection activities. We develop a novel Android library finger-printing tool, LibID, to reliably identify specific versions of in-app third-party libraries. LibID is more effective against code obfuscation than prior art. When comparing LibID with other tools in identifying the correct library version using obfuscated F-Droid apps, LibID achieves an F1 score of more than 0.5 in all cases while prior work is below 0.25. We also demonstrate the utility of LibID by detecting the use of a vulnerable version of the OkHttp library in nearly 10% of the 3 958 popular apps on the Google Play Store. Hardware-based device fingerprinting allows apps and websites to invade user privacy by tracking user activity online as the user moves between apps or websites. In particular, we present a new type of device fingerprinting attack, the factory calibration fingerprinting attack, that recovers embedded per-device factory calibration data from motion sensors in a smartphone. We investigate the calibration behaviour of each sensor and show that the calibration fingerprint is fast to generate, does not change over time or after a factory reset, and can be obtained without any special user permissions. We estimate the entropy of the calibration fingerprint and find the fingerprint is very likely to be globally unique for iOS devices (~67 bits of entropy for iPhone 6S) and recent Google Pixel devices (~57 bits of entropy for Pixel 4/4 XL). By comparison, the fingerprint generated by previous work has at most 13 bits of entropy. Following our disclosures, Apple deployed a fix in iOS 12.2 and Google in Android 11. Both code obfuscation and factory calibration help to hide software and hardware idiosyncrasies from third-parties, but this dissertation demonstrates that reliable software and hardware fingerprints can still be generated given sufficient knowledge and a suitable approach. Our work has significant practical implications and can be used to improve platform security and protect user privacy.China Scholarship Council The Boeing Company Microsoft Researc

Machine learning techniques for identification using mobile and social media data

Networked access and mobile devices provide near constant data generation and collection. Users, environments, applications, each generate different types of data; from the voluntarily provided data posted in social networks to data collected by sensors on mobile devices, it is becoming trivial to access big data caches. Processing sufficiently large amounts of data results in inferences that can be characterized as privacy invasive. In order to address privacy risks we must understand the limits of the data exploring relationships between variables and how the user is reflected in them. In this dissertation we look at data collected from social networks and sensors to identify some aspect of the user or their surroundings. In particular, we find that from social media metadata we identify individual user accounts and from the magnetic field readings we identify both the (unique) cellphone device owned by the user and their course-grained location. In each project we collect real-world datasets and apply supervised learning techniques, particularly multi-class classification algorithms to test our hypotheses. We use both leave-one-out cross validation as well as k-fold cross validation to reduce any bias in the results. Throughout the dissertation we find that unprotected data reveals sensitive information about users. Each chapter also contains a discussion about possible obfuscation techniques or countermeasures and their effectiveness with regards to the conclusions we present. Overall our results show that deriving information about users is attainable and, with each of these results, users would have limited if any indication that any type of analysis was taking place

MULTI-SENSOR LOCALIZATION AND TRACKING IN DISASTER MANAGEMENT AND INDOOR WAYFINDING FOR VISUALLY IMPAIRED USERS

This dissertation proposes a series of multi-sensor localization and tracking algorithms particularly developed for two important application domains, which are disaster management and indoor wayfinding for blind and visually impaired (BVI) users. For disaster management, we developed two different localization algorithms, one each for Radio Frequency Identification (RFID) and Bluetooth Low Energy (BLE) technology, which enable the disaster management system to track patients in real-time. Both algorithms work in the absence of any pre-deployed infrastructure along with smartphones and wearable devices. Regarding indoor wayfinding for BVI users, we have explored several types of indoor positioning techniques including BLE-based, inertial, visual and hybrid approaches to offer accurate and reliable location and orientation in complex navigation spaces. In this dissertation, significant contributions have been made in the design and implementation of various localization and tracking algorithms under different requirements of certain applications

Anticipatory Mobile Computing: A Survey of the State of the Art and Research Challenges

Today's mobile phones are far from mere communication devices they were ten years ago. Equipped with sophisticated sensors and advanced computing hardware, phones can be used to infer users' location, activity, social setting and more. As devices become increasingly intelligent, their capabilities evolve beyond inferring context to predicting it, and then reasoning and acting upon the predicted context. This article provides an overview of the current state of the art in mobile sensing and context prediction paving the way for full-fledged anticipatory mobile computing. We present a survey of phenomena that mobile phones can infer and predict, and offer a description of machine learning techniques used for such predictions. We then discuss proactive decision making and decision delivery via the user-device feedback loop. Finally, we discuss the challenges and opportunities of anticipatory mobile computing.Comment: 29 pages, 5 figure