Sensing Movement on Smartphone Devices to Assess User Interaction for Face Verification

Unlocking and protecting smartphone devices has become easier with the introduction of biometric face verification, as it has the promise of a secure and quick authentication solution to prevent unauthorised access. However, there are still many challenges for this biometric modality in a mobile context, where the user’s posture and capture device are not constrained. This research proposes a method to assess user interaction by analysing sensor data collected in the background of smartphone devices during verification sample capture. From accelerometer data, we have extracted magnitude variations and angular acceleration for pitch, roll, and yaw (angles around the x-axis, y-axis, and z-axis of the smartphone respectively) as features to describe the amplitude and number of movements during a facial image capture process. Results obtained from this experiment demonstrate that it can be possible to ensure good sample quality and high biometric performance by applying an appropriate threshold that will regulate the amplitude on variations of the smartphone movements during facial image capture. Moreover, the results suggest that better quality images are obtained when users spend more time positioning the smartphone before taking an image

Identifying Users with Wearable Sensors based on Activity Patterns

We live in a world where ubiquitous systems surround us in the form of automated homes, smart appliances and wearable devices. These ubiquitous systems not only enhance productivity but can also provide assistance given a variety of different scenarios. However, these systems are vulnerable to the risk of unauthorized access, hence the ability to authenticate the end-user seamlessly and securely is important. This paper presents an approach for user identification given the physical activity patterns captured using on-body wearable sensors, such as accelerometer, gyroscope, and magnetometer. Three machine learning classifiers have been used to discover the activity patterns of users given the data captured from wearable sensors. The recognition results prove that the proposed scheme can effectively recognize a user’s identity based on his/her daily living physical activity patterns

Effective Identity Management on Mobile Devices Using Multi-Sensor Measurements

Due to the dramatic increase in popularity of mobile devices in the past decade, sensitive user information is stored and accessed on these devices every day. Securing sensitive data stored and accessed from mobile devices, makes user-identity management a problem of paramount importance. The tension between security and usability renders the task of user-identity verification on mobile devices challenging. Meanwhile, an appropriate identity management approach is missing since most existing technologies for user-identity verification are either one-shot user verification or only work in restricted controlled environments. To solve the aforementioned problems, we investigated and sought approaches from the sensor data generated by human-mobile interactions. The data are collected from the on-board sensors, including voice data from microphone, acceleration data from accelerometer, angular acceleration data from gyroscope, magnetic force data from magnetometer, and multi-touch gesture input data from touchscreen. We studied the feasibility of extracting biometric and behaviour features from the on-board sensor data and how to efficiently employ the features extracted to perform user-identity verification on the smartphone device. Based on the experimental results of the single-sensor modalities, we further investigated how to integrate them with hardware such as fingerprint and Trust Zone to practically fulfill a usable identity management system for both local application and remote services control. User studies and on-device testing sessions were held for privacy and usability evaluation.Computer Science, Department o

Multi-Factor Authentication: A Survey

Today, digitalization decisively penetrates all the sides of the modern society. One of the key enablers to maintain this process secure is authentication. It covers many different areas of a hyper-connected world, including online payments, communications, access right management, etc. This work sheds light on the evolution of authentication systems towards Multi-Factor Authentication (MFA) starting from Single-Factor Authentication (SFA) and through Two-Factor Authentication (2FA). Particularly, MFA is expected to be utilized for human-to-everything interactions by enabling fast, user-friendly, and reliable authentication when accessing a service. This paper surveys the already available and emerging sensors (factor providers) that allow for authenticating a user with the system directly or by involving the cloud. The corresponding challenges from the user as well as the service provider perspective are also reviewed. The MFA system based on reversed Lagrange polynomial within Shamir’s Secret Sharing (SSS) scheme is further proposed to enable more flexible authentication. This solution covers the cases of authenticating the user even if some of the factors are mismatched or absent. Our framework allows for qualifying the missing factors by authenticating the user without disclosing sensitive biometric data to the verification entity. Finally, a vision of the future trends in MFA is discussed.Peer reviewe

Mobile devices for the remote acquisition of physiological and behavioral biomarkers in psychiatric clinical research

Psychiatric disorders are linked to a variety of biological, psychological, and contextual causes and consequences. Laboratory studies have elucidated the importance of several key physiological and behavioral biomarkers in the study of psychiatric disorders, but much less is known about the role of these biomarkers in naturalistic settings. These gaps are largely driven by methodological barriers to assessing biomarker data rapidly, reliably, and frequently outside the clinic or laboratory. Mobile health (mHealth) tools offer new opportunities to study relevant biomarkers in concert with other types of data (e.g., self-reports, global positioning system data). This review provides an overview on the state of this emerging field and describes examples from the literature where mHealth tools have been used to measure a wide array of biomarkers in the context of psychiatric functioning (e.g., psychological stress, anxiety, autism, substance use). We also outline advantages and special considerations for incorporating mHealth tools for remote biomarker measurement into studies of psychiatric illness and treatment and identify several specific opportunities for expanding this promising methodology. Integrating mHealth tools into this area may dramatically improve psychiatric science and facilitate highly personalized clinical care of psychiatric disorders

Exploring Audio Sensing in Detecting Social Interactions Using Smartphone Devices

In recent years, the fast proliferation of smartphones devices has provided powerful and portable methodologies for integrating sensing systems which can run continuously and provide feedback in real-time. The mobile crowd-sensing of human behaviour is an emerging computing paradigm that offers a challenge of sensing everyday social interactions performed by people who carry smartphone devices upon themselves. Typical smartphone sensors and the mobile crowd-sensing paradigm compose a process where the sensors present, such as the microphone, are used to infer social relationships between people in diverse social settings, where environmental factors can be dynamic and the infrastructure of buildings can vary. The typical approaches in detecting social interactions between people consider the use of co-location as a proxy for real-world interactions. Such approaches can under-perform in challenging situations where multiple social interactions can occur within close proximity to each other, for example when people are in a queue at the supermarket but not a part of the same social interaction. Other approaches involve a limitation where all participants of a social interaction must carry a smartphone device with themselves at all times and each smartphone must have the sensing app installed. The problem here is the feasibility of the sensing system, which relies heavily on each participant's smartphone acting as nodes within a social graph, connected together with weighted edges of proximity between the devices; when users uninstall the app or disable background sensing, the system is unable to accurately determine the correct number of participants. In this thesis, we present two novel approaches to detecting co-located social interac- tions using smartphones. The first relies on the use of WiFi signals and audio signals to distinguish social groups interacting within a few meters from each other with 88% precision. We orchestrated preliminary experiments using WiFi as a proxy for co-location between people who are socially interacting. Initial results showed that in more challenging scenarios, WiFi is not accurate enough to determine if people are socially interacting within the same social group. We then made use of audio as a second modality to capture the sound patterns of conversations to identify and segment social groups within close proximity to each other. Through a range of real-world experiments (social interactions in meeting scenarios, coffee shop scenarios, conference scenarios), we demonstrate a technique that utilises WiFi fingerprinting, along with sound fingerprinting to identify these social groups. We built a system which performs well, and then optimized the power consumption and improved the performance to 88% precision in the most challenging scenarios using duty cycling and data averaging techniques. The second approach explores the feasibility of detecting social interactions without the need of all social contacts to carry a social sensing device. This work explores the use of supervised and unsupervised Deep Learning techniques before concluding on the use of an Autoencoder model to perform a Speaker Identification task. We demonstrate how machine learning can be used with the audio data collected from a singular device as a speaker identification framework. Speech from people is used as the input to our Autoencoder model and then classified against a list of "social contacts" to determine if the user has spoken a person before or not. By doing this, the system can count the number of social contacts belonging to the user, and develop a database of common social contacts. Through the use 100 randomly-generated social conversations and the use of state-of-the-art Deep Learning techniques, we demonstrate how this system can accurately distinguish new and existing speakers from a data set of voices, to count the number of daily social interactions a user encounters with a precision of 75%. We then optimize the model using Hyperparameter Optimization to ensure that the model is most optimal for the task. Unlike most systems in the literature, this approach would work without the need to modify the existing infrastructure of a building, and without all participants needing to install the same ap

Data Behind Mobile Behavioural Biometrics – a Survey

Behavioural biometrics are becoming more and more popular. It is hard to find a sensor that is embedded in a mobile/wearable device, which can’t be exploited to extract behavioural biometric data. In this paper, we investigate data in behavioural biometrics and how this data is used in experiments, especially examining papers that introduce new datasets. We will not examine performance accomplished by the algorithms used since a system’s performance is enormously affected by the data used, its amount and quality. Altogether, 32 papers are examined, assessing how often they are cited, have databases published, what modality data are collected, and how the data is used. We offer a roadmap that should be taken into account when designing behavioural data collection and using collected data. We further look at the General Data Protection Regulation, and its significance to the scientific research in the field of biometrics. It is possible to conclude that there is a need for publicly available datasets with comprehensive experimental protocols, similarly established in facial recognition

Distributed, Low-Cost, Non-Expert Fine Dust Sensing with Smartphones

Diese Dissertation behandelt die Frage, wie mit kostengünstiger Sensorik Feinstäube in hoher zeitlicher und räumlicher Auflösung gemessen werden können. Dazu wird ein neues Sensorsystem auf Basis kostengünstiger off-the-shelf-Sensoren und Smartphones vorgestellt, entsprechende robuste Algorithmen zur Signalverarbeitung entwickelt und Erkenntnisse zur Interaktions-Gestaltung für die Messung durch Laien präsentiert. Atmosphärische Aerosolpartikel stellen im globalen Maßstab ein gravierendes Problem für die menschliche Gesundheit dar, welches sich in Atemwegs- und Herz-Kreislauf-Erkrankungen äußert und eine Verkürzung der Lebenserwartung verursacht. Bisher wird Luftqualität ausschließlich anhand von Daten relativ weniger fester Messstellen beurteilt und mittels Modellen auf eine hohe räumliche Auflösung gebracht, so dass deren Repräsentativität für die flächendeckende Exposition der Bevölkerung ungeklärt bleibt. Es ist unmöglich, derartige räumliche Abbildungen mit den derzeitigen statischen Messnetzen zu bestimmen. Bei der gesundheitsbezogenen Bewertung von Schadstoffen geht der Trend daher stark zu räumlich differenzierenden Messungen. Ein vielversprechender Ansatz um eine hohe räumliche und zeitliche Abdeckung zu erreichen ist dabei Participatory Sensing, also die verteilte Messung durch Endanwender unter Zuhilfenahme ihrer persönlichen Endgeräte. Insbesondere für Luftqualitätsmessungen ergeben sich dabei eine Reihe von Herausforderungen - von neuer Sensorik, die kostengünstig und tragbar ist, über robuste Algorithmen zur Signalauswertung und Kalibrierung bis hin zu Anwendungen, die Laien bei der korrekten Ausführung von Messungen unterstützen und ihre Privatsphäre schützen. Diese Arbeit konzentriert sich auf das Anwendungsszenario Partizipatorischer Umweltmessungen, bei denen Smartphone-basierte Sensorik zum Messen der Umwelt eingesetzt wird und üblicherweise Laien die Messungen in relativ unkontrollierter Art und Weise ausführen. Die Hauptbeiträge hierzu sind: 1. Systeme zum Erfassen von Feinstaub mit Smartphones (Low-cost Sensorik und neue Hardware): Ausgehend von früher Forschung zur Feinstaubmessung mit kostengünstiger off-the-shelf-Sensorik wurde ein Sensorkonzept entwickelt, bei dem die Feinstaub-Messung mit Hilfe eines passiven Aufsatzes auf einer Smartphone-Kamera durchgeführt wird. Zur Beurteilung der Sensorperformance wurden teilweise Labor-Messungen mit künstlich erzeugtem Staub und teilweise Feldevaluationen in Ko-Lokation mit offiziellen Messstationen des Landes durchgeführt. 2. Algorithmen zur Signalverarbeitung und Auswertung: Im Zuge neuer Sensordesigns werden Kombinationen bekannter OpenCV-Bildverarbeitungsalgorithmen (Background-Subtraction, Contour Detection etc.) zur Bildanalyse eingesetzt. Der resultierende Algorithmus erlaubt im Gegensatz zur Auswertung von Lichtstreuungs-Summensignalen die direkte Zählung von Partikeln anhand individueller Lichtspuren. Ein zweiter neuartiger Algorithmus nutzt aus, dass es bei solchen Prozessen ein signalabhängiges Rauschen gibt, dessen Verhältnis zum Mittelwert des Signals bekannt ist. Dadurch wird es möglich, Signale die von systematischen unbekannten Fehlern betroffen sind auf Basis ihres Rauschens zu analysieren und das "echte" Signal zu rekonstruieren. 3. Algorithmen zur verteilten Kalibrierung bei gleichzeitigem Schutz der Privatsphäre: Eine Herausforderung partizipatorischer Umweltmessungen ist die wiederkehrende Notwendigkeit der Sensorkalibrierung. Dies beruht zum einen auf der Instabilität insbesondere kostengünstiger Luftqualitätssensorik und zum anderen auf der Problematik, dass Endbenutzern die Mittel für eine Kalibrierung üblicherweise fehlen. Bestehende Ansätze zur sogenannten Cross-Kalibrierung von Sensoren, die sich in Ko-Lokation mit einer Referenzstation oder anderen Sensoren befinden, wurden auf Daten günstiger Feinstaubsensorik angewendet sowie um Mechanismen erweitert, die eine Kalibrierung von Sensoren untereinander ohne Preisgabe privater Informationen (Identität, Ort) ermöglicht. 4. Mensch-Maschine-Interaktions-Gestaltungsrichtlinien für Participatory Sensing: Auf Basis mehrerer kleiner explorativer Nutzerstudien wurde empirisch eine Taxonomie der Fehler erstellt, die Laien beim Messen von Umweltinformationen mit Smartphones machen. Davon ausgehend wurden mögliche Gegenmaßnahmen gesammelt und klassifiziert. In einer großen summativen Studie mit einer hohen Teilnehmerzahl wurde der Effekt verschiedener dieser Maßnahmen durch den Vergleich vier unterschiedlicher Varianten einer App zur partizipatorischen Messung von Umgebungslautstärke evaluiert. Die dabei gefundenen Erkenntnisse bilden die Basis für Richtlinien zur Gestaltung effizienter Nutzerschnittstellen für Participatory Sensing auf Mobilgeräten. 5. Design Patterns für Participatory Sensing Games auf Mobilgeräten (Gamification): Ein weiterer erforschter Ansatz beschäftigt sich mit der Gamifizierung des Messprozesses um Nutzerfehler durch den Einsatz geeigneter Spielmechanismen zu minimieren. Dabei wird der Messprozess z.B. in ein Smartphone-Spiel (sog. Minigame) eingebettet, das im Hintergrund bei geeignetem Kontext die Messung durchführt. Zur Entwicklung dieses "Sensified Gaming" getauften Konzepts wurden Kernaufgaben im Participatory Sensing identifiziert und mit aus der Literatur zu sammelnden Spielmechanismen (Game Design Patterns) gegenübergestellt

Performance of the Intrac Wireless Activity Tracking System for the Afari Assistive Device

Afari is a mobility device that was designed to be more recreational, aesthetic, and functional outside than the typical mobility devices commonly used today such as walkers, crutches, and rollators. The Afari transfers weight from a user through the arm rests and enforces an upright posture while walking with correct adjustments to the arm rest height. In addition to assisting with walking or running, a sensor system fitted to the Afari device has been designed to analyze different aspects of activity tracking such as the dynamic loading applied to the arm rests, spatial-temporal gait parameters, speed, and distance. This includes various sensors, namely, load cells for each arm rest, an inertial measurement unit, and a speed and distance sensor that wirelessly transmit data via Bluetooth Low Energy (BLE) to either a smartphone or computer. The total distance, pitch angle, right and left loading on each armrest can be viewed in real time by the user. An algorithm was created in MATLAB to process all the raw data and compute cadence, stride length, average toe-off and heel strike angle, swing and stance time, and speed over the duration of active use. An Afari user can monitor these different aspects of their activity and adjust accordingly to potentially improve their balance or gait