Continuous Authentication for Voice Assistants

Voice has become an increasingly popular User Interaction (UI) channel, mainly contributing to the ongoing trend of wearables, smart vehicles, and home automation systems. Voice assistants such as Siri, Google Now and Cortana, have become our everyday fixtures, especially in scenarios where touch interfaces are inconvenient or even dangerous to use, such as driving or exercising. Nevertheless, the open nature of the voice channel makes voice assistants difficult to secure and exposed to various attacks as demonstrated by security researchers. In this paper, we present VAuth, the first system that provides continuous and usable authentication for voice assistants. We design VAuth to fit in various widely-adopted wearable devices, such as eyeglasses, earphones/buds and necklaces, where it collects the body-surface vibrations of the user and matches it with the speech signal received by the voice assistant's microphone. VAuth guarantees that the voice assistant executes only the commands that originate from the voice of the owner. We have evaluated VAuth with 18 users and 30 voice commands and find it to achieve an almost perfect matching accuracy with less than 0.1% false positive rate, regardless of VAuth's position on the body and the user's language, accent or mobility. VAuth successfully thwarts different practical attacks, such as replayed attacks, mangled voice attacks, or impersonation attacks. It also has low energy and latency overheads and is compatible with most existing voice assistants

The development of a SmartAbility Framework to enhance multimodal interaction for people with reduced physical ability.

Assistive technologies are an evolving market due to the number of people worldwide who have conditions resulting in reduced physical ability (also known as disability). Various classification schemes exist to categorise disabilities, as well as government legislations to ensure equal opportunities within the community. However, there is a notable absence of a process to map physical conditions to technologies in order to improve Quality of Life for this user group. This research is characterised primarily under the Human Computer Interaction (HCI) domain, although aspects of Systems of Systems (SoS) and Assistive Technologies have been applied. The thesis focuses on examples of multimodal interactions leading to the development of a SmartAbility Framework that aims to assist people with reduced physical ability by utilising their abilities to suggest interaction mediums and technologies. The framework was developed through a predominantly Interpretivism methodology approach consisting of a variety of research methods including state- of-the-art literature reviews, requirements elicitation, feasibility trials and controlled usability evaluations to compare multimodal interactions. The developed framework was subsequently validated through the involvement of the intended user community and domain experts and supported by a concept demonstrator incorporating the SmartATRS case study. The aim and objectives of this research were achieved through the following key outputs and findings: - A comprehensive state-of-the-art literature review focussing on physical conditions and their classifications, HCI concepts relevant to multimodal interaction (Ergonomics of human-system interaction, Design For All and Universal Design), SoS definition and analysis techniques involving System of Interest (SoI), and currently-available products with potential uses as assistive technologies. - A two-phased requirements elicitation process applying surveys and semi-structured interviews to elicit the daily challenges for people with reduced physical ability, their interests in technology and the requirements for assistive technologies obtained through collaboration with a manufacturer. - Findings from feasibility trials involving monitoring brain activity using an electroencephalograph (EEG), tracking facial features through Tracking Learning Detection (TLD), applying iOS Switch Control to track head movements and investigating smartglasses. - Results of controlled usability evaluations comparing multimodal interactions with the technologies deemed to be feasible from the trials. The user community of people with reduced physical ability were involved during the process to maximise the usefulness of the data obtained. - An initial SmartDisability Framework developed from the results and observations ascertained through requirements elicitation, feasibility trials and controlled usability evaluations, which was validated through an approach of semi-structured interviews and a focus group. - An enhanced SmartAbility Framework to address the SmartDisability validation feedback by reducing the number of elements, using simplified and positive terminology and incorporating concepts from Quality Function Deployment (QFD). - A final consolidated version of the SmartAbility Framework that has been validated through semi-structured interviews with additional domain experts and addressed all key suggestions. The results demonstrated that it is possible to map technologies to people with physical conditions by considering the abilities that they can perform independently without external support and the exertion of significant physical effort. This led to a realisation that the term ‘disability’ has a negative connotation that can be avoided through the use of the phrase ‘reduced physical ability’. It is important to promote this rationale to the wider community, through exploitation of the framework. This requires a SmartAbility smartphone application to be developed that allows users to input their abilities in order for recommendations of interaction mediums and technologies to be provided

Development and Evaluation of Tongue Operated Robotic Rehabilitation Paradigm for Stroke Survivors with Upper Limb Paralysis

Stroke is a devastating condition that may cause upper limb paralysis. Robotic rehabilitation with self-initiated and assisted movements is a promising technology that could help restore upper limb function. The objective of this research is to develop and evaluate a tongue-operated exoskeleton that will harness the intention of stroke survivors with upper limb paralysis via tongue motion to control robotic exoskeleton during rehabilitation to achieve functional restoration and improve quality of life. Specifically, a tongue operated assistive technology called the Tongue Drive System is used to harness the tongue gesture to generate commands. And, the generated command is used to control rehabilitation robot such as wrist-based exoskeleton Hand Mentor ProTM (HM) and upper limb-based exoskeleton KINARMTM. Through a pilot experiment with 3 healthy participants, we have demonstrated the functionality of an enhanced TDS-HM with pressure-sensing capability. The system can add a programmable load force to increase the exercise intensity in isotonic mode. Through experiments with healthy and stroke subjects, we have demonstrated that the TDS-KINARM system could accurately translate tongue commands to exoskeleton arm movements, quantify function of the upper limb and perform rehabilitation training. Specifically, all healthy subjects and stroke survivors successfully performed target reaching and tracking tasks in all control modes. One of the stroke patients showed clinically significant improvement. We also analyzed the arm reaching kinematics of healthy subjects in 4 modes (active, active viscous, discrete tongue, and proportional tongue) of TDS-KINARM operation. The results indicated that the proportional tongue mode was a better candidate than the discrete tongue mode for the tongue assisted rehabilitation. This study also provided initial insights into possible kinematic similarities between tongue-operated and voluntary arm movements. Furthermore, the results showed that the viscous resistance to arm motion did not affect kinematics of arm reaching movements significantly. Finally, through a 6 healthy subject experiment, we observed a tendency of a facilitatory effect of adding tongue movement to limb movement on event-related desynchronization in EEG, implying enhanced brain excitability. This effect may contribute to enhanced rehabilitation outcome in stroke survivors using TDS with motor rehabilitation.Ph.D

SCALING ARTIFICIAL INTELLIGENCE IN ENDOSCOPY: FROM MODEL DEVELOPMENT TO MACHINE LEARNING OPERATIONS FRAMEWORKS

Questa tesi esplora l'integrazione dell'intelligenza artificiale (IA) in Otorinolaringoiatria – Chirurgia di Testa e Collo, concentrandosi sui progressi della computer vision per l’endoscopia e le procedure chirurgiche. La ricerca inizia con una revisione completa dello stato dell’arte dell'IA e della computer vision in questo campo, identificando aree per ulteriori sviluppi. L'obiettivo principale è stato quello di sviluppare un sistema di computer vision per l'analisi di immagini e video endoscopici. La ricerca ha coinvolto la progettazione di strumenti per la rilevazione e segmentazione di neoplasie nelle vie aerodigestive superiori (VADS) e la valutazione della motilità delle corde vocali, cruciale nella stadiazione del carcinoma laringeo. Inoltre, lo studio si è focalizzato sul potenziale dei foundation vision models, vision transformers basati su self-supervised learning, per ridurre la necessità di annotazione da parte di esperti, approccio particolarmente vantaggioso in campi con dati limitati. Inoltre, la ricerca ha incluso lo sviluppo di un'applicazione web per migliorare e velocizzare il processo di annotazione in endoscopia delle VADS, nell’ambito generale delle tecniche di MLOps. La tesi copre varie fasi della ricerca, a partire dalla definizione del quadro concettuale e della metodologia, denominata "Videomics". Include una revisione della letteratura sull'IA in endoscopia clinica, focalizzata sulla Narrow Band Imaging (NBI) e sulle reti neurali convoluzionali (CNN). Lo studio progredisce attraverso diverse fasi, dalla valutazione della qualità delle immagini endoscopiche alla caratterizzazione approfondita delle lesioni neoplastiche. Si affronta anche la necessità di standard nel reporting degli studi di computer vision in ambito medico e si valuta l'applicazione dell'IA in setting dinamici come la motilità delle corde vocali. Una parte significativa della ricerca indaga l'uso di algoritmi di computer vision generalizzati (“foundation models”) e la “commoditization” degli algoritmi di machine learning, utilizzando polipi nasali e il carcinoma orofaringeo come casi studio. Infine, la tesi discute lo sviluppo di ENDO-CLOUD, un sistema basato su cloud per l’analisi della videolaringoscopia, evidenziando le sfide e le soluzioni nella gestione dei dati e l’utilizzo su larga scala di modelli di IA nell'imaging medico.This thesis explores the integration of artificial intelligence (AI) in Otolaryngology – Head and Neck Surgery, focusing on advancements in computer vision for endoscopy and surgical procedures. It begins with a comprehensive review of AI and computer vision advancements in this field, identifying areas for further exploration. The primary aim was to develop a computer vision system for endoscopy analysis. The research involved designing tools for detecting and segmenting neoplasms in the upper aerodigestive tract (UADT) and assessing vocal fold motility, crucial in laryngeal cancer staging. Further, the study delves into the potential of vision foundation models, like vision transformers trained via self-supervision, to reduce the need for expert annotations, particularly beneficial in fields with limited cases. Additionally, the research includes the development of a web application for enhancing and speeding up the annotation process in UADT endoscopy, under the umbrella of Machine Learning Operations (MLOps). The thesis covers various phases of research, starting with defining the conceptual framework and methodology, termed "Videomics". It includes a literature review on AI in clinical endoscopy, focusing on Narrow Band Imaging (NBI) and convolutional neural networks (CNNs). The research progresses through different stages, from quality assessment of endoscopic images to in-depth characterization of neoplastic lesions. It also addresses the need for standards in medical computer vision study reporting and evaluates the application of AI in dynamic vision scenarios like vocal fold motility. A significant part of the research investigates the use of "general purpose" vision algorithms and the commoditization of machine learning algorithms, using nasal polyps and oropharyngeal cancer as case studies. Finally, the thesis discusses the development of ENDO-CLOUD, a cloud-based system for videolaryngoscopy, highlighting the challenges and solutions in data management and the large-scale deployment of AI models in medical imaging

A Systematic Review of Urban Navigation Systems for Visually Impaired People

Blind and Visually impaired people (BVIP) face a range of practical difficulties when undertaking outdoor journeys as pedestrians. Over the past decade, a variety of assistive devices have been researched and developed to help BVIP navigate more safely and independently. In~addition, research in overlapping domains are addressing the problem of automatic environment interpretation using computer vision and machine learning, particularly deep learning, approaches. Our aim in this article is to present a comprehensive review of research directly in, or relevant to, assistive outdoor navigation for BVIP. We breakdown the navigation area into a series of navigation phases and tasks. We then use this structure for our systematic review of research, analysing articles, methods, datasets and current limitations by task. We also provide an overview of commercial and non-commercial navigation applications targeted at BVIP. Our review contributes to the body of knowledge by providing a comprehensive, structured analysis of work in the domain, including the state of the art, and guidance on future directions. It will support both researchers and other stakeholders in the domain to establish an informed view of research progress

Adaptive threshold optimisation for colour-based lip segmentation in automatic lip-reading systems

A thesis submitted to the Faculty of Engineering and the Built Environment, University of the Witwatersrand, Johannesburg, in ful lment of the requirements for the degree of Doctor of Philosophy. Johannesburg, September 2016Having survived the ordeal of a laryngectomy, the patient must come to terms with the resulting loss of speech. With recent advances in portable computing power, automatic lip-reading (ALR) may become a viable approach to voice restoration. This thesis addresses the image processing aspect of ALR, and focuses three contributions to colour-based lip segmentation. The rst contribution concerns the colour transform to enhance the contrast between the lips and skin. This thesis presents the most comprehensive study to date by measuring the overlap between lip and skin histograms for 33 di erent colour transforms. The hue component of HSV obtains the lowest overlap of 6:15%, and results show that selecting the correct transform can increase the segmentation accuracy by up to three times. The second contribution is the development of a new lip segmentation algorithm that utilises the best colour transforms from the comparative study. The algorithm is tested on 895 images and achieves percentage overlap (OL) of 92:23% and segmentation error (SE) of 7:39 %. The third contribution focuses on the impact of the histogram threshold on the segmentation accuracy, and introduces a novel technique called Adaptive Threshold Optimisation (ATO) to select a better threshold value. The rst stage of ATO incorporates -SVR to train the lip shape model. ATO then uses feedback of shape information to validate and optimise the threshold. After applying ATO, the SE decreases from 7:65% to 6:50%, corresponding to an absolute improvement of 1:15 pp or relative improvement of 15:1%. While this thesis concerns lip segmentation in particular, ATO is a threshold selection technique that can be used in various segmentation applications.MT201

Novel Bidirectional Body - Machine Interface to Control Upper Limb Prosthesis

Objective. The journey of a bionic prosthetic user is characterized by the opportunities and limitations involved in adopting a device (the prosthesis) that should enable activities of daily living (ADL). Within this context, experiencing a bionic hand as a functional (and, possibly, embodied) limb constitutes the premise for mitigating the risk of its abandonment through the continuous use of the device. To achieve such a result, different aspects must be considered for making the artificial limb an effective support for carrying out ADLs. Among them, intuitive and robust control is fundamental to improving amputees’ quality of life using upper limb prostheses. Still, as artificial proprioception is essential to perceive the prosthesis movement without constant visual attention, a good control framework may not be enough to restore practical functionality to the limb. To overcome this, bidirectional communication between the user and the prosthesis has been recently introduced and is a requirement of utmost importance in developing prosthetic hands. Indeed, closing the control loop between the user and a prosthesis by providing artificial sensory feedback is a fundamental step towards the complete restoration of the lost sensory-motor functions. Within my PhD work, I proposed the development of a more controllable and sensitive human-like hand prosthesis, i.e., the Hannes prosthetic hand, to improve its usability and effectiveness. Approach. To achieve the objectives of this thesis work, I developed a modular and scalable software and firmware architecture to control the Hannes prosthetic multi-Degree of Freedom (DoF) system and to fit all users’ needs (hand aperture, wrist rotation, and wrist flexion in different combinations). On top of this, I developed several Pattern Recognition (PR) algorithms to translate electromyographic (EMG) activity into complex movements. However, stability and repeatability were still unmet requirements in multi-DoF upper limb systems; hence, I started by investigating different strategies to produce a more robust control. To do this, EMG signals were collected from trans-radial amputees using an array of up to six sensors placed over the skin. Secondly, I developed a vibrotactile system to implement haptic feedback to restore proprioception and create a bidirectional connection between the user and the prosthesis. Similarly, I implemented an object stiffness detection to restore tactile sensation able to connect the user with the external word. This closed-loop control between EMG and vibration feedback is essential to implementing a Bidirectional Body - Machine Interface to impact amputees’ daily life strongly. For each of these three activities: (i) implementation of robust pattern recognition control algorithms, (ii) restoration of proprioception, and (iii) restoration of the feeling of the grasped object's stiffness, I performed a study where data from healthy subjects and amputees was collected, in order to demonstrate the efficacy and usability of my implementations. In each study, I evaluated both the algorithms and the subjects’ ability to use the prosthesis by means of the F1Score parameter (offline) and the Target Achievement Control test-TAC (online). With this test, I analyzed the error rate, path efficiency, and time efficiency in completing different tasks. Main results. Among the several tested methods for Pattern Recognition, the Non-Linear Logistic Regression (NLR) resulted to be the best algorithm in terms of F1Score (99%, robustness), whereas the minimum number of electrodes needed for its functioning was determined to be 4 in the conducted offline analyses. Further, I demonstrated that its low computational burden allowed its implementation and integration on a microcontroller running at a sampling frequency of 300Hz (efficiency). Finally, the online implementation allowed the subject to simultaneously control the Hannes prosthesis DoFs, in a bioinspired and human-like way. In addition, I performed further tests with the same NLR-based control by endowing it with closed-loop proprioceptive feedback. In this scenario, the results achieved during the TAC test obtained an error rate of 15% and a path efficiency of 60% in experiments where no sources of information were available (no visual and no audio feedback). Such results demonstrated an improvement in the controllability of the system with an impact on user experience. Significance. The obtained results confirmed the hypothesis of improving robustness and efficiency of a prosthetic control thanks to of the implemented closed-loop approach. The bidirectional communication between the user and the prosthesis is capable to restore the loss of sensory functionality, with promising implications on direct translation in the clinical practice

Secure and Usable User Authentication

Authentication is a ubiquitous task in users\u27 daily lives. The dominant form of user authentication are text passwords. They protect private accounts like online banking, gaming, and email, but also assets in organisations. Yet, many issues are associated with text passwords, leading to challenges faced by both, users and organisations. This thesis contributes to the body of research enabling secure and usable user authentication, benefiting both, users and organisations. To that end, it addresses three distinct challenges. The first challenge addressed in this thesis is the creation of correct, complete, understandable, and effective password security awareness materials. To this end, a systematic process for the creation of awareness materials was developed and applied to create a password security awareness material. This process comprises four steps. First, relevant content for an initial version is aggregated (i.e. descriptions of attacks on passwords and user accounts, descriptions of defences to these attacks, and common misconceptions about password and user account security). Then, feedback from information security experts is gathered to ensure the correctness and completeness of the awareness material. Thereafter, feedback from lay-users is gathered to ensure the understandability of the awareness material. Finally, a formal evaluation of the awareness material is conducted to ensure its effectiveness (i.e. whether the material improves participant\u27s ability to assess the security of passwords as well as password-related behaviour and decreases the prevalence of common misconceptions about password and user account security). The results of the evaluation show the effectiveness of the awareness material: it significantly improved the participants\u27 ability to assess the security of password-related behaviour as well as passwords and significantly decreased the prevalence of misconceptions about password and user account security. The second challenge addressed in this thesis is shoulder-surfing resistant text password entry with gamepads (as an example of very constrained input devices) in shared spaces. To this end, the very first investigation of text password entry with gamepads is conducted. First, the requirements of authentication in the gamepad context are described. Then, these requirements are applied to assess schemes already deployed in the gamepad context and shoulder-surfing resistant authentication schemes from the literature proposed for non-gamepad contexts. The results of this assessment show that none of the currently deployed and only four of the proposals in the literature fulfil all requirements. Furthermore, the results of the assessment also indicate a need for an empirical evaluation in order to exactly gauge the shoulder-surfing threat in the gamepad context and compare alternatives to the incumbent on-screen keyboard. Based on these results, two user studies (one online study and one lab study) are conducted to investigate the shoulder-surfing resistance and usability of three authentication schemes in the gamepad context: the on-screen keyboard (as de-facto standard in this context), the grid-based scheme (an existing proposal from the literature identified as the most viable candidate adaptable to the gamepad context during the assessment), and Colorwheels (a novel shoulder-surfing resistant authentication scheme specifically designed for the gamepad context). The results of these two user studies show that on-screen keyboards are highly susceptible to opportunistic shoulder-surfing, but also show the most favourable usability properties among the three schemes. Colorwheels offers the most robust shoulder-surfing resistance and scores highest with respect to participants\u27 intention to use it in the future, while showing more favourable usability results than the grid-based scheme. The third challenge addressed in this thesis is secure and efficient storage of passwords in portfolio authentication schemes. Portfolio authentication is used to counter capture attacks such as shoulder-surfing or eavesdropping on network traffic. While usability studies of portfolio authentication schemes showed promising results, a verification scheme which allows secure and efficient storage of the portfolio authentication secret had been missing until now. To remedy this problem, the (t,n)-threshold verification scheme is proposed. It is based on secret sharing and key derivation functions. The security as well as the efficiency properties of two variants of the scheme (one based on Blakley secret sharing and one based on Shamir secret sharing) are evaluated against each other and against a naive approach. These evaluations show that the two (t,n)-threshold verification scheme variants always exhibit more favourable properties than the naive approach and that when deciding between the two variants, the exact application scenario must be considered. Three use cases illustrate as exemplary application scenarios the versatility of the proposed (t,n)-threshold verification scheme. By addressing the aforementioned three distinct challenges, this thesis demonstrates the breadth of the field of usable and secure user authentication ranging from awareness materials, to the assessment and evaluation of authentication schemes, to applying cryptography to craft secure password storage solutions. The research processes, results, and insights described in this thesis represent important and meaningful contributions to the state of the art in the research on usable and secure user authentication, offering benefits for users, organisations, and researchers alike

Application of Smartphone Photography and 3G Wireless Internet Technology in Free Flap Monitoring: A Prospective Study

AIMS OF THE STUDY: 1. To study the role of Smartphone photography and 3G wireless technology for monitoring free flap. OBJECTIVES: 1. To study the feasibility of Smartphone photography in free flap monitoring. 2. To study the feasibility of Smartphone photography and 3G wireless internet technology in monitoring free flaps. MATERIALS AND METHODS: The feasibility of using Smartphone technology and 3G wireless internet technology as an adjunct in free flap monitoring in patients with oral cavity malignancy who have undergone microvascular free flap reconstruction with a visible skin paddle for post excisional defects was studied . A prospective study was conducted from November 2012 to September 2014. A standardized color card was used for the assessment of photographs. The principal investigator (P.I) was involved in taking photographs every six hours and send it to three surgeons using 3G wireless internet technology. The three surgeons: the Operating Surgeon(OPS), who was monitoring the free flaps using clinical data and photograph of the flap; the Observing Surgeon1(ObS1), who monitored using photographs only; and the Observing Surgeon 2(ObS2), who monitored using only the clinical data of the free flap. Inter-observer variability and accuracy rate of each observer in the assessment of the free flap status was used for analysis. The decision on re-exploration of the free flap was made by the operating surgeon although the input from the other two surgeons (ObS1 &ObS2) was provided to the OPS if the free flap viability was questionable. RESULTS: A total of 18 cases were analyzed from January 2013 to September 2014, with 100% free flap survival rate. Five patients were re-explored and salvaged completely. The indication for operation was neck hematoma in 4 patients and venous thrombosis in 1 patient. In the last patient, venous congestion was identified by photograph and later on clinical grounds. The accuracy rate with the use of photographs was 100%. CONCLUSION: In this study, the Smartphone photography with 3G internet technology prove to be a useful adjunct in free flap monitoring with a success rate of 100%. There was no free flap failure during the study period. The technique was useful in identifying a potential venous thrombosis which helped in decision tore explore and revise the microvascular anastomosis. The incorporation this technique in the current protocols of free flap monitoring may help identify impending flap failures even before the assessment by the senior team member

DETECTION OF HEALTH-RELATED BEHAVIOURS USING HEAD-MOUNTED DEVICES

The detection of health-related behaviors is the basis of many mobile-sensing applications for healthcare and can trigger other inquiries or interventions. Wearable sensors have been widely used for mobile sensing due to their ever-decreasing cost, ease of deployment, and ability to provide continuous monitoring. In this dissertation, we develop a generalizable approach to sensing eating-related behavior. First, we developed Auracle, a wearable earpiece that can automatically detect eating episodes. Using an off-the-shelf contact microphone placed behind the ear, Auracle captures the sound of a person chewing as it passes through the head. This audio data is then processed by a custom circuit board. We collected data with 14 participants for 32 hours in free-living conditions and achieved accuracy exceeding 92.8% and F1 score exceeding77.5% for eating detection with 1-minute resolution. Second, we adapted Auracle for measuring children’s eating behavior, and improved the accuracy and robustness of the eating-activity detection algorithms. We used this improved prototype in a laboratory study with a sample of 10 children for 60 total sessions and collected 22.3 hours of data in both meal and snack scenarios. Overall, we achieved 95.5% accuracy and 95.7% F1 score for eating detection with 1-minute resolution. Third, we developed a computer-vision approach for eating detection in free-living scenarios. Using a miniature head-mounted camera, we collected data with 10 participants for about 55 hours. The camera was fixed under the brim of a cap, pointing to the mouth of the wearer and continuously recording video (but not audio) throughout their normal daily activity. We evaluated performance for eating detection using four different Convolutional Neural Network (CNN) models. The best model achieved 90.9% accuracy and 78.7%F1 score for eating detection with 1-minute resolution. Finally, we validated the feasibility of deploying the 3D CNN model in wearable or mobile platforms when considering computation, memory, and power constraints