Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.Comment: Accepted to 42nd International Conference on Software Engineerin

Mitigating security and privacy threats from untrusted application components on Android

Aufgrund von Androids datenzentrierter und Open-Source Natur sowie von fehlerhaften/bösartigen Apps durch das lockere Marktzulassungsverfahren, ist die Privatsphäre von Benutzern besonders gefährdet. Diese Dissertation präsentiert eine Reihe von Forschungsarbeiten, die die Bedrohung der Sicherheit/Privatsphäre durch nicht vertrauenswürdige Appkomponenten mindern. Die erste Arbeit stellt eine Compiler-basierte Kompartmentalisierungslösung vor, die Privilegientrennung nutzt, um eine starke Barriere zwischen der Host-App und Bibliothekskomponenten zu etablieren, und somit sensible Daten vor der Kompromittierung durch neugierige/bösartige Werbe-Bibliotheken schützt. Für fehleranfällige Bibliotheken von Drittanbietern implementieren wir in der zweiten Arbeit ein auf API-Kompatibilität basierendes Bibliothek-Update-Framework, das veraltete Bibliotheken durch Drop-Ins aktualisiert, um das durch Bibliotheken verursachte Zeitfenster der Verwundbarkeit zu minimieren. Die neueste Arbeit untersucht die missbräuchliche Nutzung von privilegierten Accessibility(a11y)-Funktionen in bösartigen Apps. Wir zeigen ein datenschutzfreundliches a11y-Framework, das die a11y-Logik wie eine Pipeline behandelt, die aus mehreren Modulen besteht, die in verschiedenen Sandboxen laufen. Weiterhin erzwingen wir eine Flusskontrolle über die Kommunikation zwischen den Modulen, wodurch die Angriffsfläche für den Missbrauch von a11y-APIs verringert wird, während die Vorteile von a11y erhalten bleiben.While Android’s data-intensive and open-source nature, combined with its less-than-strict market approval process, has allowed the installation of flawed and even malicious apps, its coarse-grained security model and update bottleneck in the app ecosystem make the platform’s privacy and security situation more worrying. This dissertation introduces a line of works that mitigate privacy and security threats from untrusted app components. The first work presents a compiler-based library compartmentalization solution that utilizes privilege separation to establish a strong trustworthy boundary between the host app and untrusted lib components, thus protecting sensitive user data from being compromised by curious or malicious ad libraries. While for vulnerable third-party libraries, we then build the second work that implements an API-compatibility-based library update framework using drop-in replacements of outdated libraries to minimize the open vulnerability window caused by libraries and we perform multiple dynamic tests and case studies to investigate its feasibility. Our latest work focuses on the misusing of powerful accessibility (a11y) features in untrusted apps. We present a privacy-enhanced a11y framework that treats the a11y logic as a pipeline composed of multiple modules running in different sandboxes. We further enforce flow control over the communication between modules, thus reducing the attack surface from abusing a11y APIs while preserving the a11y benefits

Afluentia: suporte à comunicação para afasia fluente

Aphasia is a language disorder caused by brain damage (eg, stroke) that affects a person’s ability to communicate. Involves different degrees of impairment and it can manifest by difficulties in speaking fluently or difficulty finding words (anomia), but can also entail impairment in spoken language comprehension, inability to repeat words or phrases, impairments in written expression (agraphia), in reading comprehension (alexia) or a combination of any of these difficulties. It can thus result in limitations in the way the person with aphasia interacts with others, for instance, to express how they are feeling as well as their needs, making it hard for them to have a more independent life or have their difficulties addressed. Additionally, this condition also has a strong impact in the life of those around them (e.g., family, carers) as the difficulties of communication, should anything happen, can lead to fear of leaving these patients unattended. Several challenges arise when addressing the communication needs of people with aphasia deriving from the diverse and idiosyncratic nature of their condition and although assisted communication tools have been proposed in the literature (e.g. using pictograms), the characteristics of aphasia often render them as partial solutions. In this sense, this project focuses on understanding the characteristics and needs of aphasic patients and also on the proposal of technology-mediated communication tools that address them in their daily lives. This work adopts a user-centered design and development approach to explore how people with aphasia can be supported in their day-today communication resorting to technology mediation. It was thus achieving an iterative design with the development and evaluation of a proof-of-concept solution for communication aspects, which was progressively implemented and refined having in consideration the identified requirements and the continuous evaluation of the proposed solutions, carried out with a focus group composed by a Speech and Language Therapist (SLT) and a Human Computer Interaction (HCI) Expert. After a first version of the system was achieved, an evaluation phase with Speech and Language Therapists with a strong experience with patients with aphasia took place in order to understand and validate the achieved application, which led to more refinement phases. At its current stage of development, evaluation results show a good level of usability and satisfaction and establish Afluentia as promising ground for further evolving the research on communication mediated by technology to support people with aphasia.A afasia é um distúrbio da linguagem provocada por danos cerebrais (por exemplo, acidente vascular cerebral) e que afeta a capacidade de comunicação de uma pessoa. Envolve diferentes graus de deficiência e pode-se manifestar por dificuldades em falar fluentemente ou dificuldade em encontrar palavras (anomia), mas também pode acarretar prejuízo na compreensão da linguagem falada, incapacidade de repetir palavras ou frases, deficiências na expressão escrita (agrafia), na compreensão da leitura (alexia) ou numa combinação de qualquer uma dessas dificuldades. Pode assim resultar em limitações na forma como a pessoa com afasia interage com outras para, por exemplo, exprimir como se está a sentir assim como as suas necessidades, impossibilitando que tenha uma vida mais independente ou tenha as suas dificuldades abordadas. Além disso, a afasia também tem um forte impacto na vida das pessoas ao redor do afásico (por exemplo, familiares, cuidadores), pois as dificuldades de comunicação da pessoa com afasia podem levar ao medo de deixála desacompanhada por aqueles que a rodeiam. Vários desafios surgem ao abordar as necessidades de comunicação de pessoas com afasia decorrentes da natureza diversa e idiossincrática da sua condição e embora ferramentas de comunicação assistida tenham sido propostas na literatura (por exemplo, usando pictogramas), as características da afasia geralmente tornam essas soluções parciais. Nesse sentido, este projeto foca-se na compreensão das características e necessidades dos pacientes afásicos e também na proposta de ferramentas de comunicação mediada por tecnologia que os abordem, no seu cotidiano. Este trabalho adota uma abordagem de design e desenvolvimento centrada no utilizador de modo explorar como as pessoas com afasia podem ser apoiadas na sua comunicação quotidiana recorrendo à mediação tecnológica. Foi assim conseguindo um design iterativo com desenvolvimento e avaliação de uma solução de prova de conceito para aspectos de comunicação, que foi progressivamente implementada e aperfeiçoada tendo em consideração os requisitos identificados e a avaliação contínua das soluções propostas, realizada com um grupo de foco composto por uma Terapeuta da Fala e um Especialista em Interacção Humano Computador. Após a obtenção de uma primeira versão do sistema, uma fase de avaliação com Terapeutas da Fala com forte experiência com pacientes com afasia também ocorreu de modo a entender e validar a aplicação alcançada, o que levou a mais fases de refinamento. Na sua atual fase de desenvolvimento, os resultados da avaliação mostram um bom nível de usabilidade e satisfação e definem o Afluentia como um terreno promissor para evoluir ainda mais a pesquisa em comunicação mediada por tecnologia de suporte a pessoas com afasia.Mestrado em Engenharia de Computadores e Telemátic

A framework to develop adaptive multimodal dialog systems for Android-based mobile devices

Proceedings of: 9th International Conference (HAIS 2014), Salamanca, Spain, June 11-13, 2014Mobile devices programming has emerged as a new trend in software development. The main developers of operating systems for such devices have provided APIs for developers to implement their own applications, including different solutions for developing voice control. Android, the most popular alternative among developers, offers libraries to build interfaces including different resources for graphical layouts as well as speech recognition and text-to-speech synthesis. Despite the usefulness of such classes, there are no strategies defined for multimodal interface development for Android systems, and developers create ad-hoc solutions that make apps costly to implement and difficult to compare and maintain. In this paper we propose a framework to facilitate the software engineering life cycle for multimodal interfaces in Android. Our proposal integrates the facilities of the Android API in a modular architecture that emphasizes interaction management and context-awareness to build sophisticated, robust and maintainable applications.This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485)

e-mission: an open source, extensible platform for human mobility systems

Transportation is the single largest source of carbon emissions in the US. Decarbonizing it is challenging because it depends on individual behaviors, which in turn, depend on local land use planning. The interdisciplinary field of Computational Mobility, focusing on collecting, analysing and influencing human travel behavior, can frame solutions to this challenge.Innovation flows in interdisciplinary fields are bi-directional. The flow to the domain is focused on building a strong foundation for methodological improvements. As the improvements are deployed, they result in use-inspired computational research. This temporal dependency results in our initial focus on the modularity, accuracy and reproducibility of e-mission, an extensible platform for instrumenting human mobility. This open source platform has a modular architecture that supports power efficient duty cycling using virtual sensors, a read-only data model and a pipeline with novel algorithm adaptations for smartphone sensing.We also perform the first empirical evaluations of smartphone-based platforms in this domain. The architectural evaluation is based on three real world deployments: a classic travel diary, a crowdsourcing initiative, and a behavioral study. The accuracy evaluation is based on an novel procedure that uses artificial trips and multiple parallel phones to mitigate concerns over privacy, context sensitive power consumption and inherent sensing error. Data collected from three artifical timelines was used to evaluate the trajectory, segmentation and classification accuracies vs. power for various configurations.On computational side, challenges derived from the deployments can contribute to ongoing CS research in privacy, trustworthiness, incentivization and decision making. On the mobility side, this enables methodological innovations such as Agile Urban Planning for prototyping infrastructure changes

Context-driven progressive enhancement of mobile web applications: a multicriteria decision-making approach

Personal computing has become all about mobile and embedded devices. As a result, the adoption rate of smartphones is rapidly increasing and this trend has set a need for mobile applications to be available at anytime, anywhere and on any device. Despite the obvious advantages of such immersive mobile applications, software developers are increasingly facing the challenges related to device fragmentation. Current application development solutions are insufficiently prepared for handling the enormous variety of software platforms and hardware characteristics covering the mobile eco-system. As a result, maintaining a viable balance between development costs and market coverage has turned out to be a challenging issue when developing mobile applications. This article proposes a context-aware software platform for the development and delivery of self-adaptive mobile applications over the Web. An adaptive application composition approach is introduced, capable of autonomously bypassing context-related fragmentation issues. This goal is achieved by incorporating and validating the concept of fine-grained progressive application enhancements based on a multicriteria decision-making strategy