StoryDroid: Automated Generation of Storyboard for Android Apps

Mobile apps are now ubiquitous. Before developing a new app, the development team usually endeavors painstaking efforts to review many existing apps with similar purposes. The review process is crucial in the sense that it reduces market risks and provides inspiration for app development. However, manual exploration of hundreds of existing apps by different roles (e.g., product manager, UI/UX designer, developer) in a development team can be ineffective. For example, it is difficult to completely explore all the functionalities of the app in a short period of time. Inspired by the conception of storyboard in movie production, we propose a system, StoryDroid, to automatically generate the storyboard for Android apps, and assist different roles to review apps efficiently. Specifically, StoryDroid extracts the activity transition graph and leverages static analysis techniques to render UI pages to visualize the storyboard with the rendered pages. The mapping relations between UI pages and the corresponding implementation code (e.g., layout code, activity code, and method hierarchy) are also provided to users. Our comprehensive experiments unveil that StoryDroid is effective and indeed useful to assist app development. The outputs of StoryDroid enable several potential applications, such as the recommendation of UI design and layout code

Code clone detection in obfuscated Android apps

The Android operating system has long become one of the main global smartphone operating systems. Both developers and malware authors often reuse code to expedite the process of creating new apps and malware samples. Code cloning is the most common way of reusing code in the process of developing Android apps. Finding code clones through the analysis of Android binary code is a challenging task that becomes more sophisticated when instances of code reuse are non-contiguous, reordered, or intertwined with other code. We introduce an approach for detecting cloned methods as well as small and non-contiguous code clones in obfuscated Android applications by simulating the execution of Android apps and then analyzing the subsequent execution traces. We first validate our approach’s ability on finding different types of code clones on 20 injected clones. Next we validate the resistance of our approach against obfuscation by comparing its results on a set of 1085 apps before and after code obfuscation. We obtain 78-87% similarity between the finding from non-obfuscated applications and four sets of obfuscated applications. We also investigated the presence of code clones among 1603 Android applications. We were able to find 44,776 code clones where 34% of code clones were seen from different applications and the rest are among different versions of an application. We also performed a comparative analysis between the clones found by our approach and the clones detected by Nicad on the source code of applications. Finally, we show a practical application of our approach for detecting variants of Android banking malware. Among 60,057 code clone clusters that are found among a dataset of banking malware, 92.9% of them were unique to one malware family or benign applications

Automated Testing and Bug Reproduction of Android Apps

The large demand of mobile devices creates significant concerns about the quality of mobile applications (apps). The corresponding increase in app complexity has made app testing and maintenance activities more challenging. During app development phase, developers need to test the app in order to guarantee its quality before releasing it to the market. During the deployment phase, developers heavily rely on bug reports to reproduce failures reported by users. Because of the rapid releasing cycle of apps and limited human resources, it is difficult for developers to manually construct test cases for testing the apps or diagnose failures from a large number of bug reports. However, existing automated test case generation techniques are ineffective in exploring most effective events that can quickly improve code coverage and fault detection capability. In addition, none of existing techniques can reproduce failures directly from bug reports. This dissertation provides a framework that employs artifact intelligence (AI) techniques to improve testing and debugging of mobile apps. Specifically, the testing approach employs a Q-network that learns a behavior model from a set of existing apps and the learned model can be used to explore and generate tests for new apps. The framework is able to capture the fine-grained details of GUI events (e.g., visiting times of events, text on the widgets) and use them as features that are fed into a deep neural network, which acts as the agent to guide the app exploration. The debugging approach focuses on automatically reproducing crashes from bug reports for mobile apps. The approach uses a combination of natural language processing (NLP), deep learning, and dynamic GUI exploration to synthesize event sequences with the goal of reproducing the reported crash

Understanding Android Obfuscation Techniques: A Large-Scale Investigation in the Wild

In this paper, we seek to better understand Android obfuscation and depict a holistic view of the usage of obfuscation through a large-scale investigation in the wild. In particular, we focus on four popular obfuscation approaches: identifier renaming, string encryption, Java reflection, and packing. To obtain the meaningful statistical results, we designed efficient and lightweight detection models for each obfuscation technique and applied them to our massive APK datasets (collected from Google Play, multiple third-party markets, and malware databases). We have learned several interesting facts from the result. For example, malware authors use string encryption more frequently, and more apps on third-party markets than Google Play are packed. We are also interested in the explanation of each finding. Therefore we carry out in-depth code analysis on some Android apps after sampling. We believe our study will help developers select the most suitable obfuscation approach, and in the meantime help researchers improve code analysis systems in the right direction

Synesthesia: Detecting Screen Content via Remote Acoustic Side Channels

We show that subtle acoustic noises emanating from within computer screens can be used to detect the content displayed on the screens. This sound can be picked up by ordinary microphones built into webcams or screens, and is inadvertently transmitted to other parties, e.g., during a videoconference call or archived recordings. It can also be recorded by a smartphone or "smart speaker" placed on a desk next to the screen, or from as far as 10 meters away using a parabolic microphone. Empirically demonstrating various attack scenarios, we show how this channel can be used for real-time detection of on-screen text, or users' input into on-screen virtual keyboards. We also demonstrate how an attacker can analyze the audio received during video call (e.g., on Google Hangout) to infer whether the other side is browsing the web in lieu of watching the video call, and which web site is displayed on their screen

Ontological Engineering For Source Code Generation

Source Code Generation (SCG) is the sub-domain of the Automatic Programming (AP) that helps programmers to program using high-level abstraction. Recently, many researchers investigated many techniques to access SCG. The problem is to use the appropriate technique to generate the source code due to its purposes and the inputs. This paper introduces a review and an analysis related SCG techniques. Moreover, comparisons are presented for: techniques mapping, Natural Language Processing (NLP), knowledge base, ontology, Specification Configuration Template (SCT) model and deep learnin

Aplicação para smartphone ’Practice As You Walk’: mobile learning e gamification em ensaio coral

With the worldwide massification of mobile devices, the use of technology for pedagogical purposes in the context of music learning has proven to be an indispensable tool for ensuring motivation among students. By exploring the implementation of concepts such as gamification and mobile learning in music education, mentioning relevant case studies in this field, this dissertation culminates with the development of an application for Android smartphones entitled ’Practice As You Walk’. As the name implies, this learning tool consists in the reproduction of musical excerpts at the user’s walking pace, being a form of musical practice that stimulates the memorization of music pieces and synchronization ability of the individual. For the development of the application, this work explores innovative methods used for step detection through integrated sensors on mobile devices, such as the accelerometer and the gyroscope, and also presents the fundamentals of the MIDI communication protocol for the digital transmission of events related with musical performance. Two methods for smartphone-based step detection are proposed, with the rulebased method attaining an F1-score of 99% and the machine learning method attaining an F1-score of 95.84%. The development of the application, initially in the Unity platform, consists of integrating classes for MIDI file manipulation and processing with the ability to interpret and reproduce them at the user’s walking pace. Due to some faults identified in the music playback mechanism, migration to the Android Studio IDE took place through a third-party library that integrates the Sonivox EAS synthesizer. This abstraction from the playback mechanism allowed direct incorporation of the core functionalities developed in Unity and focus on the construction of a captivating user interface. Finally, within the pedagogical purpose of the present work, the application was tested by members of a children and youth choir. The questionnaire revealed general satisfaction with the application, allowing collection of opinions and suggestions on potential future improvements.Com a massificação global dos dispositivos móveis, o uso da tecnologia para fins pedagógicos no contexto da aprendizagem musical revelou ser uma ferramenta imprescindível para assegurar a motivação dos estudantes. Explorando a implementação dos conceitos de gamification e mobile learning no ensino da música, referindo casos de estudo relevantes neste campo, esta dissertação culmina com o desenvolvimento de uma aplicação para smartphones Android denominada ’Practice As You Walk’. Conforme indica o nome, esta ferramenta de aprendizagem consiste na reprodução de excertos musicais ao ritmo determinado pelo passo do utilizador, sendo uma forma de prática musical que estimula no indivíduo a memorização de obras musicais e a capacidade de sincronização. Com vista ao desenvolvimento da aplicação, são explorados neste trabalho métodos inovadores utilizados na deteção de passo através de sensores incorporados nos dispositivos móveis, tais como o acelerómetro e o giroscópio, e também apresentados os fundamentos do protocolo de comunicação MIDI para a transmissão digital de eventos relacionados com a interpretação musical. São propostos dois métodos para a deteção de passo com recurso a um smartphone, tendo o método baseado em regras atingido um F1-score de 99% e o método baseado em aprendizagem automática um F1-score de 95.84%. O desenvolvimento da aplicação, inicialmente na plataforma Unity, consiste na integração de classes para a manipulação e processamento de ficheiros MIDI com a capacidade de leitura e reprodução dos mesmos ao ritmo do passo do utilizador. Devido a alguns defeitos identificados no mecanismo de reprodução, segue-se a migração para o ambiente de desenvolvimento Android Studio recorrendo a uma biblioteca que integra o sintetizador Sonivox EAS. Esta abstração do mecanismo de reprodução permite a integração direta das funcionalidades desenvolvidas em Unity e um maior foco na construção de uma interface de utilizador cativante. Por fim, no âmbito pedagógico deste trabalho, a aplicação foi testada por membros de um coro infanto-juvenil. Um questionário revelou satisfação geral com a aplicação e permitiu a recolha de opiniões e sugestões tendo em vista potenciais melhorias.Mestrado em Engenharia Eletrónica e Telecomunicaçõe