Rakenduste kasutajaarvustustest informatsiooni kaevandamine tarkvara arendustegevuste soodustamiseks

Kasutajate vajaduste ja ootuste hindamine on arendajate jaoks oluline oma tarkvararakenduste kvaliteedi parandamiseks. Mobiilirakenduste platvormidele sisestatud arvustused on kasulikuks infoallikaks kasutajate pidevalt muutuvate vajaduste hindamiseks. Igapäevaselt rakenduste platvormidele esitatud arvustuste suur maht nõuab aga automaatseid meetodeid neist kasuliku info leidmiseks. Arvustuste automaatseks liigitamiseks, nt veateatis või uue funktsionaalsuse küsimine, saab kasutada teksti klassifitseerimismudeleid. Rakenduse funktsioonide automaatne kaevandamine arvustustest aitab teha kokkuvõtteid kasutajate meelsusest rakenduse olemasolevate funktsioonide osas. Kõigepealt eksperimenteerime erinevate tekstiklassifitseerimise mudelitega ning võrdleme lihtsaid, leksikaalseid tunnuseid kasutavaid mudeleid keerukamatega, mis kasutavad rikkalikke lingvistilisi tunnuseid või mis põhinevad tehisnärvivõrkudel. Erinevate faktorite mõju uurimiseks funktsioonide kaevandamise meetoditele me teeme kõigepealt kindlaks erinevate meetodite baastaseme täpsuse rakendades neid samades eksperimentaalsetes tingimustes. Seejärel võrdleme neid meetodeid erinevates tingimustes, varieerides treenimiseks kasutatud annoteeritud andmestikke ning hindamismeetodeid. Kuna juhendatud masinõppel baseeruvad kaevandamismeetodid on võrreldes reeglipõhistega tundlikumad (1) andmete märgendamisel kasutatud annoteerimisjuhistele ning (2) märgendatatud andmestiku suurusele, siis uurisime nende faktorite mõju juhendatud masinõppe kontekstis ja pakkusime välja uued annoteerimisjuhised, mis võivad aidata funktsioonide kaevandamise täpsust parandada. Käesoleva doktoritöö projekti tulemusel valmis ka kontseptuaalne tööriist, mis võimaldab konkureerivaid rakendusi omavahel võrrelda. Tööriist kombineerib arvustuse tekstide klassifitseerimise ja rakenduse funktsioonide kaevandamise meetodid. Tööriista hinnanud kümme tarkvaraarendajat leidsid, et sellest võib olla kasu rakenduse kvaliteedi parandamiselFor app developers, it is important to continuously evaluate the needs and expectations of their users to improve app quality. User reviews submitted to app marketplaces are regarded as a useful information source to re-access evolving user needs. The large volume of user reviews received every day requires automatic methods to find such information in user reviews. Text classification models can be used to categorize review information into types such as feature requests and bug reports, while automatic app feature extraction from user reviews can help in summarizing users’ sentiments at the level of app features. For classifying review information, we perform experiments to compare the performance of simple models using only lexical features to models with rich linguistic features and models built on deep learning architectures, i.e., Convolutional Neural Network (CNN). To investigate factors influencing the performance of automatic app feature extraction methods, i.e. rule-based and supervised machine learning, we first establish a baseline in a single experimental setting and then compare the performances in different experimental settings (i.e., varying annotated datasets and evaluation methods). Since the performance of supervised feature extraction methods is more sensitive than rule- based methods to (1) guidelines used to annotate app features in user reviews and (2) the size of the annotated data, we investigate their impact on the performance of supervised feature extraction models and suggest new annotation guidelines that have the potential to improve feature extraction performance. To make the research results of the thesis project also applicable for non-experts, we developed a proof-of-concept tool for comparing competing apps. The tool combines review classification and app feature extraction methods and has been evaluated by ten developers from industry who perceived it useful for improving the app quality.  https://www.ester.ee/record=b529379

Mining app reviews to support software engineering

The thesis studies how mining app reviews can support software engineering. App reviews —short user reviews of an app in app stores— provide a potentially rich source of information to help software development teams maintain and evolve their products. Exploiting this information is however difficult due to the large number of reviews and the difficulty in extracting useful actionable information from short informal texts. A variety of app review mining techniques have been proposed to classify reviews and to extract information such as feature requests, bug descriptions, and user sentiments but the usefulness of these techniques in practice is still unknown. Research in this area has grown rapidly, resulting in a large number of scientific publications (at least 182 between 2010 and 2020) but nearly no independent evaluation and description of how diverse techniques fit together to support specific software engineering tasks have been performed so far. The thesis presents a series of contributions to address these limitations. We first report the findings of a systematic literature review in app review mining exposing the breadth and limitations of research in this area. Using findings from the literature review, we then present a reference model that relates features of app review mining tools to specific software engineering tasks supporting requirements engineering, software maintenance and evolution. We then present two additional contributions extending previous evaluations of app review mining techniques. We present a novel independent evaluation of opinion mining techniques using an annotated dataset created for our experiment. Our evaluation finds lower effectiveness than initially reported by the techniques authors. A final part of the thesis, evaluates approaches in searching for app reviews pertinent to a particular feature. The findings show a general purpose search technique is more effective than the state-of-the-art purpose-built app review mining techniques; and suggest their usefulness for requirements elicitation. Overall, the thesis contributes to improving the empirical evaluation of app review mining techniques and their application in software engineering practice. Researchers and developers of future app mining tools will benefit from the novel reference model, detailed experiments designs, and publicly available datasets presented in the thesis

Text Annotation Handbook: A Practical Guide for Machine Learning Projects

This handbook is a hands-on guide on how to approach text annotation tasks. It provides a gentle introduction to the topic, an overview of theoretical concepts as well as practical advice. The topics covered are mostly technical, but business, ethical and regulatory issues are also touched upon. The focus lies on readability and conciseness rather than completeness and scientific rigor. Experience with annotation and knowledge of machine learning are useful but not required. The document may serve as a primer or reference book for a wide range of professions such as team leaders, project managers, IT architects, software developers and machine learning engineers.Comment: 30 pages, white pape

Detection of spam review on mobile app stores, evaluation of helpfulness of user reviews and extraction of quality aspects using machine learning techniques

As mobile devices have overtaken fixed Internet access, mobile applications and distribution platforms have gained in importance. App stores enable users to search and purchase mobile applications and then to give feedback in the form of reviews and ratings. A review might contain critical information about user experience, feature requests and bug reports. User reviews are valuable not only to developers and software organizations interested in learning the opinion of their customers but also to prospective users who would like to find out what others think about an app. Even though some surveys have inventoried techniques and methods in opinion mining and sentiment analysis, no systematic literature review (SLR) study had yet reported on mobile app store opinion mining and spam review detection problems. Mining opinions from app store reviews requires pre-processing at the text and content levels, including filtering-out nonopinionated content and evaluating trustworthiness and genuineness of the reviews. In addition, the relevance of the extracted features are not cross-validated with main software engineering concepts. This research project first conducted a systematic literature review (SLR) on the evaluation of mobile app store opinion mining studies. Next, to fill the identified gaps in the literature, we used a novel convolutional neural network to learn document representation for deceptive spam review detection by characterizing an app store review dataset which includes truthful and spam reviews for the first time in the literature. Our experiments reported that our neural network based method achieved 82.5% accuracy, while a baseline Support Vector Machine (SVM) classification model reached only 70% accuracy despite leveraging various feature combinations. We next compared four classification models to assess app store user review helpfulness and proposed a predictive model which makes use of review meta-data along with structural and lexical features for helpfulness prediction. In the last part of this research study, we constructed an annotated app store review dataset for the aspect extraction task, based on ISO 25010 - Systems and software Product Quality Requirements and Evaluation standard and two deep neural network models: Bi-directional Long-Short Term Memory and Conditional Random Field (Bi-LSTM+CRF) and Deep Convolutional Neural Networks and Conditional Random Field (CNN+CRF) for aspect extraction from app store user reviews. Both models achieved nearly 80% F1 score (the weighted average of precision and recall which takes both false positives and false negatives into account) in exact aspect matching and 86% F1 score in partial aspect matching

Argumentation Mining in User-Generated Web Discourse

The goal of argumentation mining, an evolving research field in computational linguistics, is to design methods capable of analyzing people's argumentation. In this article, we go beyond the state of the art in several ways. (i) We deal with actual Web data and take up the challenges given by the variety of registers, multiple domains, and unrestricted noisy user-generated Web discourse. (ii) We bridge the gap between normative argumentation theories and argumentation phenomena encountered in actual data by adapting an argumentation model tested in an extensive annotation study. (iii) We create a new gold standard corpus (90k tokens in 340 documents) and experiment with several machine learning methods to identify argument components. We offer the data, source codes, and annotation guidelines to the community under free licenses. Our findings show that argumentation mining in user-generated Web discourse is a feasible but challenging task.Comment: Cite as: Habernal, I. & Gurevych, I. (2017). Argumentation Mining in User-Generated Web Discourse. Computational Linguistics 43(1), pp. 125-17