Clustering and its Application in Requirements Engineering

Large scale software systems challenge almost every activity in the software development life-cycle, including tasks related to eliciting, analyzing, and specifying requirements. Fortunately many of these complexities can be addressed through clustering the requirements in order to create abstractions that are meaningful to human stakeholders. For example, the requirements elicitation process can be supported through dynamically clustering incoming stakeholders’ requests into themes. Cross-cutting concerns, which have a significant impact on the architectural design, can be identified through the use of fuzzy clustering techniques and metrics designed to detect when a theme cross-cuts the dominant decomposition of the system. Finally, traceability techniques, required in critical software projects by many regulatory bodies, can be automated and enhanced by the use of cluster-based information retrieval methods. Unfortunately, despite a significant body of work describing document clustering techniques, there is almost no prior work which directly addresses the challenges, constraints, and nuances of requirements clustering. As a result, the effectiveness of software engineering tools and processes that depend on requirements clustering is severely limited. This report directly addresses the problem of clustering requirements through surveying standard clustering techniques and discussing their application to the requirements clustering process

Probabilistic Graphical Modeling on Big Data

The rise of Big Data in recent years brings many challenges to modern statistical analysis and modeling. In toxicogenomics, the advancement of high-throughput screening technologies facilitates the generation of massive amount of biological data, a big data phenomena in biomedical science. Yet, researchers still heavily rely on key word search and/or literature review to navigate the databases and analyses are often done in rather small-scale. As a result, the rich information of a database has not been fully utilized, particularly for the information embedded in the interactive nature between data points that are largely ignored and buried. For the past 10 years, probabilistic topic modeling has been recognized as an effective machine learning algorithm to annotate the hidden thematic structure of massive collection of documents. The analogy between text corpus and large-scale genomic data enables the application of text mining tools, like probabilistic topic models, to explore hidden patterns of genomic data and to the extension of altered biological functions. In this study, we developed a generalized probabilistic topic model to analyze a toxicogenomics data set that consists of a large number of gene expression data from the rat livers treated with drugs in multiple dose and time-points. We discovered the hidden patterns in gene expression associated with the effect of doses and time-points of treatment. Finally, we illustrated the ability of our model to identify the evidence of potential reduction of animal use. In online Social network, Social network services have hundreds of millions, sometimes even billions, of monthly active users. These complex and vast Social networks are tremendous resources for understanding the human interactions. Especially, characterizing the strength of Social interactions becomes essential task for researching or marketing Social networks. Instead of traditional dichotomy of strong and weak tie assumption, we believe that there are more types of Social ties than just two. We use cosine similarity to measure the strength of the Social ties and apply incremental Dirichlet process Gaussian mixture model to group tie into different clusters of ties. Comparing to other methods, our approach generates superior accuracy in classification on data with ground truth. The incremental algorithm also allow data to be added or deleted in a dynamic Social network with minimal computer cost. In addition, it has been shown that the network constraints of individuals can be used to predict ones\u27 career successes. Under our multiple type of ties assumption, individuals are profiled based on their surrounding relationships. We demonstrate that network profile of a individual is directly linked to Social significance in real world

Towards improving WEBSOM with multi-word expressions

Dissertação para obtenção do Grau de Mestre em Engenharia InformáticaLarge quantities of free-text documents are usually rich in information and covers several topics. However, since their dimension is very large, searching and filtering data is an exhaustive task. A large text collection covers a set of topics where each topic is affiliated to a group of documents. This thesis presents a method for building a document map about the core contents covered in the collection. WEBSOM is an approach that combines document encoding methods and Self-Organising Maps (SOM) to generate a document map. However, this methodology has a weakness in the document encoding method because it uses single words to characterise documents. Single words tend to be ambiguous and semantically vague, so some documents can be incorrectly related. This thesis proposes a new document encoding method to improve the WEBSOM approach by using multi word expressions (MWEs) to describe documents. Previous research and ongoing experiments encourage us to use MWEs to characterise documents because these are semantically more accurate than single words and more descriptive

Método de agrupamiento no supervisado para el procesamiento del lenguaje natural utilizando medidas de similitud asimétricas y propiedades paradigmáticas

Una de las tareas más comunes para el ser humano, pero de con una alta complejidad es la agrupación y clasificación. Por otro lado, la debilidad del ser humano es la capacidad de procesar altas cantidades de datos y de forma rápida, característica propia de los computadores. Hoy en día se generan grandes cantidades de datos en el Internet, datos de distintos tipos y con diferentes objetivos. Para esto se necesitan de algoritmos de agrupación que nos permitan identificar los distintos grupos y características de estos grupos, de forma automática sin conocimiento previo. Por otro lado, es importante definir con claridad qué medida de similitud se utilizará en el proceso de agrupación, la gran mayoría de las medidas de agrupación se enfocan en un aspecto simétrico. En la presente tesis se propone una novedosa medida de similitud asimétrica, Coeficiente d Similitud Unilateral Jaccard (uJaccard), similitud no es igual entre dos objetos uJaccard(a,b) ≠ uJaccard(b,a). Así también se presenta una similitud asimétrica con pesos Coeficiente Ponderado de Similitud Unilateral Jaccard, la cual mide el nivel de incertidumbre entre dos objetos. Así también en esta tesis se propone una nueva propiedad de grafos, la propiedad paradigmática la cual considera la equivalencia regular como característica fundamental y por último se propone un algoritmo de agrupación PaC, por sus siglas en inglés Paradigmatic Clustering, el cual incorpora la uJaccard y la propiedad paradigmática. Se ha realizado evaluaciones extensivas con datos pequeños, reales, sintéticos y se ha procesado 3 grandes corpus. Se ha demostrado que PaC es un algoritmo que sobre pasa los resultados de algoritmos de agrupación del estado del arte. Más aun PaC es un algoritmo capas de ser ejecutado de forma paralela, distribuida, incremental y en flujo, características que se necesitan para el procedimiento de grandes cantidades de datos y de constante generación de dato

Swarm intelligence for clustering dynamic data sets for web usage mining and personalization.

Swarm Intelligence (SI) techniques were inspired by bee swarms, ant colonies, and most recently, bird flocks. Flock-based Swarm Intelligence (FSI) has several unique features, namely decentralized control, collaborative learning, high exploration ability, and inspiration from dynamic social behavior. Thus FSI offers a natural choice for modeling dynamic social data and solving problems in such domains. One particular case of dynamic social data is online/web usage data which is rich in information about user activities, interests and choices. This natural analogy between SI and social behavior is the main motivation for the topic of investigation in this dissertation, with a focus on Flock based systems which have not been well investigated for this purpose. More specifically, we investigate the use of flock-based SI to solve two related and challenging problems by developing algorithms that form critical building blocks of intelligent personalized websites, namely, (i) providing a better understanding of the online users and their activities or interests, for example using clustering techniques that can discover the groups that are hidden within the data; and (ii) reducing information overload by providing guidance to the users on websites and services, typically by using web personalization techniques, such as recommender systems. Recommender systems aim to recommend items that will be potentially liked by a user. To support a better understanding of the online user activities, we developed clustering algorithms that address two challenges of mining online usage data: the need for scalability to large data and the need to adapt cluster sing to dynamic data sets. To address the scalability challenge, we developed new clustering algorithms using a hybridization of traditional Flock-based clustering with faster K-Means based partitional clustering algorithms. We tested our algorithms on synthetic data, real VCI Machine Learning repository benchmark data, and a data set consisting of real Web user sessions. Having linear complexity with respect to the number of data records, the resulting algorithms are considerably faster than traditional Flock-based clustering (which has quadratic complexity). Moreover, our experiments demonstrate that scalability was gained without sacrificing quality. To address the challenge of adapting to dynamic data, we developed a dynamic clustering algorithm that can handle the following dynamic properties of online usage data: (1) New data records can be added at any time (example: a new user is added on the site); (2) Existing data records can be removed at any time. For example, an existing user of the site, who no longer subscribes to a service, or who is terminated because of violating policies; (3) New parts of existing records can arrive at any time or old parts of the existing data record can change. The user\u27s record can change as a result of additional activity such as purchasing new products, returning a product, rating new products, or modifying the existing rating of a product. We tested our dynamic clustering algorithm on synthetic dynamic data, and on a data set consisting of real online user ratings for movies. Our algorithm was shown to handle the dynamic nature of data without sacrificing quality compared to a traditional Flock-based clustering algorithm that is re-run from scratch with each change in the data. To support reducing online information overload, we developed a Flock-based recommender system to predict the interests of users, in particular focusing on collaborative filtering or social recommender systems. Our Flock-based recommender algorithm (FlockRecom) iteratively adjusts the position and speed of dynamic flocks of agents, such that each agent represents a user, on a visualization panel. Then it generates the top-n recommendations for a user based on the ratings of the users that are represented by its neighboring agents. Our recommendation system was tested on a real data set consisting of online user ratings for a set of jokes, and compared to traditional user-based Collaborative Filtering (CF). Our results demonstrated that our recommender system starts performing at the same level of quality as traditional CF, and then, with more iterations for exploration, surpasses CF\u27s recommendation quality, in terms of precision and recall. Another unique advantage of our recommendation system compared to traditional CF is its ability to generate more variety or diversity in the set of recommended items. Our contributions advance the state of the art in Flock-based 81 for clustering and making predictions in dynamic Web usage data, and therefore have an impact on improving the quality of online services

Learning Image Re-Rank: Query-Dependent Image Re-Ranking Using Semantic Signature

ABSTRACT: Image re-ranking, is an effective way to improve the results of web-based image search and has been adopted by current commercial search engines such as Bing and Google. When a query keyword is given, a list of images are first retrieved based on textual information given by the user. By asking the user to select a query image from the pool of images, the remaining images are re-ranked based on their index with the query image. A major challenge is that sometimes semantic meanings may interpret user's search intention. Many people recently proposed to match images in a semantic space which used attributes or reference classes closely related to the semantic meanings of images as basis. In this paper, we propose a novel image re-ranking framework, in which automatically offline learns different semantic spaces for different query keywords and displays with the image details in the form of augmented images. The images are projected into their related semantic spaces to get semantic signatures with the help of one click feedback from the user. At the online stage, images are re-ranked by comparing their semantic signatures obtained from the semantic space specified by the query keyword given by the user. The proposed query-specific semantic signatures significantly improve both the accuracy and efficiency of image re-ranking. Experimental results show that 25-40 percent relative improvement has been achieved on re-ranking precisions compared with the state-of-the-art methods

A Survey on Important Aspects of Information Retrieval

Information retrieval has become an important field of study and research under computer science due to the explosive growth of information available in the form of full text, hypertext, administrative text, directory, numeric or bibliographic text. The research work is going on various aspects of information retrieval systems so as to improve its efficiency and reliability. This paper presents a comprehensive survey discussing not only the emergence and evolution of information retrieval but also include different information retrieval models and some important aspects such as document representation, similarity measure and query expansion