Text categorization and similarity analysis: similarity measure, literature review

Document classification and provenance has become an important area of computer science as the amount of digital information is growing significantly. Organisations are storing documents on computers rather than in paper form. Software is now required that will show the similarities between documents (i.e. document classification) and to point out duplicates and possibly the history of each document (i.e. provenance). Poor organisation is common and leads to situations like above. There exists a number of software solutions in this area designed to make document organisation as simple as possible. I'm doing my project with Pingar who are a company based in Auckland who aim to help organise the growing amount of unstructured digital data. This reports analyses the existing literature in this area with the aim to determine what already exists and how my project will be different from existing solutions

Measuring Semantic Similarity by Latent Relational Analysis

This paper introduces Latent Relational Analysis (LRA), a method for measuring semantic similarity. LRA measures similarity in the semantic relations between two pairs of words. When two pairs have a high degree of relational similarity, they are analogous. For example, the pair cat:meow is analogous to the pair dog:bark. There is evidence from cognitive science that relational similarity is fundamental to many cognitive and linguistic tasks (e.g., analogical reasoning). In the Vector Space Model (VSM) approach to measuring relational similarity, the similarity between two pairs is calculated by the cosine of the angle between the vectors that represent the two pairs. The elements in the vectors are based on the frequencies of manually constructed patterns in a large corpus. LRA extends the VSM approach in three ways: (1) patterns are derived automatically from the corpus, (2) Singular Value Decomposition is used to smooth the frequency data, and (3) synonyms are used to reformulate word pairs. This paper describes the LRA algorithm and experimentally compares LRA to VSM on two tasks, answering college-level multiple-choice word analogy questions and classifying semantic relations in noun-modifier expressions. LRA achieves state-of-the-art results, reaching human-level performance on the analogy questions and significantly exceeding VSM performance on both tasks

Using entropy-based local weighting to improve similarity assessment

This paper enhances and analyses the power of local weighted similarity measures. The paper proposes a new entropy-based local weighting algorithm to be used in similarity assessment to improve the performance of the CBR retrieval task. It has been carried out a comparative analysis of the performance of unweighted similarity measures, global weighted similarity measures, and local weighting similarity measures. The testing has been done using several similarity measures, and some data sets from the UCI Machine Learning Database Repository and other environmental databases.Postprint (published version

Characterization and Similarity Analysis of 15 Tomato Genotypes in Lowlands Based on Morphological Characters

This study aimed to obtain information about the characteristics of 15 genotypes and to study a genetic similarity of each genotype that will be used for producing superior tomato varieties in lowlands. The research was conducted from March to August 2012 at the Experimental Field Leuwikopo Bogor Agricultural University, Darmaga Bogor. The experiment used The Randomized Complete Block Design (RCBD) using a single factor of genotype with three replications. Characterization and similarity analysis used the method of principal component analysis and cluster analysis. Based on principal component analysis and cluster analysis of tomato genotypes, it can be classified into three groups: group I (IPBT1, IPBT4, IPBT8, IPBT13, IPBT58, IPBT83 and IPBT84), Group II (IPBT3, IPBT23, IPBT30, IPBT33, IPBT34, IPBT53 and IPBT57) and group III (IPBT80). Characters with an influence on the genetic diversity of each component are the size of the cork layer between the scar stalk and the size of the center of the fruit in transverse slices. The genotypes with a high genetic similarity were IPBT1 and IPBT8, while IPBT30 with IPBT80 had a low genetic similarit

Text categorization and similarity analysis: similarity measure, architecture and design

This research looks at the most appropriate similarity measure to use for a document classification problem. The goal is to find a method that is accurate in finding both semantically and version related documents. A necessary requirement is that the method is efficient in its speed and disk usage. Simhash is found to be the measure best suited to the application and it can be combined with other software to increase the accuracy. Pingar have provided an API that will extract the entities from a document and create a taxonomy displaying the relationships and this extra information can be used to accurately classify input documents. Two algorithms are designed incorporating the Pingar API and then finally an efficient comparison algorithm is introduced to cut down the comparisons required

Similarity Search Over Graphs Using Localized Spectral Analysis

This paper provides a new similarity detection algorithm. Given an input set of multi-dimensional data points, where each data point is assumed to be multi-dimensional, and an additional reference data point for similarity finding, the algorithm uses kernel method that embeds the data points into a low dimensional manifold. Unlike other kernel methods, which consider the entire data for the embedding, our method selects a specific set of kernel eigenvectors. The eigenvectors are chosen to separate between the data points and the reference data point so that similar data points can be easily identified as being distinct from most of the members in the dataset.Comment: Published in SampTA 201

Similarity-based data mining in files of two-dimensional chemical structures using fingerprint measures of molecular resemblance

This paper reviews the use of measures of intermolecular similarity for processing databases of chemical structures, which play an important role in the discovery of new drugs by the pharmaceutical industry. The similarity measures considered here are based on the use of a fingerprint representation of molecular structure, where a fingerprint is a vector encoding the presence of fragment substructures in a molecule and where the similarity between pairs of such fingerprints is computed using an association coefficient such as the Tanimoto coefficient. The Similar Property Principle provides the basic rationale for the use of similarity methods in three important chemoinformatics applications—similarity searching, database clustering, and molecular diversity analysis. Similarity searching enables the identification of those molecules in a database that are most similar to a user-defined, biologically active query molecule, with data fusion providing an effective way of combining the results of multiple similarity searches. Cluster analysis, typically using the Jarvis–Patrick, Ward, or divisive k-means clustering methods, enables the cost-effective selection of molecules for biological testing, for property prediction and for investigating database overlap. Molecular diversity analysis, typically using cluster-based, dissimilarity-based, or optimization-based approaches, enables the identification of structurally diverse sets of molecules, so as to ensure that the full chemical space spanned by a database is tested in the search for novel bioactive molecules. © 2011 John Wiley & Sons, Inc. WIREs Data Mining Knowl Discov 2011 1 241–251 DOI: 10.1002/widm.2

The irony of choice in recruitment: when similarity turns recruiters to other candidates

Across two experimental studies, we examine the influence of similarity perceptions on recruiters’ job fit perceptions of job applicants. In addition, a robustness study extends the effect of similarity by introducing work-related sources of similarity and tests the relationship between workrelated similarities on similarity perceptions. Moreover, we explore the emotional and cognitive mechanisms behind the effects of similarity perceptions on job fit. We also propose and test a boundary condition, such that, when job desirability is low, the effect of demographic similarity on perceived similarity is reversed. The sample for the three studies consist of specialized master’s students with work experience in human resources management who acted as recruiters in a resume screening situation. The results show that the effects of similarity are not always positive for job fit perceptions. The studies provide evidence that when recruiters perceive applicants as similar to themselves, biased evaluations occur. Finally, we provide results that show the effects of mediation and moderation analysis whereby liking mediates the relationship between similarity perceptions and job fit perceptions through emotional, cognitive and motivational sequential mediators. Additionally, job desirability moderates the relationship between demographic similarity and similarity perceptions so that when job desirability is low, the effect of demographic similarity on perceived similarity is reversed

Path Similarity Analysis: a Method for Quantifying Macromolecular Pathways

Diverse classes of proteins function through large-scale conformational changes; sophisticated enhanced sampling methods have been proposed to generate these macromolecular transition paths. As such paths are curves in a high-dimensional space, they have been difficult to compare quantitatively, a prerequisite to, for instance, assess the quality of different sampling algorithms. The Path Similarity Analysis (PSA) approach alleviates these difficulties by utilizing the full information in 3N-dimensional trajectories in configuration space. PSA employs the Hausdorff or Fr\'echet path metrics---adopted from computational geometry---enabling us to quantify path (dis)similarity, while the new concept of a Hausdorff-pair map permits the extraction of atomic-scale determinants responsible for path differences. Combined with clustering techniques, PSA facilitates the comparison of many paths, including collections of transition ensembles. We use the closed-to-open transition of the enzyme adenylate kinase (AdK)---a commonly used testbed for the assessment enhanced sampling algorithms---to examine multiple microsecond equilibrium molecular dynamics (MD) transitions of AdK in its substrate-free form alongside transition ensembles from the MD-based dynamic importance sampling (DIMS-MD) and targeted MD (TMD) methods, and a geometrical targeting algorithm (FRODA). A Hausdorff pairs analysis of these ensembles revealed, for instance, that differences in DIMS-MD and FRODA paths were mediated by a set of conserved salt bridges whose charge-charge interactions are fully modeled in DIMS-MD but not in FRODA. We also demonstrate how existing trajectory analysis methods relying on pre-defined collective variables, such as native contacts or geometric quantities, can be used synergistically with PSA, as well as the application of PSA to more complex systems such as membrane transporter proteins.Comment: 9 figures, 3 tables in the main manuscript; supplementary information includes 7 texts (S1 Text - S7 Text) and 11 figures (S1 Fig - S11 Fig) (also available from journal site