Sub-word indexing and blind relevance feedback for English, Bengali, Hindi, and Marathi IR

The Forum for Information Retrieval Evaluation (FIRE) provides document collections, topics, and relevance assessments for information retrieval (IR) experiments on Indian languages. Several research questions are explored in this paper: 1. how to create create a simple, languageindependent corpus-based stemmer, 2. how to identify sub-words and which types of sub-words are suitable as indexing units, and 3. how to apply blind relevance feedback on sub-words and how feedback term selection is affected by the type of the indexing unit. More than 140 IR experiments are conducted using the BM25 retrieval model on the topic titles and descriptions (TD) for the FIRE 2008 English, Bengali, Hindi, and Marathi document collections. The major findings are: The corpus-based stemming approach is effective as a knowledge-light term conation step and useful in case of few language-specific resources. For English, the corpusbased stemmer performs nearly as well as the Porter stemmer and significantly better than the baseline of indexing words when combined with query expansion. In combination with blind relevance feedback, it also performs significantly better than the baseline for Bengali and Marathi IR. Sub-words such as consonant-vowel sequences and word prefixes can yield similar or better performance in comparison to word indexing. There is no best performing method for all languages. For English, indexing using the Porter stemmer performs best, for Bengali and Marathi, overlapping 3-grams obtain the best result, and for Hindi, 4-prefixes yield the highest MAP. However, in combination with blind relevance feedback using 10 documents and 20 terms, 6-prefixes for English and 4-prefixes for Bengali, Hindi, and Marathi IR yield the highest MAP. Sub-word identification is a general case of decompounding. It results in one or more index terms for a single word form and increases the number of index terms but decreases their average length. The corresponding retrieval experiments show that relevance feedback on sub-words benefits from selecting a larger number of index terms in comparison with retrieval on word forms. Similarly, selecting the number of relevance feedback terms depending on the ratio of word vocabulary size to sub-word vocabulary size almost always slightly increases information retrieval effectiveness compared to using a fixed number of terms for different languages

A study on n-gram indexing of musical features

Since only simple symbol-based manipulations are needed, n-gram indexing is used for natural languages where syntactic or semantic analyses are often difficult. Music, whose automatic analysis for patterns such as motifs and phrases are difficult, inaccurate or computationally expensive, is thus similar to natural languages. The use of n-gram in music retrieval systems is thus a natural choice. In this paper, we study a number of issues regarding n-gram indexing of musical features using simulated queries. They are: whether combinatorial explosion is a problem in n-gram indexing of musical features, the relative discrimination power of six different musical features, the value of n needed for them, and the average amount of false positives returned when n-grams are used to index music.published_or_final_versio

Japanese/English Cross-Language Information Retrieval: Exploration of Query Translation and Transliteration

Cross-language information retrieval (CLIR), where queries and documents are in different languages, has of late become one of the major topics within the information retrieval community. This paper proposes a Japanese/English CLIR system, where we combine a query translation and retrieval modules. We currently target the retrieval of technical documents, and therefore the performance of our system is highly dependent on the quality of the translation of technical terms. However, the technical term translation is still problematic in that technical terms are often compound words, and thus new terms are progressively created by combining existing base words. In addition, Japanese often represents loanwords based on its special phonogram. Consequently, existing dictionaries find it difficult to achieve sufficient coverage. To counter the first problem, we produce a Japanese/English dictionary for base words, and translate compound words on a word-by-word basis. We also use a probabilistic method to resolve translation ambiguity. For the second problem, we use a transliteration method, which corresponds words unlisted in the base word dictionary to their phonetic equivalents in the target language. We evaluate our system using a test collection for CLIR, and show that both the compound word translation and transliteration methods improve the system performance

Searching strategies for the Bulgarian language

This paper reports on the underlying IR problems encountered when indexing and searching with the Bulgarian language. For this language we propose a general light stemmer and demonstrate that it can be quite effective, producing significantly better MAP (around + 34%) than an approach not applying stemming. We implement the GL2 model derived from the Divergence from Randomness paradigm and find its retrieval effectiveness better than other probabilistic, vector-space and language models. The resulting MAP is found to be about 50% better than the classical tf idf approach. Moreover, increasing the query size enhances the MAP by around 10% (from T to TD). In order to compare the retrieval effectiveness of our suggested stopword list and the light stemmer developed for the Bulgarian language, we conduct a set of experiments on another stopword list and also a more complex and aggressive stemmer. Results tend to indicate that there is no statistically significant difference between these variants and our suggested approach. This paper evaluates other indexing strategies such as 4-gram indexing and indexing based on the automatic decompounding of compound words. Finally, we analyze certain queries to discover why we obtained poor results, when indexing Bulgarian documents using the suggested word-based approac

Sentiment Polarity Classification of Comments on Korean News Articles Using Feature Reweighting

일반적으로 인터넷 신문 기사에 대한 댓글은 그 신문 기사에 대한 주관적인 감정이나 의견을 포함하고 있다. 따라서 이런 신문 기사의 댓글에 대한 감정을 인식하고 분류하는 데에는 그 신문 기사의 원문 내용이 중요한 영향을 미친다. 이런 점에 착안하여 본 논문은 기사의 원문 내용과 감정 사전을 이용하는 가중치 조정 방법을 제안하고, 제안된 가중치 조정 방법을 이용해서 한국어 신문 기사의 댓글에 대한 감정 이진 분류 방법을 제안한다. 가중치 조정 방법에는 다양한 자질 집합이 사용되는데 그것은 댓글에 포함된 감정 단어, 그리고 감정 사전과 뉴스 기사의 본문에 관련된 자질들, 마지막으로 뉴스 기사의 카테고리 정보가 포함되어 있다. 여기서 말하는 감정 사전은 한국어 감정 사전을 의미하며 아직 공개된 것이 없기 때문에, 기존에 있는 영어 감정 사전을 이용하여 구축하였다. 본 논문에서 제안된 감정 이진 분류는 기계 학습을 이용한다. 일반적으로 기계 학습을 위해서는 학습 말뭉치가 필요한데 특별히 감정 분류 문제에서는 긍정 혹은 부정 감정 태그가 부착된 말뭉치가 필요하다. 이 말뭉치의 경우도, 공개된 한국어 감정 말뭉치가 아직 없기 때문에 말뭉치를 직접 구축하였다. 사용된 기계 학습 방법으로는 Na&iumlve Bayes, k-NN, SVM이 있고, 자질 선택 방법으로는 Document Frequency, χ^2 statistic, Information Gain이 있다. 그 결과, 댓글 안에 포함된 감정 단어와 그 댓글에 대한 기사 본문이 감정 분류에 매우 효과적인 자질임을 확인할 수 있었다.Chapter 1 Introduction 1 Chapter 2 Related Works 4 2.1 Sentiment Classification 4 2.2 Feature Weighting in Vector Space Model 5 2.3 Feature Extraction and Selection 7 2.4 Classifiers 10 2.5 Accuracy Measures 14 Chapter 3 Feature Reweighting 16 3.1 Feature extraction in Korean 16 3.2 Feature Reweighting Methods 17 3.3 Examples of Feature Reweighting Methods 18 Chapter 4 Sentiment Polarity Classification System 21 4.1 Model Generation 21 4.2 Sentiment Polarity Classification 23 Chapter 5 Data Preparation 25 5.1 Korean Sentiment Corpus 25 5.2 Korean Sentiment Lexicon 27 Chapter 6 Experiments 29 6.1 Experimental Environment 29 6.2 Experimental Results 30 Chapter 7 Conclusions and Future Works 38 Bibliography 40 Acknowledgments 4

TIR over Egyptian Hieroglyphs

We would like to thank Dr. Josep Cervello Autuori, Director of the Institut d’Estudis del Proxim Orient Antic (IEPOA) of the Universitat Autonoma de Barcelona, for introducing us to Egyptian; and Dr. Serge Rosmorduc, Associate of the Conservatoire National des Arts et Metiers (CNAM), for his support with JSESH[Abstract] This work presents an Information Retrieval system specifically designed to manage Ancient Egyptian hieroglyphic texts taking into account their peculiarities both at lexical and at encoding level for its application in Egyptology and Digital Heritage. The tool has been made freely available to the research community under a free license and, to the best of our knowledge, it is the first tool of its kindMinisterio de Economía y Competitividad; FFI2014-51978-C2-2-